Effect of the Low Risk Ankle Rule on the frequency of radiography in children with ankle injuries
=================================================================================================

* Kathy Boutis
* Paul Grootendorst
* Andrew Willan
* Amy C. Plint
* Paul Babyn
* Robert J. Brison
* Arun Sayal
* Melissa Parker
* Natalie Mamen
* Suzanne Schuh
* Jeremy Grimshaw
* David Johnson
* Unni Narayanan

## Abstract

**Background:** The Low Risk Ankle Rule is a validated clinical decision rule that has the potential to safely reduce radiography in children with acute ankle injuries. We performed a phased implementation of the Low Risk Ankle Rule and evaluated its effectiveness in reducing the frequency of radiography in children with ankle injuries.

**Methods:** Six Canadian emergency departments participated in the study from Jan. 1, 2009, to Aug. 31, 2011. At the 3 intervention sites, there were 3 consecutive 26-week phases. In phase 1, no interventions were implemented. In phase 2, we activated strategies to implement the ankle rule, including physician education, reminders and a computerized decision support system. In phase 3, we included only the decision support system. No interventions were introduced at the 3 pair-matched control sites. We examined the management of ankle injuries among children aged 3–16 years. The primary outcome was the proportion of children undergoing radiography.

**Results:** We enrolled 2151 children with ankle injuries, 1055 at intervention and 1096 at control hospitals. During phase 1, the baseline frequency of pediatric ankle radiography at intervention and control sites was 96.5% and 90.2%, respectively. During phase 2, the frequency of ankle radiography decreased significantly at intervention sites relative to control sites (between-group difference −21.9% [95% confidence interval [CI] −28.6% to −15.2%]), without significant differences in patient or physician satisfaction. All effects were sustained in phase 3. The sensitivity of the Low Risk Ankle Rule during implementation was 100% (95% CI 85.4% to 100%), and the specificity was 53.1% (95% CI 48.1% to 58.1%).

**Interpretation:** Implementation of the Low Risk Ankle Rule in several different emergency department settings reduced the rate of pediatric ankle radiography significantly and safely, without an accompanying change in physician or patient satisfaction. Trial registration: ClinicalTrials.gov, no. [NCT00785876](http://www.cmaj.ca/lookup/external-ref?link_type=CLINTRIALGOV&access_num=NCT00785876&atom=%2Fcmaj%2F185%2F15%2FE731.atom).

Pediatric ankle injuries result in more than 2 million emergency department visits in Canada and the United States each year (Jeanette Tyas, Canadian Institute of Health Information: unpublished data, 2007).1,2 Radiographs are ordered for 85%–95% of these children,3 although only 12% of these reveal a fracture.4 Thus, radiography is unnecessary for most children’s ankle injuries, and these high rates of radiography needlessly expose children to radiation and are a questionable use of resources.

The Low Risk Ankle Rule has 100% sensitivity with respect to identifying clinically important pediatric ankle fractures and has the potential to safely reduce imaging by about 60%.4 When the application of the rule suggests that radiography is not needed, it has been shown that any fractures that might be missed are clinically insignificant and can be safely and cost-effectively managed like an ankle sprain, with superior functional recovery.5 Finally, the Low Risk Ankle Rule has been shown to have excellent acceptability when tested on emergency physicians.6

The main objective of this study was to implement the ankle rule in several different emergency department settings using a multimodal knowledge translation strategy and to evaluate its impact on the frequency of radiography in children presenting with acute ankle injuries.

## Methods

### Study design and settings

We conducted this study over an 18-month period at 6 Canadian emergency departments, using an interrupted time series with pair-matched control design.7 Participating sites represented a convenience sample of 6 hospitals located in Ontario, Canada. We selected intervention sites based on the availability of infrastructure to support implementation of the intervention strategies. We matched intervention sites to control sites based on the following demographics: hospital type, annual census and characteristics of pediatric ankle injuries and emergency physicians. Sites included university-affiliated pediatric emergency departments (Pediatric) in 2 children’s hospitals, general emergency departments (General) in 2 urban university-affiliated teaching hospitals, and community emergency departments (Community) in 2 suburban hospitals with a limited teaching and academic mandate. One of the intervention sites (Pediatric) was part of the original validation of the Low Risk Ankle Rule.4 Patients were enrolled from Jan. 1, 2009, to Sept. 30, 2010, at the Pediatric and Community emergency departments, and from Jan. 1, 2010, to Aug. 31, 2011, at the General sites.

### Study population

We targeted the emergency physicians at the intervention sites for the intervention strategies. At the Pediatric hospitals, the emergency department was staffed by pediatricians, some also trained in pediatric emergency medicine. At the General and Community hospitals, emergency physicians were trained in emergency medicine.

Children between 3 and 16 years of age presenting with an isolated acute, nonpenetrating ankle injury were eligible for application of the Low Risk Ankle Rule. Children were excluded if they presented to the emergency department more than 72 hours after the injury occurred or with prior radiographs, had developmental delay (i.e., limited ambulation, inability to localize pain), were at risk for pathologic fractures or had recent prior history of injury to the same ankle.

### Definitions

The Low Risk Ankle Rule states that if a child with an ankle injury has a low-risk examination (i.e., tenderness and swelling isolated to the distal fibula and/or adjacent lateral ligaments distal to the tibial anterior joint line), ankle radiography may not be necessary to further exclude a high-risk ankle injury (Figure 1).4 A high-risk injury includes any fracture of the foot, distal tibia and fibula proximal to the distal physis, tibiofibular syndesmosis injury and ankle dislocations. Low-risk ankle injuries include lateral ankle sprains, nondisplaced Salter–Harris types I and II fractures of the distal fibula, and avulsion fractures of the distal fibula or lateral talus,4 which can all be managed functionally by supportive splinting and return to activities as tolerated by the patient.5

![Figure 1:](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/185/15/E731/F1.medium.gif)

[Figure 1:](http://www.cmaj.ca/content/185/15/E731/F1)

Figure 1: 
If a child presents with a low-risk injury on examination (i.e., tenderness and swelling isolated to the distal fibula and/or adjacent lateral ligaments distal to the tibial anterior joint line), then radiography may not be necessary to exclude a clinically important ankle injury. Reproduced, with permission, from Boutis et al.4

### Study interventions and administration

#### Intervention sites

During phase 1, emergency physicians completed a study data collection form detailing clinical findings, diagnosis and management, and physician satisfaction. Phase 1 served as an internal control at these sites with no measures to encourage rule use. In phase 2, a multimodal intervention strategy was introduced to encourage use of the Low Risk Ankle Rule based on knowledge of barriers to implementing strategies,6,8,9 available evidence7,10–17 and consultation with knowledge transfer experts. Emergency physicians underwent a teaching session and were provided with pocket cards that outlined the use of the ankle rule. Wall posters reviewing the rule were displayed in the emergency department, and charts of children with ankle injuries were flagged. Physician champions were identified at each site. We also provided a computerized decision support system for physicians to enter key clinical variables that would automatically generate the recommendation of the ankle rule for radiography. This system was not linked to ordering of ankle radiographs nor was it a requirement to complete clinical care. The data collection form from phase 1 was modified to include the ankle rule and related management strategy, whether the rule was followed and reasons for nonadherence. During this phase, physicians could detail the physical examination and use of the rule on the study form or the decision support system. Finally, physicians were trained to manage all low-risk ankle injuries in children (with or without radiography) with a removable immobilization device (e.g., brace), crutches as needed for pain and return to activities as tolerated. During phase 3, the only intervention strategy retained was the decision support system.

#### Control sites

No efforts were made to alter the usual emergency department procedures during the study period, and no patient-level physical examination data were collected to minimize any observer effects.

#### All sites

Emergency department patient logs were reviewed via tracking systems to ensure identification and data capture for all eligible participants. Consultant radiologists reported on all radiographs within 48 hours, and this diagnosis was considered conclusive. Where there was diagnostic uncertainty or no radiograph taken, we determined the final diagnosis by reviewing the emergency department notes, follow-up images or clinic notes; and/or by consensus between the collaborating radiologist and orthopedic surgeon. We contacted enrolled families by telephone on days 7 and 28 after the initial emergency department visit. For children who had not returned to baseline activities, we followed up again at 12 weeks. Parents were questioned about their satisfaction with the initial emergency department visit (on day 7), the clinical status of the child, imaging performed after the emergency department visit, physician visits and new diagnoses.

### Outcomes

The primary outcome was the proportion of eligible participants who received radiography. Because radiographic images were digitally stored on institutional digital archiving systems, records of radiography performed were complete and accurate. Secondary outcomes included the following: proportion of children with missed significant fractures for any reason per month as determined by available follow-up imaging, clinic records or parental report; proportion of children per month who did not receive ankle radiography in the emergency department but who subsequently underwent radiography; total length of stay in hours for pediatric ankle injuries measured as difference in discharge to triage time; and emergency physician and patient satisfaction measured on a 5-point ordinal rating scale (very satisfied, satisfied, neutral, unsatisfied, very unsatisfied).

### Statistical analysis

We used Monte Carlo simulation methods18 to estimate the power of the interrupted time series regression model to detect a minimally important treatment effect of a 20% reduction in rates of radiography.6 This power simulation required the following estimates derived from site administrative data: no secular trends were present in the radiography rates; an average of 4.5 ankle injuries per week per site or 2100 injuries across 78 weeks and 6 sites; and 97% radiography of these injuries. Using these methods, test power was estimated to be 95%.

We analyzed the primary outcome, proportion of radiographs, using a general linear model with a compound symmetry error structure to account for the clustering of observations within hospitals. We used a linear link function so that the effect of the Low Risk Ankle Rule could be given as the estimated difference in the probability that radiography was not ordered. Analyses of the secondary outcomes were performed using the interrupted time series “difference of difference” method.19–21 We implemented these analyses using a linear regression model that incorporated hospital site–specific indicator variables to control for unobserved time-invariant, site-specific factors that affected the specific outcome. We estimated the parameters of the time series regression model using ordinary least squares and estimated standard errors using methods that were robust to autocorrelation and heteroscedasticity. To ensure that hospital-specific temporal changes did not confound the results of our secondary analyses, we used a regression model to examine the temporal trends of secondary outcomes at intervention and control hospitals during phase 1 of the study. We measured the ankle rule’s performance by determining the sensitivity, specificity, negative and positive predictive value, and likelihood ratios. We calculated the sensitivity of the rule according to its ability to detect high-risk injuries, and we used the rule’s identification of low-risk injuries to estimate the specificity. All analyses were performed using Stata 10.

### Ethics approval

The study was approved by the Hospital for Sick Children Research Ethics Board and the research ethics boards at all participating sites. No consent was required for collection of radiography data. Consent was obtained in the emergency department for study telephone calls. When consent did not occur in the emergency department, all but 1 site (Pediatric Control) permitted a letter to be sent to families informing them of the study telephone calls and the process for opting out of the study.

## Results

We enrolled 2151 patients, 1055 at intervention and 1096 at control sites (Figure 2). Radiography data were obtained for all patients who met the eligibility criteria. Follow-up was achieved for 81.7% of participants at the intervention sites for the outcomes of missed high-risk injury, follow-up imaging and patient satisfaction. We collected the physical examination data for 694 (65.7%) of the participants at intervention sites. During phases 2 and 3, the decision support system was used to record data for 89.8% of patients (386 of 430 patients enrolled during phases 2 and 3 at intervention sites where data collection was completed by physicians either by paper form or computer decision support system). Table 1 describes the baseline characteristics of participating patients and physicians.

![Figure 2:](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/185/15/E731/F2.medium.gif)

[Figure 2:](http://www.cmaj.ca/content/185/15/E731/F2)

Figure 2: 
Flow of patients through the study. *Exclusion numbers add up to more than 617 because some patients had more than 1 exclusion criterion.

View this table:
[Table 1:](http://www.cmaj.ca/content/185/15/E731/T1)

Table 1: 
Characteristics of enrolled patients and participating physicians at intervention and control sites

During phases 2 and 3, use of the ankle rule was recorded in 68.5% of patients (430 of 628 patients enrolled during phases 2 and 3 at intervention sites). The physician followed the rule recommendations for 350 (81.4%) of these patients. The reported reasons for nonadherence in the remaining 80 were as follows: fear of missing a significant fracture (25 [31.3%]), physician preferred the Ottawa Ankle Rules (11 [13.8%]), family wanted a radiograph (8 [10.0%]), patient was difficult to examine (6 [7.5%]) and no reason documented (30 [37.5%]). If physicians had fully complied with the rule, 49.5% of radiography procedures could have been avoided (i.e., in 213 patients who met the Low Risk Ankle Rule criteria based on age and physical examination). Of the 566 children who received a diagnosis of a low-risk ankle injury at intervention sites, 532 (94.0%) received functional treatment based on symptoms and were advised to follow up as needed with a primary care physician.

There was a 21.9% reduction in weekly ankle radiography at the intervention sites from phase 1 to phase 2 compared with control sites (Table 2). The reduced rate of radiography was sustained in phase 3 at the intervention sites. There were no significant differences in the effect of the ankle rule on rates of ankle radiography between hospital types (Pediatric v. General, *p* = 0.1; Pediatric v. Community, *p* = 0.7; Community v. General, *p* = 0.2). The implementation of the ankle rule did not significantly change length of stay, nor were there any significant differences in patient or physician satisfaction, missed significant fractures or frequency of follow-up radiographs. There were no significant temporal trends during phase 1 for the study outcomes. The impact of the ankle rule by specific hospital type is detailed in Appendix 1 (available at [www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.122050/-/DC1](http://www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.122050/-/DC1)).

View this table:
[Table 2:](http://www.cmaj.ca/content/185/15/E731/T2)

Table 2: 
Effect of the Low Risk Ankle Rule on ankle radiography, length of stay, patient and physician satisfaction, missed significant fractures and follow-up radiography

Of the 694 patients with documented physical examination data, 553 (79.7%) had an emergency department radiograph as the reference standard. For the remaining 141 patients, final determination of low- or high-risk injury could be made in 117 (83.0%) by telephone contact, follow-up imaging or clinic notes. Of these, 109 (93.2%) reported return to baseline activities without pain by day 28 postinjury, and the remaining 8 by week 12. Sensitivity, specificity, negative and positive predictive values and likelihood ratios are presented in Table 3. There was no statistically significant difference between sites in sensitivity (*p* = 0.4) or specificity (*p* = 0.6) of the ankle rule. In phase 1, had the ankle rule been applied, it might have missed 1 significant fracture. This was in a 13-year-old boy who received a diagnosis in the emergency department of a stable Salter–Harris type II fracture of the fibula, whose subsequent radiology report suggested an additional distal tibial avulsion fracture. However, in the absence of any callus in subsequent radiographs, this finding was felt to be an anatomical variant.

View this table:
[Table 3:](http://www.cmaj.ca/content/185/15/E731/T3)

Table 3: 
Performance of the Low Risk Ankle Rule at intervention sites

## Interpretation

This study showed that an active multimodal strategy to implement the Low Risk Ankle Rule in several different emergency settings led to a significant reduction of radiography in children who presented with acute ankle injuries, without an increase in missed clinically important fractures or a decrease in satisfaction among physicians or parents. The sensitivity of the rule was high, with preservation of adequate specificity to provide a clinical benefit from applying the rule.

With application of the ankle rule, the frequency of pediatric ankle radiography was reduced by about 22%. The effect of the ankle rule intervention strategy on rates of radiography was comparable across different emergency settings and was similar to that of the Ottawa Ankle Rules applied in adults.14,15,17 Further, we found a 98% sensitivity and a potential reduction of about 50% of radiography procedures, which is consistent with the previously reported performance of the Low Risk Ankle Rule in 2 validation studies4,22 and is in keeping with recommended standards for high-performing clinical prediction rules.23 Two of the intervention sites did not participate in previous validation studies of the ankle rule, which provides external validation of the rule’s high sensitivity. At intervention sites, clinicians successfully managed about 95% of the low-risk injuries with a symptomatic care management strategy, which supports prior research showing that this management strategy is effective and safe for low-risk ankle fractures.5,24

The Low Risk Ankle Rule did not achieve the full potential reduction in radiography reported in validation studies.4,25 However, our results are consistent with other research on implementation of clinical prediction rules in emergency departments, which has shown reductions in imaging of up to 28% but also increases by as much as 15%.11,12,14–16 Several factors may have lead to more frequent radiography in this study. We reported a lower specificity of the ankle rule compared with that in the original validation study.4 During the validation studies4,22 all patients received radiography, which gave physicians the confidence to document only the tenderness that they considered clinically significant. In this study, without the safety net of radiography, physicians may have documented tenderness more liberally out of fear of missing a high-risk injury, leading to more imaging of injuries considered low-risk and lowering the specificity of the rule. Also, some physicians reported a preference for the Ottawa Ankle Rules, which have a low specificity in children and thus frequently recommend imaging.4,25,26 Finally, family preference for radiography was reported as a driver to opt out of rule recommendations in some cases.

Our findings support application of knowledge translation strategies in pediatrics that have been previously successful in similar studies applied to adults.11,14,17 Further, we introduced a new computerized decision support system. Clinicians found the system easy to use and preferred it to the paper data collection form in almost 90% of cases. The decision support system obviated the need for memorizing the rule, given that rule recall has previously been found to be a source of error in rule application or to result in a lack of willingness to use the rule.27 Therefore, systems like this may facilitate the use of clinical prediction rules in a busy emergency department sufficiently to justify the costs of development and implementation.

### Limitations

A convenience sample of participating sites was selected and allocation was not randomized, which may have introduced bias in the treatment effect and performance of the ankle rule. Among children who had neither radiography nor clinical follow-up, we cannot exclude the possibility of a missed high-risk injury, although no missed high-risk injuries presented at the study institutions. Because of the sample size, the study may have failed to detect important differences in the secondary outcomes. During phase 1 at the intervention sites, physicians completed study forms, which may have led to an observer effect. Finally, because this study was carried out in Canada, one cannot assume these findings to be generalizable to jurisdictions where the medicolegal climate might have a greater influence on physicians’ behaviour.

### Conclusion and implications for practice

The implementation of the Low Risk Ankle Rule led to a significant decrease in imaging, with no associated increase in clinically important fractures being missed or decrease in patient or physician satisfaction. The ankle rule has potential broad applicability to emergency departments throughout most of the developed world, and widespread implementation of this rule could safely lead to reduction of unnecessary radiography in this radiosensitive population and a more efficient use of health care resources.

## Acknowledgements

The authors acknowledge the tireless and superb efforts of lead research coordinator Johanna Crudden, and the excellent efforts of each of the following site coordinators: Chantalle Clarkin, Deborah Emerton, Kaushilya Tharmakulasingam, John Paul Oliveria and Kristy Pedder. the authors also thank Zeehsanefatema Somji for data entry and Martin Pecaric from Contrail Consulting Services on the design of the computerized decision support system and database support.

## Footnotes

*   **Competing interests:** None declared.

*   This article has been peer reviewed.

*   **Contributors:** Kathy Boutis is the primary and responsible author. All authors contributed to the intellectual content of the study design, revised the article for important intellectual content and gave final approval of the submitted version. Paul Grootendorst and Andrew Willan provided biostatistical expertise and performed most of the statistical analyses. Kathy Boutis, Paul Babyn, Amy Plint, Robert Brison, Melissa Parker, Arun Sayal, Natalie Butler and Unni Narayanan provided content and logistical expertise throughout the trial and had critical roles in data acquisition and data quality review.

*   **Funding:** Canadian Institutes of Health Research (funding reference no. 94140) and the Physicians’ Services Incorporated Foundation funded this research. The study funders provided financial support for research operations and did not have any role in design of the study, patient recruitment, analysis or final version of the manuscript.

## References

1.  Census families by number of children at home. Ottawa (ON): Statistics Canada; 2006. Available: [www12.statcan.ca/english/census-recensement/2006/dp-pd/hlt/97-551/index.cfm?lang=E](http://www12.statcan.ca/english/census-recensement/2006/dp-pd/hlt/97-551/index.cfm?lang=E) (accessed 2007 Dec. 1).
    
    

2.  Child health USA 2002. Washington (DC): US Department of Health and Human Services; 2002. Available: [http://mchb.hrsa.gov/chusa02/main\_pages/page_11.htm](http://mchb.hrsa.gov/chusa02/main_pages/page_11.htm) (accessed 2007 Dec. 1).
    
    

3.  Hospital report: emergency department care: hospital report research collaboratives. Toronto (ON): Hospital Report Research Collaboratives; 2008.
    
    

4.  Boutis K, Komar L, Jaramillo D, et al. Sensitivity of a clinical examination to predict the need for radiography in children with ankle injuries: a prospective study. Lancet 2001;358:2118–21.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0140-6736(01)07218-X&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11784626&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000172905800010&link_type=ISI) 

5.  Boutis K, Willan AR, Babyn P, et al. Randomized, controlled trial of a removable brace versus casting in children with low risk ankle fractures. Pediatrics 2007;119:e1256–63.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTE6IjExOS82L2UxMjU2IjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE4NS8xNS9FNzMxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

6.  Boutis K, Constantine E, Schuh S, et al. Pediatric emergency physician opinions on ankle radiograph clinical decision rules. Acad Emerg Med 2010;17:709–17.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1553-2712.2010.00787.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20653584&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 

7.  Lang ES, Wyer PC, Haynes RB. Knowledge translation: closing the evidence-to-practice gap. Ann Emerg Med 2007;49:355–63.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.annemergmed.2006.08.022&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17084943&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000244637000020&link_type=ISI) 

8.  Grimshaw JM, Eccles MP, Lavis JN, et al. Knowledge translation of research findings. Implement Sci 2012;7:50.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/1748-5908-7-50&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=22651257&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 

9.  Grimshaw JM, Thomas RE, Maclennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8:1–72.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=14960258&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 

10. Stiell IG, Bennett C. Implementation of clinical decision rules in the emergency department. Acad Emerg Med 2007;14:955–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1197/j.aem.2007.06.039&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17923717&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000250990700009&link_type=ISI) 

11. Stiell IG, Clement CM, Grimshaw J, et al. Implementation of the Canadian C-Spine Rule: a prospective 12-centre cluster randomized trial. BMJ 2009;339:b4146.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1136/bmj.b4146&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=19875425&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 

12. Stiell IG, Clement CM, Grimshaw JM, et al. A prospective 12-centre cluster randomized trial to implement the Canadian CT Head Rule in emergency departments. CMAJ 2010;182:1527–32.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTgyLzE0LzE1MjciO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTg1LzE1L0U3MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

13. Stiell IG, Grimshaw J, Wells G, et al. A matched pair cluster design study protocol to evaluate implementation of the Canadian C-Spine rule in hospital emergency departments: phase III. Implement Sci 2007;2:4
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/1748-5908-2-4&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17288613&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 

14. Stiell IG, McKnight DR, Greenberg GH, et al. Implementation of the Ottawa ankle rules. JAMA 1994;271:827–32.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.1994.03510350037034&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8114236&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1994MZ49800028&link_type=ISI) 

15. Stiell I, Wells G. Multicentre trial to introduce the Ottawa ankle rules for the use of radiography in acute ankle injuries. BMJ 1995;311:594–7.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTEvNzAwNS81OTQiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTg1LzE1L0U3MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

16. Stiell IG, Wells G, Hoag RH, et al. Implementation of the Ottawa Knee Rule for the use of radiography in acute knee injuries. JAMA 1997;278:2075–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.1997.03550230051036&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9403421&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997YK98700036&link_type=ISI) 

17. Auleley GRRP, Giraudeau B, Kerboull L, et al. Implementation of the Ottawa ankle rules in France. A multicenter randomized controlled trial. JAMA 1997;277:1935–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.1997.03540480035035&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9200633&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997XF08700037&link_type=ISI) 

18. Davidson R, MacKinnon JG. Econometric theory and methods. New York (NY): Oxford University Press; 2004:556–9.
    
    

19. Goldman HH, Frank RG, Burnam MA, et al. Behavioral health insurance parity for federal employees. N Engl J Med 2006;354: 1378–86.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMsa053737&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16571881&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000236338000007&link_type=ISI) 

20. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston (MA): Houghton Mifflin Company; 2002:484–584.
    
    

21. Trivedi AN, Moloo H, Mor V, et al. Increased ambulatory care copayments and hospitalizations among the elderly. N Engl J Med 2010;362:320–8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMsa0904533&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20107218&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000273975000008&link_type=ISI) 

22. Boutis K, Schuh S. Prospective validation and head-to-head comparison of 3 ankle rules in a pediatric population [comment]. Ann Emerg Med 2010;55:391–2. Comment on: *Ann Emerg Med* 2009;54:534–40.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.annemergmed.2009.12.004&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20346842&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 

23. Maguire JL, Kulik DM, Laupacis A, et al. Clinical prediction rules for children: a systematic review. Pediatrics 2011;128:e666–77.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTA6IjEyOC8zL2U2NjYiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTg1LzE1L0U3MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

24. Barnett PL, Lee MH, Oh L, et al. Functional outcome after air-stirrup ankle brace or fiberglass backslab for pediatric low-risk ankle fractures: a randomized observer-blinded controlled trial. Pediatr Emerg Care 2012;28:745–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/PEC.0b013e318262491d&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=22858744&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 

25. Gravel J, Hedrei P, Grimard G, et al. Prospective validation and head-to-head comparison of 3 ankle rules in a pediatric population. Ann Emerg Med 2009;54:534–40.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.annemergmed.2009.06.507&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=19647341&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 

26. Dowling S, Spooner CH, Liang Y, et al. Accuracy of Ottawa Ankle Rules to exclude fractures of the ankle and midfoot in children: a meta-analysis. Acad Emerg Med 2009;16:277–87.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1553-2712.2008.00333.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=19187397&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000264953300001&link_type=ISI) 

27. Gravel J, Roy M, Carrière B. 44-55-66-PM, a mnemonic that improves retention of the Ottawa Ankle and Foot rules: a randomized controlled trial. Acad Emerg Med 2010;17:859–64.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1553-2712.2010.00731.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20670323&link_type=MED&atom=%2Fcmaj%2F185%2F15%2FE731.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000280449500010&link_type=ISI)