Abstract
BACKGROUND: We previously derived the Ottawa Subarachnoid Hemorrhage Rule to identify subarachnoid hemorrhage (SAH) in patients with acute headache. Our objective was to validate the rule in a new cohort of consecutive patients who visited an emergency department.
METHODS: We conducted a multicentre prospective cohort study at 6 university-affiliated tertiary-care hospital emergency departments in Canada from January 2010 to January 2014. We included alert, neurologically intact adult patients with a headache peaking within 1 hour of onset. Treating physicians in the emergency department explicitly scored the rule before investigations were started. We defined subarachnoid hemorrhage as detection of any of the following: subarachnoid blood visible upon computed tomography of the head (from the final report by the local radiologist); xanthochromia in the cerebrospinal fluid (by visual inspection); or the presence of erythrocytes (> 1 × 106/L) in the final tube of cerebrospinal fluid, with an aneurysm or arteriovenous malformation visible upon cerebral angiography. We calculated sensitivity and specificity of the Ottawa SAH Rule for detecting or ruling out subarachnoid hemorrhage.
RESULTS: Treating physicians enrolled 1153 of 1743 (66.2%) potentially eligible patients, including 67 with subarachnoid hemorrhage. The Ottawa SAH Rule had 100% sensitivity (95% confidence interval [CI] 94.6%–100%) with a specificity of 13.6% (95% CI 13.1%–15.8%), whereas neuroimaging rates remained similar (about 87%).
INTERPRETATION: We found that the Ottawa SAH Rule was sensitive for identifying subarachnoid hemorrhage in otherwise alert and neurologically intact patients. We believe that the Ottawa SAH Rule can be used to rule out this serious diagnosis, thereby decreasing the number of cases missed while constraining rates of neuroimaging.
Subarachnoid hemorrhage is one of the most serious causes of sudden headache, yet it accounts for only 1%–3% of headaches.1–4 Although the decision to order neuroimaging in a patient with decreased level of consciousness or new neurologic deficits is relatively obvious, half of all patients with subarachnoid hemorrhage are alert and neurologically intact at first presentation.5 Making a decison to send these patients for imaging tests is difficult, because timely diagnosis and treatment result in substantially better outcomes if subarachnoid hemorrhage is the cause of the headache.
However, the desire never to miss a subarachnoid hemorrhage contributes to escalating rates of neuroimaging and a dogmatic adherence to lumbar puncture if the scan result is negative, despite the very high sensitivity of computed tomography (CT).6,7 However, a population-based study involving patients admitted to hospital through emergency departments in Ontario with a most responsible diagnosis of nontraumatic subarachnoid hemorrhage found that over 5% of confirmed subarachnoid hemorrhages were missed at initial presentation, especially in smaller hospitals.8 Therefore, identifying which patients with headache require investigations to rule out this catastrophic diagnosis, and which do not, is a critical decision in emergency medicine.
We previously derived (n = 1999)9 and refined (n = 2131)10 the Ottawa Subarachnoid Hemorrhage (SAH) Rule (Figure 1) using 2 successive prospective cohorts of patients with headache who visited the emergency department. Clinical decision rules require prospective validation in a new patient cohort before their implementation and use in routine patient care.11–14 In this study, we assessed the accuracy, clinical acceptability and potential effect on rates of neuroimaging of the Ottawa SAH Rule in a new cohort of consecutive patients with acute headache who visited the emergency department.
The Ottawa Subarachnoid Hemorrhage (SAH) Rule. Used in alert patients > 15 yr of age with new acute severe nontraumatic headache that reaches maximum intensity within 1 h of onset. Not to be used in patients with new neurological deficits, previous aneurysms, SAH or brain tumors, or a history of similar headaches (≥ 3 episodes over ≥ 6 mo).10
Methods
Study population and setting
We conducted this prospective multicentre cohort study in the emergency departments of 6 university-affiliated Canadian tertiary-care hospitals from January 2010 to January 2014. The combined annual census for all sites was 365 000 visits with over 150 attending physicians certified in emergency medicine. Each hospital had previously participated in the derivation phase of the study.10 Consecutive adult patients (16 yr of age or older) with nontraumatic headache that had reached maximal intensity within 1 hour of onset were considered for enrollment. We excluded patients who had a Glasgow Coma Scale score less than 15/15, who had sustained direct head trauma in the previous 7 days, and who presented more than 14 days after onset of the headache.15 We also excluded patients with a history of 3 or more recurrent headaches of the same character and intensity as the presenting headache over a period of greater than 6 months (i.e., established recurrent headache syndrome); who were referred from another hospital with a confirmed subarachnoid hemorrhage; who returned for reassessment of the same headache if already investigated with both CT and lumbar puncture; who had papilledema on fundoscopic examination; who had new focal neurologic deficits (e.g., isolated cranial nerve palsies, limb weakness); or who had a previous diagnosis of cerebral aneurysm, subarachnoid hemorrhage, brain neoplasm, ventricular shunt or hydrocephalus. Any patient not known to meet the exclusion criteria but not enrolled by the treating physician was deemed “missed/potentially eligible.” Participants enrolled by physicians were informed that they might be contacted by telephone in follow-up, and verbal consent was obtained from these patients at the time of telephone contact.
Assessment
Patients were assessed by attending physicians certified in emergency medicine or residents in emergency medicine who were supervised by attending physicians. Physicians were oriented to the study and the standardized assessment form during a 1-hour presentation. After assessing a patient, but before making a request for imaging tests or analysis of cerebrospinal fluid, physicians were asked to explicitly score each element of the Ottawa SAH Rule as present or absent and to record other potentially clinical relevant findings on a paper study form (we collected the additional variables to obtain a better description of the cohort and to have on hand in case any further refinement of the rule was required). To assess sensibility, physicians answered 2 questions related to interpretation and use of the Rule: Are investigations indicated for this patient according to the decision rule? (yes/no); and How comfortable would you be in actually using the rule for this patient? (based on a 5-point Likert Scale from “very comfortable” to “very uncomfortable”). We provided training to nurses at each site, using a full-day orientation session and ongoing monthly feedback from the central study coordinator, to collect data forms, verify data, confirm patient eligibility and perform telephone follow-up at 1 and 6 months after the index visit to the emergency department when necessary. Research staff also reviewed censuses from the emergency department for potentially eligible patients not enrolled by the treating physician. Missed patients had the same clinical variables extracted from the medical record and manually recorded onto standardized study forms. All data forms for both physician-enrolled and missed eligible patients were reviewed centrally by a single trained nurse coordinator, who was blind to outcome, and compared with the medical record from the emergency department. This nurse coordinator assessed the agreement between the physician assessments of the rule (i.e., investigations indicated yes/no) as written on the data form and as extracted from the clinical documentation.
Outcome measures
Subarachnoid hemorrhage was defined by any one of the following: subarachnoid blood visible on unenhanced CT scan of the head (from the final report of the local radiologist); xanthochromia in the cerebrospinal fluid (by visual inspection); or the presence of erythrocytes (> 1 × 106/L) in the final tube of cerebrospinal fluid, with an aneurysm or arteriovenous malformation visible upon cerebral angiography. This outcome was established a priori by consensus of 5 emergency medicine physicians and 1 neurosurgeon, and has been used consistently in prior phases of the development of the Ottawa SAH Rule.3,10,16 Patients with a normal result both for CT scanning and lumbar puncture testing were classified as not having a subarachnoid hemorrhage.17
We could not require all patients to undergo CT scanning and lumbar puncture in this observational study. In addition, the local ethics boards only allowed us to contact patients who had been enrolled by physicians and, as a result, had been informed about the subsequent follow-up call. Therefore, only patients enrolled by physicians and who were discharged without both a CT scan and a normal lumbar puncture were eligible for telephone follow-up. These patients underwent a structured telephone interview at 1 and 6 months after the index visit to the emergency departmentto identify a subsequent subarachnoid hemorrhage. In the follow-up telephone call, patients were asked about physician visits, changes in diagnosis and subsequent testing with lumbar puncture or imaging. We had previously validated this follow-up tool for missed subarachnoid hemorrhage during the initial derivation phase of the study.16
All medical records of patients without both normal CT scans and lumbar punctures underwent explicit review at the end of the study to identify missed subarachnoid hemorrhages. A similar review process involving medical records identified all subsequent subarachnoid hemorrhages in earlier phases of this study.10,16 We also evaluated patients not known to be alive at least 6 months after their index visits against the provincial coroner’s records to identify any deaths compatible with subarachnoid hemorrhage.
Statistical analysis
We calculated the sensitivity, specificity and likelihood ratios with 95% confidence intervals (CIs) of the Ottawa SAH Rule for subarachnoid hemorrhage. Interobserver agreement between treating physician and the nurse coordinator of the study for the rule as a whole was measured using the Cohen κ statistic test for participants enrolled by physicians. Univariate analysis used a 2-sided t test for continuous variables and the Pearson χ2 test for categorical variables. Because we wanted the sensitivity to be close to 100% to be clinically acceptable, we planned to enroll about 1200 patients, including 75 with subarachnoid hemorrhage, based on a prevalence estimate of 6.5%.
Ethics approval
The first research ethics board approval was obtained from the Ottawa Health Science Network Research Ethics Board (OHSN-REB) without the need for written consent, with additional independent approval subsequently obtained from each participating site.
Results
We prospectively enrolled 1153 patients with acute headache, including 67 (5.8%) with subarachnoid hemorrhage. We identified another 590 patients (missed eligible) whose medical records were assessed by our research staff after the initial visit to the emergency department, including 33 (5.6%) with subarachnoid hemorrhage. There were 8 patients who could not be reached for telephone follow-up (Figure 2). These patients did not present again to the regional neurosurgical centre nor were they reported to the provincial coroner’s office within 6 months of enrollment.
Participant selection for the study. SAH = subarachnoid hemorrhage.
Table 1 shows participant characteristics. Characteristics of the participants considered missed eligible (mean age 45.4 yr, female sex 60.5%, arrived by ambulance 31.0%, had CT 89.8%, had lumbar puncture 38.3% and diagnosed with subarachnoid hemorrhage 5.6%) were similar to patients who were prospectively enrolled (mean age 43.9 yr, female sex 59.8%, arrived by ambulance 26.5%, had CT 87.1%, had lumbar puncture 39.2% and diagnosed with subarachnoid hemorrhage 5.8%). Patients with subarachnoid hemorrhage were more likely to be older; had headaches that rapidly peaked in intensity; had onset of headache with exertion; had arrived by ambulance; had lost consciousness; and had neck pain or stiffness, vomiting and elevated systolic blood pressure (Table 2).
Characteristics of participants with acute headache who presented to emergency departments in 6 university-affiliated hospitals in Canada
Results for univariate correlation of variables for participants with and without subarachnoid hemorrhage
We found that the Ottawa SAH Rule had a sensitivity of 100% (95% CI 94.6%–100%) with a specificity of 13.6% (95% CI 13.1%–15.8%) (Table 3). When we pooled patients in this new validation cohort with our previous validation/refinement patient cohort (n = 3874), the sensitivity of the rule was 100% (95% CI 98.4%–100%) and specificity was 15.9% (95% CI 14.8%–17.1%) for subarachnoid hemorrhage.10
Sensitivity, specificity and negative predictive value of the Ottawa Subarachnoid Hemorrhage Rule (n = 1153)
Physicians were “uncomfortable” or “very uncomfortable” using the rule in 9.0% of patients. The rule as scored by the physician and by the central coordinating centre had excellent interobserver agreement (κ 0.82, 95% CI 0.76–0.87). In 5.1% of patients, physicians assessed the Ottawa SAH Rule as negative while the coordinating centre scored the rule as positive (i.e., physician indicated that the patient was at low risk when they were at high risk); however, none of these patients had subarachnoid hemorrhage. Neck pain or stiffness was the variable with the most frequent disagreement (documented at least once in the medical record but scored as absent by the physician).
We assessed the potential effect of the Ottawa SAH Rule on neuroimaging rates compared with actual practice. The actual neuroimaging rate in this cohort was 88.0%, and another 1.0% underwent lumbar puncture without CT, for an overall investigation rate of 89.0%. If the Ottawa SAH Rule had been followed, the investigation rate would have been 84.3%.
Interpretation
We found that the Ottawa SAH Rule was sensitive for identifying subarachnoid hemorrhage (i.e., every case was identified when the rule was used). With the findings from this study and our earlier work we believe that the Ottawa SAH Rule is ready to use to evaluate which patients who are neurologically intact with a new rapidly peaking headache require investigation. This rule provides clinicians with a validated tool to help standardize which patients with headache do not require investigation to rule out subarachnoid hemorrhage. In our experience, careful application of the rule will identify patients with subarachnoid hemorrhage while avoiding unnecessary CT scanning and painful lumbar punctures. This rule may also help smaller hospitals decide which patients with headache are unlikely to benefit from transport to larger centres for imaging.
Overall, the sensitivity of the Ottawa SAH Rule is higher than a previous, population-based estimate in Ontario that 1 in 20 patients with subarachnoid hemorrhage are missed on their first visit to the emergency department, which increases to 1 in 16 in nonteaching hospitals and includes all grades of severity.8 Physicians in the emergency department, when surveyed, have stated that 99% sensitivity is acceptable for a subarachnoid hemorrhage rule, a threshold that acknowledges the pitfall of pursuing diagnostic perfection at all cost. The narrow confidence bands of sensitivity for the Ottawa SAH Rule bracket this extremely high threshold, and the point estimate exceeds it.18 This precision was achieved only with a sustained and rigorous prospective enrollment campaign at several hospitals over many years.
This study is comparable to a previous retrospective study involving 454 patients with acute headache that reported 9 patients with subarachnoid hemorrhage. This study found that the Ottawa SAH Rule had 100% sensitivity; however, it had a wide 95% CI (62.9%–100%).19
We designed and conducted this prospective validation study according to the strict methodological standards for clinical decision rule development.11–14 We carefully defined the outcome, subarachnoid hemorrhage, and it is clinically important. Patients were included according to strict eligibility criteria, rather than on the somewhat circular logic of having already been selected for CT scanning or lumbar puncture, or known to be disease positive. We enrolled or followed all potentially eligible patients even when missed by the treating physician. In addition to accuracy, we also evaluated other important measures including interobserver agreement, clinical acceptability and potential effect on clinical practice.
Limitations
We acknowledge some potential limitations to this study. There is no established gold standard definition of a positive subarachnoid hemorrhage. We recognize that some patients with small, nonaneurysmal subarachnoid hemorrhage who are often managed conservatively were classified as disease positive using our definition, and a more meaningful outcome might be restricted to patients with subarachnoid hemorrhage who undergo intervention. Nevertheless, we have used the same definition throughout and believe that our composite outcome is clinically important and relevant.20 As with any decision tool, physicians may not apply the rule correctly. This raises the issue of how best to put the rule into operation, especially in an era of burgeoning health informatics and decision support, as well as use by operators other than emergency medicine physicians. These important issues are often the objective of subsequent implementation and knowledge translation studies.
About 1 in 3 cases were missed and not enrolled by the treating physician. In these cases, the presence or absence of specific elements of the rule were ascertained based on a review of medical records by research personnel. In these cases, the rule appeared to perform almost identically. The baseline clinical characteristics of the 2 subgroups including prevalence of subarachnoid hemorrhage were similar, which suggests little selection bias.
Validation of clinical decision rules is critical before full clinical use. Ideally, this is done in new patients at different sites from the original derivation. In this study, we enrolled new patients, but we used sites that had participated in the derivation of the Ottawa SAH Rule. This was done for feasibility reasons. Using the same sites does not cause bias regarding rule performance; however, it may overestimate the ability of physicians to use the rule correctly. Finally, the Ottawa SAH Rule is only intended for identifying patients with subarachnoid hemorrhage. It is not a rule for excluding other types of headache.
Conclusion
We found that the Ottawa SAH Rule had excellent sensitivity for identifying subarachnoid hemorrhage in a new consecutive cohort of patients with acute headache. Patients who are neurologically intact with a new rapidly peaking headache and who lack each of the 6 elements of the rule do not need further investigation to rule out subarachnoid hemorrhage. Instead, other diagnoses should be considered and managed accordingly in these patients.
Acknowledgements
The authors thank the physicians who completed our data collection forms, and the nurses and clerks in the emergency departments at all of our study sites for their cooperation. We thank the Deputy Coroner and the Office of the Coroner of Ontario for their assistance in verifying outcomes. We also thank the following research personnel at the study hospitals: Juanita Wilzer, Natalie Bilodeau and Rebecca Briscoe (Ottawa Hospital — Civic and General Campuses, Ottawa, Ontario); Jane Reid, Vlad Latiu and Vi Ho (Kingston General Hospital, Kingston, Ontario); and Patricia Chabot and Marilyne Dufresne (Hôpital de L’Enfant-Jésus, Québec City, Quebec). We thank our colleagues, Katherine Madill, Sheryl Domingo, My-Linh Tran, Catherine Clement and Angela Marcantonio, at the Ottawa Hospital Research Institute for their assistance with this project.
Footnotes
Competing interests: Jeffrey Perry, Marco Sivilotti, Jane Sutherland, Howard Lesiuk, Corinne Hohl, Marcel Émond, Lisa Calder, Christian Vaillancourt, Venkatesh Thirganasambandamoorthy and Ian Stiell received grants from the Canadian Institutes of Health Research during the conduct of the study. Christian Vaillancourt also received a grant from the Heart and Stroke Foundation of Canada during the conduct of the study. No other competing interests were declared.
This article has been peer reviewed.
Contributors: Jeffrey Perry conceived the idea and prepared the manuscript. Jane Sutherland coordinated the study and contributed to the writing of the manuscript. George Wells provided statistical assistance and revised the manuscript. Marco Sivilotti, Marcel Émond, Corinne Hohl, Lisa Calder, Christian Vaillancourt, Venkatesh Thiruganasambandamoorthy, Howard Lesiuk and Ian Stiell assisted with study design. All of the authors revised the manuscript critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.
Disclaimer: George Wells is a biostatistical consultant for CMAJ and was not involved in the editorial decision-making process for this article.
Funding: Jeffrey Perry was previously supported by a New Investigator Award from the Canadian Institutes of Health Research (CIHR). Marco Sivilotti was supported by a Clinician Scientist Development Award from the Southeastern Ontario Academic Medical Organization. Ian Stiell is a Distinguished Professor and holds a University Health Research Chair at the University of Ottawa. Corinne Hohl was supported by a Mentored Clinician Scientist Award from the Vancouver Coastal Health Research Institute and a New Investigator Award from CIHR during the study period.
- Accepted June 12, 2017.
References
In this issue
Article tools
Jump to section
Related Articles
Cited By...
- BET 1: The Ottawa Subarachnoid Hemorrhage Clinical Decision Rule shows high sensitivity but limited evidence of patient benefit
- Subarachnoid haemorrhage rules
- Response
- External validation of a clinical decision rule and neuroimaging rule-out strategy for exclusion of subarachnoid haemorrhage in the emergency department: A prospective observational cohort study
- Subarachnoid haemorrhage rules in the decision for acute CT of the head: external validation in a UK cohort
- Validation of the Ottawa Subarachnoid Hemorrhage Rule in patients with acute headache
More in this TOC Section
Similar Articles
