Abstract
Background: The San Francisco Syncope Rule has been proposed as a clinical decision rule for risk stratification of patients presenting to the emergency department with syncope. It has been validated across various populations and settings. We undertook a systematic review of its accuracy in predicting short-term serious outcomes.
Methods: We identified studies by means of systematic searches in seven electronic databases from inception to January 2011. We extracted study data in duplicate and used a bivariate random-effects model to assess the predictive accuracy and test characteristics.
Results: We included 12 studies with a total of 5316 patients, of whom 596 (11%) experienced a serious outcome. The prevalence of serious outcomes across the studies varied between 5% and 26%. The pooled estimate of sensitivity of the San Francisco Syncope Rule was 0.87 (95% confidence interval [CI] 0.79–0.93), and the pooled estimate of specificity was 0.52 (95% CI 0.43–0.62). There was substantial between-study heterogeneity (resulting in a 95% prediction interval for sensitivity of 0.55–0.98). The probability of a serious outcome given a negative score with the San Francisco Syncope Rule was 5% or lower, and the probability was 2% or lower when the rule was applied only to patients for whom no cause of syncope was identified after initial evaluation in the emergency department. The most common cause of false-negative classification for a serious outcome was cardiac arrhythmia.
Interpretation: The San Francisco Syncope Rule should be applied only for patients in whom no cause of syncope is evident after initial evaluation in the emergency department. Consideration of all available electrocardiograms, as well as arrhythmia monitoring, should be included in application of the San Francisco Syncope Rule. Between-study heterogeneity was likely due to inconsistent classification of arrhythmia.
See related commentary by Parry at www.cmaj.ca/lookup/doi/10.1503/cmaj.111529
Syncope is defined as sudden, transient loss of consciousness with the inability to maintain postural tone, followed by spontaneous recovery and return to pre-existing neurologic function.1–5 It represents a common clinical problem, accounting for 1%–3% of visits to the emergency department and up to 6% of admissions to acute care hospitals.6,7
Assessment of syncope in patients presenting to the emergency department is challenging because of the heterogeneity of underlying pathophysiologic processes and diseases. Although many underlying causes of syncope are benign, others are associated with substantial morbidity or mortality, including cardiac arrhythmia, myocardial infarction, pulmonary embolism and occult hemorrhage.4,8–10 Consequently, a considerable proportion of patients with benign causes of syncope are admitted for inpatient evaluation.11,12 Therefore, risk stratification that allows for the safe discharge of patients at low risk of a serious outcome is important for efficient management of patients in emergency departments and for reduction of costs associated with unnecessary diagnostic workup.12,13
In recent years, various prediction rules based on the probability of an adverse outcome after an episode of syncope have been proposed.3,14–16 However, the San Francisco Syncope Rule, derived by Quinn and colleagues in 2004,3 is the only prediction rule for serious outcomes that has been validated in a variety of populations and settings. This simple, five-step clinical decision rule is intended to identify patients at low risk of short-term serious outcomes3,17 (Box 1).
San Francisco Syncope Rule3
Aim
Prediction of short-term (within 30 days) serious outcomes in patients presenting to the emergency department with syncope.
Definitions
Syncope: Transient loss of consciousness with return to baseline neurologic function. Trauma-associated and alcohol- or drug-related loss of consciousness excluded, as is definite seizure or altered mental status.
Serious outcome: Death, myocardial infarction, arrhythmia, pulmonary embolism, stroke, subarachnoid hemorrhage, significant hemorrhage or any condition causing or likely to cause a return visit to the emergency department and admission to hospital for a related event.
Selection of predictors in multivariable analysis: Fifty predictor variables were evaluated for significant associations with a serious outcome and combined to create a minimal set of predictors that are highly sensitive and specific for prediction of a serious outcome.
Clinical decision rule
Five risk factors, indicated by the mnemonic “CHESS,” were identified to predict patients at high risk of a serious outcome:
-
C – History of congestive heart failure
-
H – Hematocrit < 30%
-
E – Abnormal findings on 12-lead ECG or cardiac monitoring17 (new changes or nonsinus rhythm)
-
S – History of shortness of breath
-
S – Systolic blood pressure < 90 mm Hg at triage
Note: ECG = electrocardiogram.
The aim of this study was to conduct a systematic review and meta-analysis of the accuracy of the San Francisco Syncope Rule in predicting short-term serious outcome for patients presenting to the emergency department with syncope.
Methods
Search strategy and data sources
We performed a systematic search of electronic databases (specifically, MEDLINE, Embase, MedPilot, CINAHL, the Cochrane Central Register of Controlled Trials, ClinicalTrials.gov and Web of Science) from their respective inception dates to Jan. 17, 2011. We applied no language restrictions. Our search was based on the combination of the following terms, with syncope being expanded to match synonyms, subterms or derivatives (e.g., faint, transient loss of consciousness): sfsr OR (san AND francisco AND syncope). In addition, we reviewed the bibliographies of reviews on risk stratification for syncope and entries in the UpToDate database (UpToDate, Inc., Waltham, Massachusetts; www.uptodate.com) on management of syncope in the emergency department.
Study selection
Two reviewers (R.T.S. and C.H.N.) independently screened all retrieved citations. References that were judged ineligible by both reviewers on the basis of a review of titles or abstracts were not assessed any further. We obtained the full text of each potentially eligible article. All primary studies assessing the accuracy of the San Francisco Syncope Rule to predict the defined combined end point of a short-term serious outcome (Box 1) qualified for inclusion.
Assessment of quality and validity and extraction of data
We assessed the methodologic quality of all included studies with the Quality Assessment of Diagnostic Accuracy Studies tool, a checklist of 14 items for assessing the quality of diagnostic accuracy studies and evaluating possible sources of bias.18,19 In addition, we developed an extended checklist on the basis of previous methodologic recommendations20,21 to explore possible biases in validation studies in relation to the derivation study for the San Francisco Syncope Rule. Two reviewers (R.T.S. and C.H.N.) independently extracted predefined characteristics from each study and abstracted the data to produce the 2 × 2 contingency tables needed for the statistical analysis, described below. We defined studies as prospective or retrospective according to whether the method of data collection and the end points were defined before the start of patient enrolment.22
Statistical analysis
We used a bivariate random-effects model for diagnostic and prognostic meta-analysis to calculate overall estimates of sensitivity and 1 –specificity.23 This approach allowed us to account for study size and to incorporate the negative correlation of the between-study sensitivities and specificities. Random effects incorporate latent threshold differences between studies but also allow for heterogeneity beyond chance as a consequence of differences in design and quality of the studies included. We conducted a priori specified subgroup analyses to investigate potential sources of heterogeneity among the included studies (for further information, see Appendix 1, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.101326/-/DC1).
We prepared a summary receiver operating characteristic plane by plotting the sensitivity (true positive rate) of the San Francisco Syncope Rule on the y axis against 1 – specificity (the false positive rate) on the x axis. Within the summary receiver operating characteristic plane, we provided the pooled estimate, sampled results of a Markov chain Monte Carlo simulation and individual study estimates weighted according to sample size.24 To illustrate the precision of the pooled estimates, we plotted the 95% confidence region. This region contains the true underlying pair of sensitivity and specificity for the population with 95% certainty. On the basis of the random-effects model, we also calculated 95% prediction intervals and plotted the corresponding 95% prediction area. The prediction area corresponds to the variability or heterogeneity of the estimates of test accuracy of the individual studies. It illustrates the range that contains 95% of all pairs of individual study characteristics.
Results
Studies included in analysis
We identified 459 unique, potentially relevant articles in our initial searches. Twelve of the studies, involving a total of 5684 patients, met our inclusion criteria (Figure 1). Of the patients enrolled, 368 were lost to follow-up, leaving a total of 5316 patients for inclusion in our analysis. Data from the individual studies were sufficient to construct contingency tables for all studies. The general characteristics of the included studies are summarized in Table 1. Two early validation studies25,26 were published only as abstracts, one study30 included only patients aged 65 or older, and one study was a pilot study.1 A chronology of the included studies, as well as reviews and communications related to the San Francisco Syncope Rule, can be found in Appendix 2 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.101326/-/DC1). All included studies were published in English between the years 20043 and 2010.16,31,32
The 12 included studies reported a serious outcome in a total of 596 cases (11%). The prevalence of serious outcomes in the individual studies ranged from 5% to 26% (Table 1). The total number of patients with a serious outcome in whom the classification was falsely negative was 96 (16%) (range 0 [0%] to 24 [48%]). A detailed overview of cases with a serious outcome that were classified incorrectly appears in Table 2.
Nine studies1,3,16,25,27–31 reported the results of the San Francisco Syncope Rule for all patients who presented to the emergency department with syncope. Three of these studies27,29,31 also reported results for patients for whom no cause of syncope was evident during the initial assessment in the emergency department. Two other studies4,32 excluded all patients in whom a cause of syncope was evident in the emergency department, and one study26 included only patients who were admitted to hospital. None of the validation studies applied the San Francisco Syncope Rule as originally stated in the derivation study (Appendix 3, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.101326/-/DC1).
Quality of studies
Nine studies1,3,4,16,25,27–29,32 were prospective and three studies26,30,31 were retrospective. The number of patients enrolled but lost to follow-up ranged from 0 (0%) to 122 (19%). Four of the studies27,29–31 used sensitivity analyses to assess the effect of attrition on performance of the San Francisco Syncope Rule. Three studies27,29,31 found no substantial differences in estimates of test performance, and one30 reported that the sensitivity varied between 34% to 90% according to assumptions about patients lost to follow-up. Three studies1.16,32 applied a different outcome definition than the original derivation study. The representativeness of the patient samples for the population, as defined in the original derivation study, was unclear in three studies25,28,31 and was not given in two studies26,30 The reference standard was judged inadequate in two studies26,32 and was unclear in another study.25 Three studies1,26,30 did not report on blinded application of the San Francisco Syncope Rule (i.e., blinding for outcome results), and five1,25,26,30,32 did not report on blinding of outcome assessment (i.e., blinding for prognostic factors). The remaining items in the Quality Assessment of Diagnostic Accuracy Studies tool were fulfilled (Table 3; see also Appendix 4, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.101326/-/DC1).
Pooled analyses and between-study heterogeneity
The pooled estimate of overall sensitivity in our bivariate random-effects model was 0.87 (95% confidence interval [CI] 0.79–0.93), and the pooled estimate of overall specificity was 0.52 (95% CI 0.43–0.62) (Figure 2). Calibration of the San Francisco Syncope Rule toward sensitivity resulted in a pooled negative predictive value of 0.97 (95% CI 0.95–0.99) (Table 4). We found substantial between-study heterogeneity for sensitivity, which resulted in a 95% prediction interval of 0.55–0.98. The between-study heterogeneity was somewhat less for specificity, which resulted in a 95% prediction interval of 0.22–0.81. There was no evidence of a relation between prevalence of a serious outcome and diagnostic performance of the San Francisco Syncope Rule across studies. The results of sensitivity analyses for various subgroups are displayed in Table 4.
Interpretation
We summarized the accuracy of the San Francisco Syncope Rule in predicting a serious outcome in patients presenting to the emergency department with syncope. In our analysis, the modelled probability of a serious outcome in patients with a negative score with this rule was 5% or lower (95% CI 1%–5%), based on prior probabilities of a serious outcome between 5% and 26% and a pooled sensitivity estimate of 87% and using the upper bound of the 95% CI for the estimate (i.e., a conservative interpretation of the data). Because of between-study heterogeneity, these summary results should be interpreted with caution.
Originally, the San Francisco Syncope Rule was derived in a prospective study involving all patients presenting to the emergency department with syncope, and it was thus conceptualized as a tool for initial risk stratification.3 Over the past few years, investigators have modified the original San Francisco Syncope Rule, as well as its role in evaluating syncope.4,17 These modifications may have been an important cause of the heterogeneity of predictive accuracy in our systematic review. In their validation study, Quinn and colleagues4 focused on patients in whom a cause of syncope was not evident in the emergency department. They emphasized that the San Francisco Syncope Rule should be part of a thorough medical history and physical examination and should not prevent investigation for other associated symptoms.34–36 We identified three studies27,29,31 that directly compared the “front-end” application of the rule for all patients with application of the rule only for patients without an evident cause of syncope. One of these studies31 reported better performance of the rule, whereas the other two studies27,29 reported poorer performance of the rule in patients without an evident cause of syncope.
The management of syncope in the emergency department varies greatly among physicians and centres,37 although several evidence-based recommendations have been published in recent years.38–40 Therefore, the population of patients without an evident cause of syncope may vary considerably according to the practices of local emergency departments, and judgment for generalization of the San Francisco Syncope Rule in this population remains uncertain. Nevertheless, restriction of application of the rule to patients without an evident cause of syncope after initial evaluation appears to reduce the prevalence of serious outcomes. According to Bayes’ rule, the post-test probability will be reduced accordingly, given that patient selection does not substantially affect performance of the San Francisco Syncope Rule. Our subgroup analyses confirmed a reduction in the modelled probability of a serious outcome, given a negative score, from 5% or lower to 2% or lower (using the upper bound of the 95% confidence interval), when the San Francisco Syncope Rule was restricted to patients without an evident cause of syncope.
We identified cardiac arrhythmia as an important issue and a potential source of heterogeneity. In their original publications, the authors of the San Francisco Syncope Rule specified that the result was positive if any nonsinus rhythm or any new changes were present on the electrocardiogram (ECG).3,4 They recently clarified that the ECG criterion was not to be limited to a single evaluation but should include any available ECGs, as well as cardiac monitoring in the emergency department.17 It is likely that in studies performed after the original derivation study, cardiac arrhythmia registered during monitoring in the emergency department was inconsistently classified as a missed serious outcome.
Schladenhaufen and associates30 reported 17 cases of arrhythmia out of 23 serious outcomes missed in a retrospective cohort of patients aged 65 years or older. Birnbaum and coworkers29,41 applied the San Francisco Syncope Rule to a minority population of African American and Hispanic individuals with a high admission rate (86%). They reported a sensitivity of only 74% (95% CI 61%–84%), missing a total of 16 serious outcomes, including 8 cases of arrhythmia. In contrast, Quinn and colleagues4 reported 23 cases of arrhythmia diagnosed after the initial visit to the emergency department, but they reported only one missed serious outcome. Finally, in the study by Thiruganasambandamoorthy and coauthors,31 four of the five cases with a missed serious outcome involved arrhythmia (for one of which the serious outcome was “return for admission to hospital”). With inclusion of abnormalities detected during cardiac monitoring in the emergency department, three of these arrhythmias would have been classified as positive with the San Francisco Syncope Rule (i.e., true positive), which would have improved the sensitivity from 0.90 to 0.96 (95% CI 0.87–0.99). However, which patients with syncope should undergo routine ECG monitoring in the emergency department and the duration of the monitoring period have been insufficiently specified for the San Francisco Syncope Rule, and further investigation is required.17
Validation studies have reported single cases of stroke, myocardial infarction, subarachnoid hemorrhage and other outcomes that were missed by the San Francisco Syncope Rule. Fischer and coworkers25 reported six strokes as missed serious outcomes. However, their study was published only as an abstract, and it is questionable whether the inclusion criteria of the San Francisco Syncope Rule were correctly applied and whether patients whose neurologic function did not return to baseline were excluded (see Box 1).
None of the validation studies applied the San Francisco Syncope Rule as originally outlined in the derivation study (Appendix 3, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.101326/-/DC1). Important factors other than arrhythmia that may explain the variable performance of the San Francisco Syncope Rule in these validation studies are differences in the definition of an abnormal ECG,42,43 the professional role of the person in charge of interpreting the ECG, the definition of a serious outcome and the study design (prospective or retrospective). However, the small number of studies and the multitude of potentially confounding factors prevent a meaningful sensitivity analysis of the influence of these factors on accuracy of the rule.
Extension of the rule with predictors selected in recently derived scores may enhance its predictive accuracy. Levels of natriuretic peptide in the brain, oxygen saturation and chest pain are possible candidates that were relevant in the recently derived ROSE (Risk stratification Of Syncope in the Emergency department) rule.16 A list of predictors employed in other rules is presented in Appendix 5 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.101326/-/DC1).
In October 2010, Serrano and coworkers33 published a general systematic review and meta-analysis on clinical decision rules for syncope in the emergency department. They reported similar pooled accuracy estimates for the San Francisco Syncope Rule, with sensitivity 87% (95% CI 79%–92%) and specificity 48% (95% CI 38%–59%). These authors explored heterogeneity between studies through subgroup analysis related to study design, outcome period, definition of ECG, ECG determination (by caring physician v. others) and patient selection (stable v. unstable). They found no statistically significant explanation for the variability in performance of the San Francisco Syncope Rule.
We conducted this systematic review according to a prespecified protocol, with a strict focus on studies employing the San Francisco Syncope Rule. We identified relevant studies, reviews and correspondence using a comprehensive literature search in seven databases. In comparison to the the systematic review of Serrano and coworkers,33 we found three additional studies.1,25,26 However, the possibility of publication bias can never be fully excluded and is difficult to assess for our particular research question, because the use of funnel plots may be misleading in diagnostic or prognostic accuracy studies.44
We assessed possible biases with the Quality Assessment of Diagnostic Accuracy Studies instrument individually for each study and elaborated on inconsistent use of the San Francisco Syncope Rule in the process of evaluating syncope. Our critical review of individual study differences further elucidates between-study heterogeneity and provides important supplementary information that will allow clinicians to better understand how to optimally apply the San Francisco Syncope Rule. Furthermore, we employed modern statistical methods to provide easily interpretable negative and positive post-test probabilities.45,46 In addition, we have provided a comprehensive overview of the components of serious outcomes that were missed by the San Francisco Syncope Rule.
Limitations
The degree of between-study heterogeneity for both sensitivity and specificity in the included studies was an important limitation of this analysis. Although the general quality of the included studies was good, there were several potential confounding factors with high interrelation in individual studies. Because of the small number of studies identified, subgroup sensitivity analyses were of only limited value, and we were unable to attribute between-study variability to any specific factor.
Conclusions
The San Francisco Syncope Rule is an important step toward reliable risk stratification of patients presenting to the emergency department with syncope. The strength of the rule is its thorough derivation process and the number of validation studies that have been performed. As such, it is currently the most thoroughly investigated prediction rule for serious outcome and should be used as the basis for further developments in syncope-related risk prediction. However, validation studies have shown inconsistent results, which may be related to different classifications of arrhythmia captured on monitoring during the initial patient work-up in the emergency department and not apparent on the initial ECG.
As recommended by the original authors, the San Francisco Syncope Rule should be used during initial evaluation of syncope at the discretion of the treating emergency physician and should include all available ECGs, as well as results of cardiac monitoring. In patients without an evident cause of syncope, a negative score is associated with a posterior probability of a serious outcome of 2% or lower, which should allow for safe discharge of the patient.
Further research is needed to determine whether the accuracy of the San Francisco Syncope Rule improves with routine inclusion of cardiac monitoring or if additional factors should be employed, especially for elderly patients. Recently developed rules or biomarkers may provide suitable criteria for refinement or extension of the San Francisco Syncope Rule.
Footnotes
-
Competing interests: Roland Bingisser has served as a board member for Sanofi Aventis and as a consulant for Philipps; has received grants from ThermoFisher and speaker fees from AstraZeneca; and has received payment for patents from Brahms and for development of educational presentations from Pfizer. No competing interests declared by Ramon T. Saccilotto, Christian H. Nickel, Heiner C. Bucher, Ewout W. Steyerberg or Michael T. Koller.
-
This article has been peer reviewed.
-
Contributors: Ramon Saccilotto, Christian Nickel, Michael Koller, Rolland Bingisser and Heiner Bucher contributed to the conception and design of the review. Ramon Saccilotto and Michael Koller designed the study searches. Christian Nickel and Ramon Saccilotto reviewed all of the publications that were retrieved and extracted the data. Michael Koller, Ramon Saccilotto and Ewout Steyerberg designed and conducted the statistical analyses. Ramon Saccilotto, Michael Koller and Heiner Bucher drafted the manuscript. All coauthors contributed to interpreting the data and revising the manuscript for important intellectual content. All authors provided final approval of the article. All authors certify that they have participated sufficiently in the work, believe in its overall validity and take public responsibility for appropriate portions of its content.
-
Funding: This study was funded by unrestricted grants to Ramon T. Saccilotto, Heiner C. Bucher and Michael T. Koller from Santésuisse and the Gottfried and Julia Bangerter-Rhyner-Foundation.