Diagnostic accuracy of the TIMI risk score in patients with chest pain in the emergency department: a meta-analysis

Eligibility criteria

We included studies that prospectively validated the TIMI risk score in patients in the emergency department with chest pain. We excluded retrospective studies, case reports, studies not based on original research and studies not involving patients in the emergency department (e.g., hospital ward, intensive care unit).

Search strategy

An expert librarian (P.J.E.) designed a comprehensive search strategy with input from the clinical lead (E.P.H.). Our electronic search included five databases (MEDLINE, EMBASE, Cochrane Database of Systematic Reviews, Web of Science and Scopus; Appendix 1, available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). We applied no language restrictions to the search strategy. We also hand-searched conference proceedings from 2007 to 2009 from the Society for Academic Emergency Medicine, the Canadian Association of Emergency Physicians and the American College of Emergency Physicians. We reviewed the reference lists of eligible articles and consulted with content experts (J.E.H., I.G.S.) to identify additional published reports.

Study selection

Two investigators (E.P.H., D.A.) independently screened the titles and abstracts of all identified records (phase I). If either reviewer believed that the study might be eligible, we obtained the full report. The same two investigators then independently assessed the eligibility of each full report (phase II). We used the Cohen κ to measure the chance-corrected agreement between reviewers for each phase of study selection. Any disagreements were resolved by consensus.

Quality assessment

Two reviewers (D.A., S.C.) independently assessed the methodological quality of each eligible article. Because there is no single quality assessment tool that has been designed to assess the methodological quality of studies of risk scores, we selected criteria from assessment tools 10^–12 that, based on a consensus among investigators, most closely indicated study quality. We selected the following seven criteria: patients selected in an unbiased fashion (consecutive or random sample); study sample representative of a wide spectrum of the severity of disease; predictor variables assessed without knowledge of the outcome; outcome assessed without knowledge of the predictor variables; outcomes accurately defined (particularly, acute myocardial infarction); explicit interpretation of the risk score by clinicians in practice without knowledge of the outcome; and adequate follow-up (arbitrarily defined as a lost-to-follow-up rate of < 10%). We assessed the degree of interobserver agreement for quality assessment using the Cohen κ. Any disagreements were resolved by consensus.

Data extraction and outcomes

Using a standardized data extraction sheet, two authors (D.A., S.C.) independently extracted data from each included report. Because most reports did not differentiate between patients with TIMI risk scores from 5 to 7, we classified patients with TIMI risk scores ≥ 5 into a single risk stratum. If the data were not available in the original report or were unclear, we contacted the corresponding author for clarification. Clinical outcomes were acute myocardial infarction, coronary revascularization or death from any cause. Disagreements were resolved by consensus.

Data synthesis and analysis

We present all continuous data as either means with standard deviations (SD) or medians with interquartile ranges as reported in the primary study. Categorical data are presented as per cent frequency of occurrence. We performed meta-regression to determine whether a linear relation exists between TIMI risk score (independent variable) and the logit event rate (dependent variable) using Comprehensive Meta-Analysis version 2 (Biostat Inc., Englewood, NJ). We assessed the prognostic performance of the TIMI risk score at potential decision thresholds using Meta-DiSc software (Unit of Clinical Biostatistics and the Ramon y Cajal Hospital).

We used the I² statistic to quantify statistical heterogeneity between studies. We hypothesized that statistical heterogeneity identified by the I² statistic could be explained by differences in the prevalence of cardiac events between studies. To test this hypothesis, we arbitrarily defined low prevalence as less than 10% and conducted an a priori subgroup analysis of studies with low (< 10%) versus high (≥ 10%) prevalence using a test of statistical interaction as defined by Altman. 13 Estimates from each subgroup are analyzed on a log scale. The test is defined as the ratio of the difference in the log estimates to the standard error of this difference. The associated p value is derived from the normal distribution.

Study selection

Figure 1 shows the study selection process. The initial search strategy identified 383 records. After removing 68 duplicate records, 315 records remained for screening in phase I. We identified 27 potentially eligible records for further review based on review of the titles and abstracts (κ = 0.87, 95% CI 0.76–0.97). Review of the full articles in phase II identified 10 that met our inclusion criteria (κ = 0.78, 95% CI 0.54–1.00). All 10 studies were included in the qualitative analysis. We were able to contact authors for missing data and verification of data extraction for 8 of the 10 included studies. We were unable to obtain TIMI risk score data in two studies, 14^,15 leaving eight studies for the quantitative synthesis.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1: Flow diagram of the study selection process. Note: ED = emergency department.

Characteristics of the included studies

The 10 studies included in the review enrolled 17 265 emergency department patients with chest pain from 5 countries (Table 1). 16^–23 Three studies were conducted in the United States, two in the United Kingdom, two in Spain, two in Italy and one in Canada. The mean age ranged from 53.6 to 71.3 years. The proportion of males to females enrolled in each study was approximately equal.

Quality assessment

Assessment of study quality for all 10 studies had an inter-observer agreement of 92.8% (κ = 0.85, 95% CI 0.71–0.98). Although the predictor variables were assessed without knowledge of the outcome in all studies, only one study 16 explicitly reported blinding outcome assessors to knowledge of the predictor variables (Appendix 2, available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). Two studies did not report how acute myocardial infarction was defined. 14^,15 Physicians explicitly interpreted the TIMI risk score in practice without knowledge of the outcome in three studies. 17^–19 All studies that reported the fidelity of patient follow-up had lost-to-follow-up rates of less than 10%.

Heterogeneity

Cardiac troponin or creatine kinase–muscle and brain (CK-MB) or both were used as cardiac biomarkers in all eight studies included in the quantitative analysis. However, only three studies 16^,20^,21 reported the specific assay used (including the manufacturer, platform, 99th percentile reference limit and coefficient of variation) as recommended by standardized reporting guidelines. 24 This makes it difficult to assess potential heterogeneity between studies because of the different cardiac biomarker thresholds used to define acute myocardial infarction. However, all studies validated the TIMI risk score in emergency department patients and assessed the same predictor variables and the same composite outcome. Thirty-day outcome data were available for all studies included in the quantitative synthesis except one 17 in which outcomes were assessed at the time of discharge from hospital or emergency department. The prevalence of cardiac events varied from 3.7% to 25.5% between studies.

Classification characteristics and diagnostic performance

The number of patients and the number with cardiac events in each risk stratum of the TIMI risk score for the eight studies included in the quantitative synthesis are shown in Appendix 3 (available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). The prevalence of cardiac events in patients with a TIMI risk score of zero was 1.8%, and the prevalence in patients with a score of one was 4.0%. Although the total number of patients in each risk stratum progressively decreased with each successive increase in TIMI risk score, the proportion with adverse outcomes progressively increased up to a score of 5 or greater.

The results of the meta-regression of the TIMI risk score on the logit event rate are shown in Figure 2. The regression coefficient was significant (p < 0.001), indicating a strong linear relation between TIMI risk score and cumulative incidence of cardiac events.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2: Meta regression of TIMI risk scores on the logit event rate. Each circle represents data from an individual study, and the size of the circle represents the size of the cohort. There is a strong linear relation (p < 0.001) between the TIMI risk score and the cumulative incidence of cardiac events. The consistent dispersion of the data represents heterogeneity in the frequency of cardiac events between studies.

Table 2 shows the classification performance of the TIMI risk score at potential decision thresholds. At a cut-off of > 0, the sensitivity was 97.2% (95% CI 96.4–97.8; I² = 99.9%) and the specificity was 25.0% (95% CI 24.3–25.7; I² = 100.0%). The negative likelihood ratio was 0.11 (95% CI 0.09–0.15; I² = 83.3%). At a cut-off of > 1, the sensitivity decreased to 90.6% (95% CI 89.3–91.8; I² = 97.9%) and the specificity increased to 50.9% (95% CI 50.0–51.7; I² = 99.5%).

We also conducted a sensitivity analysis excluding the study by Pellicia and colleagues, 17 because this study reported in-hospital rather than 30-day outcomes. There were no substantial differences in the classification performance of the TIMI risk score when this study was excluded (data not shown).

Subgroup analyses

The test for interaction comparing studies with a low (< 10%) prevalence of cardiac events at a threshold of TIMI > 0 was significant (diagnostic odds ratio 5.58, 95% CI 4.21–7.39 v. (34.21, 95% CI 17.01–68.78; p < 0.001). The test for interaction at a threshold of TIMI > 1 was also significant (diagnostic odds ratio 5.12, 95% CI 4.27–6.15 v. 18.55, 95% CI 13.88–24.78; p < 0.001).

Strengths and limitations

The strengths of this study include a search strategy that involved five electronic databases, searching the bibliographies of the included articles and contact with content experts and authors of the included studies. This minimized, but did not exclude, the potential for publication bias. We also used sound methods in conducting the review, including assessment of interrater reliability for study selection and quality assessment and dual data extraction. Our findings also represent the performance of the TIMI risk score in unselected “real life” patients. These patients are generally at increased risk of adverse events compared with patients in randomized controlled trials. Thus, the estimates of risk provided in this review may be more relevant to general cardiology practices.

The study is limited by the relatively small number of studies included in the review and the statistical heterogeneity between studies, which limits the quality of the evidence and our ability to assess publication bias. Although the Global Registry of Acute Coronary Events (GRACE) risk score has been shown to have greater discriminative ability than the TIMI risk score for mortality in-hospital and at six months in patients with confirmed acute coronary syndromes, 25 few studies have assessed the performance of the GRACE score in emergency department patients with potential acute coronary syndromes. One study reported a sensitivity of 100% and a specificity of 13%, with data needed to calculate a complete GRACE score missing for 24% of patients. 22 The focused nature of the research question, however, enabled careful assessment and meta-analysis of studies with limited clinical heterogeneity.

Our review is also limited by the lack of reporting of the characteristics of the cardiac biomarker assays and the thresholds used to define acute myocardial infarction. All of the studies included in the meta-analysis, however, used either CK-MB or cardiac troponin as recommend by the Joint European Society of Cardiology, American College of Cardiology, American Heart Association and World Health Federation Task Force for the Redefinition of Myocardial Infarction. 26

Reporting bias may also have affected the results of this review. Despite repeated attempts to contact the primary authors to obtain data about TIMI risk scores, we were unable to obtain data for two studies. 14^,15 The large number of remaining patients included in the review and the strong linear relation between TIMI risk score and the cumulative incidence of cardiac events in the eight studies included in the quantitative synthesis suggests that the lack of data from these two studies may not substantially reduce the scientific value of this research.

External validity, clinical implications and future research

This review included 10 validation studies conducted in 5 countries, encompassing 17 265 patients. The broad sample of patients from which the statistical estimates were derived suggests a high degree of external validity of the findings. The 1.8% miss rate suggests that this degree of accuracy may be achieved using clinical, electrocardiography and laboratory data without need for urgent stress testing or complex rule-out protocols, which could result in important cost savings in practice settings without observation units. However, a miss rate of 1.8% may not be acceptable in all practice settings.

Conclusion

Our analysis supports the consistent performance of the TIMI risk score in the risk stratification of patients in the emergency department with potential acute coronary syndromes. For every 1000 patients in the lowest risk category (TIMI score of zero), 20 will experience a cardiac event within 30 days of the visit to the emergency department. Although this rate may render the TIMI risk score an unacceptable single reason to dismiss patients from the emergency department, this rate highlights the potential utility of this score as an adjunct to clinical acumen, as a source of information for patient decision aids and as a benchmark for researchers developing new prognostic and disposition tools.