Abstract
Background: Several clinical prediction rules for diagnosing group A streptococcal infection in children with pharyngitis are available. We aimed to compare the diagnostic accuracy of rules-based selective testing strategies in a prospective cohort of children with pharyngitis.
Methods: We identified clinical prediction rules through a systematic search of MEDLINE and Embase (1975–2014), which we then validated in a prospective cohort involving French children who presented with pharyngitis during a 1-year period (2010–2011). We diagnosed infection with group A streptococcus using two throat swabs: one obtained for a rapid antigen detection test (StreptAtest, Dectrapharm) and one obtained for culture (reference standard). We validated rules-based selective testing strategies as follows: low risk of group A streptococcal infection, no further testing or antibiotic therapy needed; intermediate risk of infection, rapid antigen detection for all patients and antibiotic therapy for those with a positive test result; and high risk of infection, empiric antibiotic treatment.
Results: We identified 8 clinical prediction rules, 6 of which could be prospectively validated. Sensitivity and specificity of rules-based selective testing strategies ranged from 66% (95% confidence interval [CI] 61–72) to 94% (95% CI 92–97) and from 40% (95% CI 35–45) to 88% (95% CI 85–91), respectively. Use of rapid antigen detection testing following the clinical prediction rule ranged from 24% (95% CI 21–27) to 86% (95% CI 84–89). None of the rules-based selective testing strategies achieved our diagnostic accuracy target (sensitivity and specificity > 85%).
Interpretation: Rules-based selective testing strategies did not show sufficient diagnostic accuracy in this study population. The relevance of clinical prediction rules for determining which children with pharyngitis should undergo a rapid antigen detection test remains questionable.
See related commentary on page 13 and at www.cmaj.ca/lookup/doi/10.1503/cmaj.141532; see also case on page 50 and at www.cmaj.ca/lookup/doi/10.1503/cmaj.140155
Pharyngitis accounts for about 6% of visits by children to primary care physicians each year in high-income nations.1 Group A streptococcus is found in 30%–40% of cases of childhood pharyngitis; the remaining cases are considered viral.2 Antibiotic treatment is indicated for group A streptococcal infection to prevent suppurative (e.g., retropharyngeal abscess and quinsy) and nonsuppurative complications (e.g., acute rheumatic fever and rheumatic heart disease) and to reduce the duration of symptoms and the spread of the condition.3 In settings where poststreptococcal diseases have become uncommon, such as Western Europe and North America,4 the public health goal is shifting from preventing complications to minimizing the inappropriate use of antibiotic agents to contain antimicrobial resistance.5 However, 60%–70% of the visits by children with pharyngitis to American primary care physicians result in antibiotic agents being prescribed.6–8
Because signs and symptoms of streptococcal and viral pharyngitis overlap, most experts recommend that the diagnosis of group A streptococcal infection be confirmed by a throat culture or rapid antigen detection test.9–13 Whereas European guidelines suggest all children with pharyngitis undergo such testing,14 North American guidelines recommend that clinicians select patients on the basis of clinical and epidemiologic grounds.11–13 Currently, there is no guidance from the Canadian Medical Association or Canadian Paediatric Society for the management of pharyngitis.
Various clinical prediction rules that combine signs and symptoms have been proposed to help clinicians define groups of patients according to the clinical likelihood of group A streptococcal infection.15–18 These rules aim to identify patients at low risk in whom the disease can be managed without further testing and without antibiotic treatment, and patients at high risk who could receive empiric antibiotic treatment without testing.16 Clinical prediction rules for pharyngitis have not been sufficiently validated for clinical practice and have never been compared head-to-head in a single pediatric population from a high-income country.18
The purpose of our study was to externally validate and directly compare the diagnostic accuracy of relevant rules-based selective testing strategies with original data from a French prospective multicentre cohort of children with pharyngitis. To optimize this validation study, we first conducted a systematic review of existing clinical prediction rules.
Methods
Systematic review
All published studies that reported the derivation or update of clinical prediction rules for group A streptococcal pharyngitis were eligible for inclusion. We defined a clinical prediction rule as a decision-making tool that included 2 or more variables obtained from the history, physical examination or simple diagnostic tests and provided a probability of a disease or suggested a clinical course of action for an individual patient.19–23 All studies that provided data specific to children were eligible. For clinical prediction rules updated by their original authors, only the updated study was included. We excluded clinical prediction rules that were explicitly intended for use in low-resource settings (i.e., those derived in developing countries explicitly intended for use in settings where rapid antigen detection tests are not routinely available). We systematically searched MEDLINE and Embase for studies using the search strategy described in Appendix 1 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1). We screened the titles and abstracts of the identified studies; if they were considered potentially relevant (or in cases of uncertainty), we evaluated the full text.
We extracted a prespecified set of data related to study setting and design, and the characteristics of patients and clinical prediction rules. We assessed the quality of included studies using a combination of items from methodological standards for clinical prediction rules19–22 and clinical prediction models,24–26 and from QUADAS-227 (Appendix 2, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1). If published reports referred to previous papers, we extracted the corresponding items. We classified clinical prediction rules according to the hierarchy of evidence for clinical prediction rules21 (with the assessment of validation based on results from previous reviews).15–18 One reviewer (JFC) performed the study selection, data extraction, and assessment of study quality and level of evidence, which were then checked by a second reviewer (MC). Discrepancies were discussed and resolved by consensus.
External validation cohort
We used data from a study for which patients and methods were described previously.28–30 Briefly, this French prospective multicentre study aimed to evaluate the diagnostic accuracy of a rapid antigen detection test in children aged 3–14 years, with throat culture as the clinical reference standard. Seventeen private office-based pediatricians participated. From Oct. 1, 2010, to May 31, 2011, double throat swabs were collected from 678 consecutive children who had a diagnosis of pharyngitis and did not receive antibiotics for 7 days before inclusion. Swab 1 was used for the rapid antigen detection test (StreptAtest, Dectrapharm, France) and swab 2 was sent to the hospital laboratory for throat culture on a blood agar plate according to standard methods.
The following clinical features were recorded for each patient: age, sex, sudden onset of sore throat, maximum body temperature (as reported by a parent), throat pain, cough, rhinorrhea, conjunctivitis, headache, erythema of the pharynx, tonsillar swelling, tonsillar exudate, palatal petechiae, nausea, abdominal pain, vomiting, diarrhea, presence of anterior cervical lymph nodes, size of the nodes (< 1, 1–2 or > 2 cm), tenderness of the nodes, skin rash, and aspect of the rash (scarlatiniform or nonspecific).
Microbiologists were blinded to individual clinical data and results of the rapid antigen detection tests. Parent and patient approval for participation was obtained before inclusion. The study protocol was approved by the Ile-de-France XI institutional review board (no. 09016).
Statistical analysis
Our primary outcome was the diagnostic accuracy of rules-based selective testing strategies, with throat culture as the reference standard. After reviewing the literature and discussing until consensus within the review team, and assuming a prevalence of group A streptococcal infection of 35%2 and a maximally acceptable antibiotics prescription rate of 40%, we defined the target zone of accuracy as sensitivity and specificity greater than 85% (Appendix 3, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1).12,31–33 For each rules-based selective testing strategy, we used a graphical approach to test whether the one-sided rectangular 95% confidence region for sensitivity and specificity lay entirely within the target zone of accuracy.34
Our secondary outcome was the number of rapid antigen detection tests used after each clinical prediction rule. A selective testing strategy was defined as being clinically relevant if less than 80% of children would undergo testing after its application (experts’ opinions). For each selective testing strategy, we tested whether the number of tests used was less than 80% using a one-sided exact binomial probability test.
Each clinical prediction rule was applied to each patient from the validation cohort as follows: low risk of group A streptococcal infection, no further testing or antibiotic therapy; intermediate risk of infection, rapid antigen detection testing for all patients and antibiotic therapy for those with positive test results; high risk of infection, empiric antibiotic treatment. The result of the selective testing strategy was considered positive if the patient would have received antibiotic treatment and negative if the patient would not have received antibiotic treatment after the rule’s application. For rules that rely on scoring systems, we chose cut-offs for defining risk groups according to previously published studies, when available (Appendix 4, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1). This method allowed us to calculate sensitivity, specificity, C-index and number of tests used for each rules-based strategy. We also performed receiver operating characteristic (ROC) analysis for each scoring system.
Another secondary objective was to assess the calibration of clinical prediction rules. For each risk level defined by the rule (low v. intermediate v. high risk of group A streptococcal infection), we compared the probability of group A streptococcal infection seen in our validation cohort (observed probability) to that reported by the authors of the clinical prediction rule in their cohort (predicted probability). We adjusted predicted probabilities for the prevalence of group A streptococcus in the validation cohort.22,35 We evaluated calibration using plots26 and by assessing whether clinical prediction rules were able to identify patients at low risk (probability ≤ 12%, mean prevalence of asymptomatic carriage of group A streptococcus in children)15 or high risk (probability ≥ 85%, minimal predictive value of a positive result on rapid antigen detection testing).15
Values were missing for 0.4%–6.2% patients for each clinical variable. We used multiple imputations with chained equations (m = 10) with predictive mean matching for continuous variables and logistic regression for categorical variables to generate values for missing data (Appendix 5, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1). We used Stata/SE 13 (StataCorp, College Station, Texas) for data analysis.
Results
Systematic review
We performed our literature search on Aug.21, 2014. From 1272 articles identified, we assessed the full text of 38 reports and included 8 articles in our final total (Figure 1). The studies were published between 1975 and 2010 and involved a variety of settings, with a median of 313 patients (range 94–3877) per study, and a median prevalence of group A streptococcus of 36% (range 21%–48%) (Table 1). Eight clinical prediction rules were identified for external validation (Table 2).36–43 The median number of predictors in each rule was 9 (range 5–14) (Appendix 6, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1). The 5 most frequent predictors used were tenderness of lymph nodes, fever, age, tonsillar swelling and exudate.
Selection of clinical prediction rules included in the study.
Description of included studies
Description of clinical prediction rules for group A streptococcal infection
The quality of included studies is summarized in Appendix 7 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1). All of the studies were prospective. The reference standard used and inclusion criteria were sufficiently reported for 5 of 8 studies (63%). Sensitivity and specificity of the clinical prediction rule was reported in 4 of the studies, with corresponding confidence intervals in 2. There are several external validation studies for the Breese,39 McIsaac40 and Wald42 scores. Attia’s clinical prediction rule43 was validated once (Appendix 7). The remaining 4 clinical prediction rules36–38,41 have never been validated. The statistical analysis used for the derivation of clinical prediction rules was poor. All of the models relied on categorization of continuous predictors, such as age and fever. Only 3 of 8 reports (38%) described the use of multivariable analysis. Only 1 article described interaction testing. Assessment of discrimination and calibration of the model were never reported.
External validation cohort
A total of 678 children met our inclusion criteria; we excluded 1 patient with an uninterpretable rapid antigen detection test result and 1 patient whose throat swab was lost. Thus, our analysis included data from 676 children (313 girls [46%], mean age 6.1 [± 2.5] yr). Prevalence of group A streptococcus was 41% (95% confidence interval [CI] 38–45), and rapid antigen detection test sensitivity and specificity were 93% (95% CI 89–95) and 88% (95% CI 85–91), respectively. We had to adapt some of the clinical prediction rules because the authors provided insufficient detail for validation or some clinical variables were not assessed in the validation set; 2 rules could not be validated (Appendix 4).36,41
Diagnostic accuracy of selective testing strategies
The sensitivity of rules-based selective testing strategies ranged from 66% (95% CI 61–72)38 to 94% (95% CI 92–97)40; specificity ranged from 40% (95% CI 35–45)37 to 88% (95% CI 85–91)43; and the C-index ranged from 0.64 (95% CI 0.61–0.67)37 to 0.87 (95% CI 0.85–0.90)43 (Table 3, Figure 2). None of the rules-based strategies fit within our target zone of accuracy. The area under the ROC curve of clinical scores ranged from 0.56 (95% CI 0.52–0.60)40 to 0.62 (95% CI 0.58–0.66)43 (Table 3 and Appendix 8, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1).
External validation of clinical prediction rules in terms of diagnostic accuracy and use of rapid antigen detection testing (n = 676)
External validation of the diagnostic accuracy of rules-based selective testing strategies. Graph shows sensitivity and specificity estimates with their one-sided rectangular 95% confidence regions. Numbers indicate the rules-based selective testing strategies as follows: 1 = Breese,39 2 = Wald,42 3 = Attia,43 4 = Edmond,38 5 = McIsaac40 and 6 = Joachim.37 The target zone of accuracy (sensitivity > 85%, specificity > 85%) is delineated by the dashed horizontal and vertical lines.
Rapid antigen detection testing
Use of rapid antigen detection tests after applying the clinical prediction rule ranged from 24% (95% CI 21–27)37 to 86% (95% CI 84–89)43 (Table 3). Further exploratory analyses found a strong correlation between the number of tests used after the clinical prediction rule and the accuracy of the selective testing strategy as measured by the C-index (Spearman’s ρ = 0.94, p < 0.01; Appendix 9, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1).
Calibration of clinical prediction rules
The prevalence of group A streptococcus increased with increasing risk level as defined by each of the clinical prediction rules. The calibration of 3 clinical prediction rules could not be assessed because the authors did not report prevalence by risk level.36,38,41 For the remaining rules, calibration was poor (Appendices 10 and 11, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1). None of the clinical prediction rules were able to identify patients at low risk (probability of group A streptococcus ≤ 12%) or high risk (probability ≥ 85%). The highest observed probability of group A streptococcus was 81% (score ≥ 4 based on the clinical prediction rule of Attia and colleagues),43 but this group comprised only 6% of patients in our cohort.
Interpretation
Several clinical prediction rules are available for diagnosing childhood pharyngitis.44 Most of these rules are clinical scores based on the same panel of clinical findings classically associated with group A streptococcal or viral cause.16 On external validation, none of the rules-based selective testing strategies showed sufficient accuracy, and none were able to identify patients at low or high risk whose condition could be managed without microbiologic testing.
Our primary aim was to identify a rules-based selective testing strategy that would have a sensitivity and specificity of more than 85%. For the definition of such a target zone of accuracy, one needs to account for both end points in the diagnosis of group A streptococcal pharyngitis. The first is diagnostic sensitivity. Patients with false-negative test results might show complications of streptococcal pharyngitis. However, evidence supporting antibiotic treatment to prevent such complications in high-income settings is limited. From a Cochrane review, the number needed to treat with antibiotics to prevent 1 case of quinsy is about 50,3 but a recent study in the United Kingdom calculated the number needed to treat at about 4000.45 Acute rheumatic fever seems to have almost disappeared in high-income settings,46 and all clinical trials done after 1975 (6 trials including 2484 patients) did not show any efficacy of antibiotics in reducing its incidence, with no case of acute rheumatic fever in either arm.3
The second end point in the diagnosis of streptococcal pharyngitis is diagnostic specificity. Patients with false-positive test results might receive unnecessary antibiotics, thus contributing to the antimicrobial resistance.47 In outpatient settings in the United States, respiratory tract infections account for more than 75% of antibiotic use in children under 12 years of age, and pharyngitis is the most frequent diagnosis associated with prescription of antibiotics to children aged 6–12 years.48 Because of the low impact of antibiotic treatment of streptococcal pharyngitis in children in most industrialized countries and the emergence of antibiotic-resistant bacteria, some northern European countries (e.g., The Netherlands) do not recommend antibiotic treatment for children with pharyngitis, or only for the small number of children with a severe infection or an increased risk of complications.49
We considered a rules-based selective strategy clinically relevant only if it allowed for rapid antigen detection tests to be used in less than 80% of cases. In our validation study, the selective testing strategy that had the highest accuracy43 would have resulted in testing for about 85% of children. In addition, the American Academy of Pediatrics recently recommended rapid antigen detection testing for children with a McIsaac score40 of 2 or more, but this implies that testing should be done in about 90% of cases.50 If clinicians accept such high rates of testing, they might agree to have all children presenting with pharyngitis undergo testing. Such a policy is currently recommended by European guidelines14,51 and was shown to result in a substantial reduction in antibiotic prescription rates.52
In our validation study, clinical prediction rules were unable to identify patients at low or high risk in whom testing for group A streptococcus could be avoided. Our results are consistent with those of previous validation studies (Appendix 12, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.140772/-/DC1). A recent large-scale validation of the McIsaac score40 involving 64 789 children aged 3–14 years in the US showed probabilities of 17% in children with a score of 1 and 55% in children with a score 4 or more,53 which might be far from the misclassification rates clinicians would consider acceptable. Differences in probability by risk level across external validation studies could be explained by differences in study settings and participant characteristics, as well as the low reproducibility of signs and symptoms of streptococcal pharyngitis (i.e., interobserver agreement).54
Limitations
About 20% of the children from our validation cohort had missing data for at least 1 clinical predictor. However, we used multiple imputations, which resulted in less bias than the complete-case approach.26 In a sensitivity analysis, the secondary exclusion of patients with missing data produced slightly higher estimates of diagnostic accuracy than multiple imputations (Appendix 5).
We validated clinical prediction rules using the rapid antigen detection test as a stand-alone test without throat culture, which is the current recommendation of the European Society for Clinical Microbiology and Infectious Diseases.14 Several studies have found this strategy to be the most cost-effective.33,55,56 However, whether such tests are sufficiently accurate to replace throat culture in clinical practice remains a matter of debate.56–58
All steps for undertaking the systematic review were done by 2 reviewers, but not independently, increasing the risk that some clinical prediction rules were missed or that mistakes occurred when extracting the data. A further limitation lies in the absence of an a priori sample size calculation. One of the clinical prediction rules met our target zone of accuracy based on the point estimates alone (Attia’s rule),43 but it was considered insufficient because the boundaries of the confidence intervals for sensitivity and specificity went across the prespecified limits for significance. This could be due to lack of power, and our results should be considered with caution until they are confirmed with a larger sample of patients.
Finally, although this prospective study was originally designed for the purpose of validating existing clinical prediction rules, clinical data were collected before performing the systematic review of the literature. The collection of the clinical variables required for validation was not customized to evaluate the full range of rules we finally included. Some clinical prediction rules had to be adapted because of discrepancies between our clinical variables and the specific definitions originally provided by the authors.
Conclusion
The clinical relevance of clinical prediction rules for triaging children who should undergo group A streptococcal testing was poor in this study, in which both high sensitivity and specificity were expected. Further prospective validation studies comparing multiple clinical prediction rules head-to-head are needed to confirm our results. Future efforts should also aim to better define misclassification rates and the threshold for rapid antigen detection testing that clinicians would consider acceptable.
Acknowledgments
This article is dedicated to the memory of Professor Edouard Bingen (1946–2012), Department of Microbiology, Robert Debré Hospital and Paris Diderot University. The authors thank A. Liboz, Dr. M. Boucherat, Dr. F. de La Rocque, I. Ramay, D. Menguy and M. Fernandes, and the physician investigators who participated in the study: Dr. F. Corrard, Dr. P. Deberdt, Dr. A. Elbez, Dr. M. Goldrey, Dr. J. Gosselin, Dr. M. Koskas, Dr. P. Martin, Dr. A.S. Michot, Dr. N. D’Ovidio, Dr. D. Qutob, Dr. C. Romain, Dr. O. Romain, Dr. C. Schlemmer and Dr. A. Wollner. The authors also thank R. Spijker for his assistance on the literature search.
Footnotes
Competing interests: Robert Cohen and Corinne Levy have received grant funding from Dectrapharm. No other competing interests were declared.
This article has been peer reviewed.
Contributors: Jérémie Cohen and Martin Chalumeau had full access to all data and take responsibility for the integrity of the data and the accuracy of the analyses. Jérémie Cohen, Robert Cohen, Corinne Levy, Philippe Bidet and Martin Chalumeau conceived and designed the study. Robert Cohen, Franck Thollot, Mohammed Benani and Philippe Bidet acquired the data. Jérémie Cohen and Martin Chalumeau performed the statistical analysis. Jérémie Cohen wrote the first draft of the manuscript, which was revised for important intellectual content by all of the authors. All of the authors read and approved the final version submitted for publication and agree to act as guarantors of the work.
Funding: Jérémie Cohen is a doctoral student supported by educational grants from Agence Régionale de Santé d’Ile-de-France, Laboratoires Guigoz - Société Française de Pédiatrie – Groupe de Pédiatrie Générale – Groupe de Recherches Epidémiologiques en Pédiatrie and research grants from the French Ministry of Health (Régional 2012, 12089) and Assistance Publique-Hôpitaux de Paris (Année Médaille de l’Internat 2013). The clinical study for external validation was internally funded by the Association Clinique et Thérapeutique Infantile du Val-de-Marne and externally by Dectrapharm, manufacturer of the rapid antigen detection test. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.
Data sharing: No additional data are available.
References
In this issue
Article tools
Jump to section
Related Articles
Cited By...
- Single-centre, open-label, randomised, trial to compare rapid molecular point-of-care streptococcal testing to standard laboratory-based testing for the management of streptococcal pharyngitis in children: study protocol
- Pharyngitis: Approach to diagnosis and treatment
- Pharyngite: Approche diagnostique et therapeutique
- Group A Streptococcus
- Point-Counterpoint: A Nucleic Acid Amplification Test for Streptococcus pyogenes Should Replace Antigen Detection and Culture for Detection of Bacterial Pharyngitis
- Group A streptococcus
More in this TOC Section
Similar Articles
