If nothing happens, is everything all right? Distinguishing genuine reassurance from a false sense of security
==============================================================================================================

* Yoon Kong Loke
* Katharina Mattishent

See related research article on page [E21](http://www.cmaj.ca/lookup/volpage/187/E21) and at [www.cmaj.ca/lookup/doi/10.1503/cmaj.140848](http://www.cmaj.ca/lookup/doi/10.1503/cmaj.140848)

In a linked research paper, Gillies and colleagues present a systematic review and meta-analyses showing a favourable safety profile for amoxicillin.1 Their findings are important for prescribers and patients around the world who must weigh the benefits and harms of empiric amoxicillin therapy in situations clouded by diagnostic uncertainty. The analyses by Gillies and colleagues highlight the contentious matter of reporting adverse events, which is in need of considerable improvement.2

The meta-analyses by Gillies and colleagues were based on adverse events occurring in randomized controlled trials that compared amoxicillin (as monotherapy or in combination with clavulanic acid) against placebo.1 Intriguingly, amoxicillin monotherapy was not associated with significant harm (such as diarrhea, rash or nausea), whereas amoxicillin–clavulanic acid was associated with a significant increase in the risk of diarrhea. Gillies and colleagues judged their meta-analyses to have “low risk of bias,” which perhaps gives us greater reassurance about the apparent lack of harm with amoxicillin.

Nonetheless, several questions come to mind. How confident should we be in accepting these “null” findings, and can clinicians continue to prescribe this antibiotic without worrying too much about unintended consequences? Can we trust that there is a genuinely low risk of bias (i.e., high internal validity) in the reported absence of harm? Might there be reporting biases that prevent us from seeing the whole picture? Finally, can perceived safety based on the experience of participants in randomized trials be reliably extrapolated to patients in real-life clinical practice?

In practice, prescribing decisions are complicated by the risk of adverse effects and by emerging bacterial resistance from misuse (or overuse) of antibiotics.3 In a recent survey study, 55% of family physicians in the United Kingdom felt under pressure (from patients) to prescribe antibiotics.4 It is important that decisions to prescribe antibiotics be informed by high-quality safety data from systematic reviews.

We must recognize the inherent difficulties of accurately capturing rare or unexpected adverse events that were not specified or defined beforehand.5 Only 18% of the trials in the study by Gillies and colleagues used a diary to record harms,1 but such a lack of rigorous ascertainment of harm leads to “non-differential misclassification,” with bias toward the null (i.e., lower estimates of risk of harm).6 To put it simply, if you don’t look properly, you won’t find — a combination of poor monitoring with a lack of clear case definitions could lead to misclassification or nondetection of genuine adverse reactions.5 The paucity of recorded events in the trials included in the analysis by Gillies and colleagues potentially creates an illusion of safety. However, we wonder if the greater perceived risk of diarrhea in trials involving amoxicillin–clavulanic acid is related to more rigorous ascertainment in those trials, with null findings for amoxicillin monotherapy simply reflecting poorer ascertainment of adverse effects rather than a genuine safety advantage.

We believe that systematic reviews of harm should explicitly assess the risk of bias toward the null. Such an assessment enables clinicians and patients to be warned against a false sense of security where the drug is erroneously declared safe or not significantly different from placebo (type 2 error). Although current risk-of-bias tools are designed to detect inflated treatment differences (type 1 error, i.e., finding an effect that is not truly present), we feel strongly that studies in which “the drug showed no significant harm” must be subjected to the same rigorous scrutiny as studies with claims of efficacy. Specific tools for quality assessment of adverse events are available (e.g., the McMaster Quality Assessment Scale of Harms, also known as McHarm7). For example, a recent systematic review of harm with dexamethasone deployed the Cochrane risk-of-bias tool and the McHarm tool in parallel.8

A further important consideration in meta-analyses of adverse events is the extent of missing information. In the study by Gillies and colleagues, data on diarrhea were available from only 10 of 28 eligible amoxicillin trials. Our confidence in the findings is shaken by the sparseness of the dataset, and we recognize that existing estimates of harm could change markedly if new information emerges (either from future trials or in the form of the missing outcomes data). We suspect that the strength of this evidence might be considered low within the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework.9

Potential bias from selective outcome reporting may also reduce confidence in pooled estimates of harm. For example, adverse events might have been recorded, but researchers did not report the outcomes. The exact reasons for selective non-reporting are unclear, but certain situations may arise to create a positive impression. For instance, perhaps researchers selectively omitted reports of harm to generate a favourable benefit–harm profile, or because they felt that adverse effects were unimportant. Equally, an excess of adverse events with antibiotic (relative to placebo) might have gone unreported if the statistical analysis did not yield significant results.

In the 45 primary studies that Gillies and colleagues included in their systematic review, harms follow-up was no more than one week in nine of the trials, and 17 trials did not specify the duration of harms follow-up.1 The generalizability of findings is questionable when highly selected trial participants are followed for only a short time.5 If antibiotic-associated diarrhea is a late adverse event, short-term studies may fail to record any harm (type 2 error). Nonrandomized community studies may therefore generate higher estimates of harm if participants had more comorbidities, with longer duration of antibiotic use and longer follow-up.

If an adverse effect is not reported, does that mean that it did not happen or does not exist? Not necessarily. Although, understandably, one of the main aims of a trial is to assess efficacy and whether a particular treatment works, it is just as important (albeit not as lucrative) to accurately report adverse events.2

Amoxicillin has been widely used for decades, and it seems shameful that data on harms are missing from so many trials. For this drug, clinicians and patients must not construe “absence of evidence of harm” to be the same as “evidence of absence of harm.” The systematic review by Gillies and colleagues lends weight to the growing call for full transparency, rather than restricted access and selective release of trial data.10

Key points
*   Poor monitoring, lack of clear case definitions and missing data mean that genuine adverse reactions may go undetected or be misclassified.

*   Systematic reviews of harm should explicitly assess this risk of bias toward the null to prevent a false sense of security (type 2 error), whereby a drug is erroneously declared safe or not significantly different from the comparator.

*   Quality assessment tools for adverse events allow findings of “no significant harm” to be subjected to the same rigorous scrutiny as claims of efficacy.

*   Absence of evidence of harm should not be construed as evidence of absence of harm, and there is a pressing need for unrestricted access to complete trial datasets for both beneficial and adverse effects.

## Footnotes

*   **Competing interests:** None declared.

*   This article was solicited and has not been peer reviewed.

*   **Contributors:** Both authors contributed substantially to conception and design of the article and interpretation of the data; drafted the article; gave final approval of the version to be published; and agree to act as guarantors of the work in ensuring that questions related to any part of the work are appropriately investigated and resolved.

## References

1.  Gillies M, Ranakusuma A, Hoffman T, et al. Common harms from amoxicillin: a systematic review and meta-analysis of randomized placebo-controlled trials for any indication. CMAJ 2014; Nov. 17 [Epub ahead of print].
    
    

2.  Loke YK. Lack of clarity in reports of adverse events: Is there any harm? Pain 2013;154:183–4.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.pain.2012.10.019&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=23159572&link_type=MED&atom=%2Fcmaj%2F187%2F1%2F15.atom) 

3.  Bell BG, Schellevis F, Stobberingh E, et al. A systematic review and meta-analysis of the effects of antibiotic consumption on antibiotic resistance. BMC Infect Dis 2014;14:13.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/1471-2334-14-13&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24405683&link_type=MED&atom=%2Fcmaj%2F187%2F1%2F15.atom) 

4.  Cole A. GPs feel pressurised to prescribe unnecessary antibiotics, survey finds. BMJ 2014;349:g5238.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE4OiIzNDkvYXVnMTlfMTUvZzUyMzgiO3M6NDoiYXRvbSI7czoxOToiL2NtYWovMTg3LzEvMTUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

5.  Loke YK, Golder SP, Vandenbroucke JP. Comprehensive evaluations of the adverse effects of drugs: importance of appropriate study selection and data sources. Ther Adv Drug Saf 2011;2:59–68.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1177/2042098611401129&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=25083202&link_type=MED&atom=%2Fcmaj%2F187%2F1%2F15.atom) 

6.  Delgado-Rodríguez M, Llorca J. Bias. J Epidemiol Community Health 2004;58:635–41.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiamVjaCI7czo1OiJyZXNpZCI7czo4OiI1OC84LzYzNSI7czo0OiJhdG9tIjtzOjE5OiIvY21hai8xODcvMS8xNS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

7.  Chou R, Aronson N, Atkins D, et al. AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol 2010;63:502–12.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jclinepi.2008.06.007&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=18823754&link_type=MED&atom=%2Fcmaj%2F187%2F1%2F15.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000276534800008&link_type=ISI) 

8.  Bellis JR, Pirmohamed M, Nunn AJ, et al. Dexamethasone and haemorrhage risk in paediatric tonsillectomy: a systematic review and meta-analysis. Br J Anaesth 2014;113:23–42.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1093/bja/aeu152&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24942713&link_type=MED&atom=%2Fcmaj%2F187%2F1%2F15.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000338598700006&link_type=ISI) 

9.  Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011; 64:401–6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.07.015&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=21208779&link_type=MED&atom=%2Fcmaj%2F187%2F1%2F15.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000288364300009&link_type=ISI) 

10. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet 2014;383: 257–66.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0140-6736(13)62296-5&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24411650&link_type=MED&atom=%2Fcmaj%2F187%2F1%2F15.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000330027500035&link_type=ISI)