Review Article
The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index,

https://doi.org/10.1016/j.jclinepi.2013.10.019Get rights and content
Under a Creative Commons license
open access

Abstract

Objectives

A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes.

Study Design and Setting

A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result.

Results

The 399 eligible trials had a median sample size of 682 patients (range: 15–112,604) and a median of 112 events (range: 8–5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0–109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up.

Conclusion

The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results.

Keywords

Randomized controlled trials
Research methodology
Lost to follow-up

Cited by (0)

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/).

Conflict of interest: The authors have no conflicts of interest to declare. M.W. is supported by a New Investigator Award from the Kidney Research Scientist Core Education National Training (KRESCENT) Program. P.J.D. is supported by a Heart and Stroke Foundation of Ontario Career Investigator Award.

Ethical approval was not required for this study.

Funding: No funding was received for the conduct of this study and no sponsor was required.