Making Sense of Noninferiority: A Clinical and Statistical Perspective on Its Application to Cardiovascular Clinical Trials

doi:10.1016/j.pcad.2006.10.001

Progress in Cardiovascular Diseases

Volume 49, Issue 4, January–February 2007, Pages 284-299

https://doi.org/10.1016/j.pcad.2006.10.001 Get rights and content

Active control noninferiority trials are being used with increasing frequency in new drug or device development when standard placebo-controlled trials are considered unethical. Nevertheless, the design and analysis of these trials are founded on a number of assumptions and arbitrary criteria that are generally not well understood or justifiable. Trials designed to show noninferiority require an appropriate reference population, a proven active control and dose, an appropriate margin of noninferiority that is clinically relevant and statistically justifiable, a high level of adherence to treatment, and adequate statistical power to reliably conclude that a treatment is truly noninferior and therefore effective. Accordingly, if noninferiority trials are to be applied to clinical and regulatory decisions regarding the marketing and use of new treatments, the assumptions must be made explicit and their influence on the resultant conclusions must be assessed rigorously. When conservative criteria were applied to each of the key assumptions underlying 2 representative noninferiority trials, they materially undermined the conclusions regarding noninferiority failing to confirm reported conclusions regarding noninferiority despite enthusiastic dissemination and acceptance of the results. Because the clinical, regulatory, and economic impact of active control noninferiority trials is substantial, robust criteria should be used routinely in their design, analysis, and interpretation to reach their intended objectives and to keep them from becoming wasted efforts.

Section snippets

A Typical Noninferiority Trial

The REPLACE-2 trial is presented as representative of the typical noninferiority trial. This trial was a prospective randomized double-blind trial comparing bivalirudin, a direct thrombin inhibitor, plus provisional platelet glycoprotein (GP) IIb/IIIa inhibitor (new treatment) with unfractionated heparin plus planned GP inhibitor (standard treatment) during elective or urgent percutaneous coronary intervention (PCI) that was unrelated to acute myocardial infarction (MI) or acute coronary

Approaches to Noninferiority Analysis

There are 2 basic approaches to noninferiority analysis. The first approach seeks to determine whether the new treatment is inferior to the standard treatment by no more than some predefined margin (“fixed margin” analysis).4, 5, 6, 7, 8, 9, 10, 11 The second approach seeks to demonstrate indirectly whether the new treatment would be superior to placebo, had a placebo arm been used in the trial (“putative placebo” analysis) The putative placebo approach can also be used to determine whether the

Interpretation of Composite End Points

The analysis of composite end points poses a particular challenge to the interpretation of clinical trials whether they are designed as superiority or noninferiority trials. The construction of the composite end point is generally based on the premise that each component end point is interchangeable.³⁰ However, for this assumption to be valid, each component should be of equal or comparable clinical importance, occur with similar frequency, and be equally responsive to treatment intervention.³¹

A Composite Score for Assessment of Noninferiority

Analysis of noninferiority should ideally be founded on 3 prerequisite judgments¹¹—that the new treatment (i) exhibits “therapeutic noninferiority” (relative efficacy) to the standard treatment, (ii) would exhibit “therapeutic efficacy” in a placebo-controlled trial, and (iii) offers ancillary “nonefficacy benefits” in safety, tolerability, convenience, or cost. The new treatment should offer some or all of the nonefficacy benefits to justify its use in lieu of the standard treatment. A formal

Discussion

Active control noninferiority trials are being used with increasing frequency in the cardiovascular arena. The interpretation of these trials poses a particular challenge to most clinicians. In this paper, we suggest practical standards for the analysis and reporting of these trials to improve the accuracy of their interpretation.

There are several key aspects of the noninferiority inference (summarized in Table 5) that are critical for scientific credibility and regulatory acceptability. First,

References (39)

W.C. Blackwelder
Showing a treatment is good because it is not bad: when DOES “noninferiority” imply effectiveness?
Control Clin Trials
(2002)
J.P. Siegel
Equivalence and noninferiority trials
Am Heart J
(2000)
S. Kaul et al.
Trials and tribulations of noninferiority: the ximelagatran experience
J Am Coll Cardiol
(2005)
W. Blackwelder
Proving the null hypothesis in clinical trials
Control Clin Trials
(1982)
G.W. Stone et al.
Acute Catheterization and Urgent Intervention Triage strategY (ACUITY) trial: study design and rationale
Am Heart J
(2004)
E. Boersma et al.
Platelet glycoprotein IIb/IIIa inhibitors in acute coronary syndromes: a meta-analysis of all major randomised clinical trials
Lancet
(2002)
G.A. Diamond et al.
Prior convictions: Bayesian approaches to the analysis and interpretation of clinical megatrials
J Am Coll Cardiol
(2004)
H.C. Bucher et al.
The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials
J Clin Epidemiol
(1997)
S. Kaul et al.
Trials and tribulations of non-inferiority: the ximelagatran experience. Reply to the letter to the editor
J Am Coll Cardiol
(2006)
R. Temple et al.
Placebo-controlled trials and active-control trials in the evaluation of new treatments: Part 1. Ethical and scientific issues
Ann Intern Med
(2000)

S.S. Ellenberg et al.

Placebo-controlled trials and active-control trials in the evaluation of new treatments: Part 2. Practical issues and specific case

Ann Intern Med

(2000)

H.M.J. Hung et al.

Some fundamental issues with noninferiority testing in active controlled trials

Stat Med

(2003)

H.M.J. Hung et al.

A regulatory perspective on choice of margin and statistical inference issue in noninferiority trials

Biom J

(2005)

R.B. D'Agostino et al.

Noninferiority trials: design concepts and issues—the encounters of academic consultants in statistics

Stat Med

(2003)

S.M. Snapinn

Alternatives for discounting in the analysis of noninferiority trials

J Biopharm Stat

(2004)

M. Rothmann et al.

Design and analysis of non-inferiority mortality trials in oncology

Stat Med

(2003)

S. Kaul et al.

Good enough. A primer on the analysis and interpretation of noninferiority trials

Ann Intern Med

(2006)

A.M. Lincoff et al.

Bivalirudin and provisional glycoprotein IIb/IIIa blockade compared with heparin and planned glycoprotein IIb/IIIa blockade during percutaneous coronary intervention: REPLACE-2 randomized trial

JAMA

(2003)

International Conference on Harmonisation. Statistical principles for clinical trials (ICH E 9) (1998); International...

Cited by (38)

An analysis of statistical terminology applied in emergency medicine literature methods
2022, American Journal of Emergency Medicine
Given the importance of understanding methodical reporting and statistical terminology in ensuring evidence-based decision-making, physicians should possess statistical literacy. The purpose of this study was to distinguish statistical terminology commonly used in emergency medicine methods and describe changes in statistical methods from 2011 to 2021.
The research team chose four journals in emergency medicine widely read and clinically relevant: Academic Emergency Medicine, American Journal of Emergency Medicine, Annals of Emergency Medicine, and Journal of Emergency Medicine. A total of 400 research articles were included. The team compared overall frequencies in statistical word counts as well as differences by year (2021 vs 2011).
Included words from 2011 totaled 31,002 compared to 38,272 in 2021. Unique words for 2011 included 3801 compared to 4291 in 2021. The most common statistical words found in both years were model(s), difference(s), and regression(s). The largest increases in usage included the following words: noninferior(ity), NPV, Fixed, AUC, Mixed, Shapiro, and Wilk. Compared to 2011, results showed a 10% decrease in “p” for 2021. While the terms “Confidence” (2%) and “Intervals” (6%) decreased from 2011 to 2021, there was a 25% increase in “CI(s)”.
By understanding common statistical terms and trends over time, educational efforts can be targeted to consumers of EM literature. Additionally, this work provides evidence suggesting an overall improvement in processes in statistical methodology, enhancing the quality of research outputs and ultimately allowing better clinical decision-making.
Does Percutaneous Left Atrial Appendage Closure Stand the Test of Time?
2022, Journal of the American College of Cardiology
Understanding the Merits and Drawbacks of Noninferiority Trials in Cardiovascular Medicine
2021, Canadian Journal of Cardiology
Citation Excerpt :
The construction of a composite outcome is generally based on the premise that each component end point is interchangeable. However, for this assumption to be valid, each component should be of equal or similar clinical importance, occur with similar frequency, and be equally responsive to treatment intervention.5,6 This is seldom fulfilled, as illustrated by the examples above.
A noninferiority trial is designed to show that the new treatment is not unacceptably worse than the standard treatment by more than a predefined noninferiority margin. These trials are typically performed when standard placebo-controlled trials are considered to be unethical or impractical and the new treatment offers advantages over the existing standard treatment in terms of safety, convenience, or cost. Given that noninferiority trials are being performed with increasing frequency in cardiovascular applications, it is important to understand their complex trial design and analysis. This narrative review aims to provide readers with a detailed perspective on the goals, characteristics, design, and analysis of noninferiority trials. Trials designed to show noninferiority require an appropriate reference population, a proven standard treatment and dose, an appropriate margin of noninferiority that is statistically justifiable (based on historical placebo-controlled trials evaluating standard treatment effect) and clinically reasonable (choosing the fraction of the effect of the standard drug that should be “preserved” by the new drug), a high level of adherence to treatment, and adequate statistical power to reliably conclude that a treatment is truly noninferior and therefore effective. The merits and pitfalls of noninferiority trials, with representative contemporary cardiovascular clinical trials in interventional cardiology, cardiac surgery, and atrial fibrillation management as exemplars, are described. The key issues that challenge the design, conduct, analysis, interpretation, and implementation of these trials are discussed, and a variety of ways to identify and mitigate key errors are recommended to allow for optimal evaluation of noninferiority trials conducted in cardiovascular medicine.
Un essai de non-infériorité est conçu pour démontrer que le nouveau traitement n'est pas radicalement plus mauvais que le traitement standard, au-delà d'une marge de non-infériorité prédéfinie. Ces essais sont généralement réalisés lorsque les essais standard contrôlés par placebo sont considérés comme contraires à l'éthique ou peu applicables et que le nouveau traitement présente des avantages par rapport au traitement standard existant en termes de sécurité, de commodité ou de coût. Étant donné que les essais de non-infériorité sont réalisés de plus en plus fréquemment dans les applications cardiovasculaires, il est important d'en comprendre leur conception et leur analyse complexes. Cette synthèse narrative vise à fournir aux lecteurs une perspective détaillée sur les objectifs, les caractéristiques, la conception et l'analyse des essais de non-infériorité. Les essais conçus pour démontrer une non-infériorité nécessitent une population de référence appropriée, un traitement et une dose standard éprouvés, une marge de non-infériorité appropriée qui soit statistiquement justifiable (basée sur des essais historiques contrôlés par placebo évaluant l'effet du traitement standard) et cliniquement raisonnable (choix de la fraction de l'effet du médicament standard qui devrait être "préservée" par le nouveau médicament), un niveau élevé d'adhésion au traitement et une puissance statistique adéquate pour conclure de manière fiable qu'un traitement est réellement non-inférieur et donc efficace. Les mérites et les pièges des essais de non-infériorité sont décrits à l'aide d'exemples représentatifs d'essais cliniques cardiovasculaires contemporains en cardiologie interventionnelle, en chirurgie cardiaque et en gestion de la fibrillation auriculaire. Les principaux problèmes qui se posent lors de la conception, de la conduite, de l'analyse, de l'interprétation et de la mise en œuvre de ces essais sont discutés, et divers moyens d'identifier et d'atténuer les principales erreurs sont recommandés pour permettre une évaluation optimale des essais de non-infériorité menés en médecine cardiovasculaire.
The choice of the noninferiority margin in clinical trials was driven by baseline risk, type of primary outcome, and benefits of new treatment
2015, Journal of Clinical Epidemiology
To explore characteristics of clinical trials that influence the choice of the noninferiority margin (NIM) when planning the trial.
We conducted an experimental survey among corresponding authors of randomized controlled trials indexed in MEDLINE. We described two hypothetical studies and asked the respondents' opinion on the largest loss of effectiveness that is clinically negligible (or the smallest lost of effectiveness that is clinically important in the superiority scenario). We randomly manipulated four study attributes in each vignette, using a factorial design.
A total of 364 researchers participated. The values for NIMs were significantly lower than the differences to be detected in a superiority trial. The NIM was smaller when the primary outcome was mortality compared with treatment failure, when baseline risk in the control arm was lower, and when the advantage of the new treatment was a lower cost compared with having fewer side effects. In contrast, the population age group under study and the difficulty to recruit patients showed no effect on the choice of the NIM.
In our experimental study, the factors associated with lower NIMs were mortality as a primary outcome, low baseline risk, and a less costly new treatment.
Weighting composite endpoints in clinical trials: Essential evidence for the heart team
2012, Annals of Thoracic Surgery
Coronary revascularization trials often use a composite endpoint of major adverse cardiac and cerebrovascular events (MACCE). The usual practice in analyzing data with a composite endpoint is to assign equal weights to each of the individual MACCE elements. Noninferiority margins are used to offset effects of presumably less important components, but their magnitudes are subject to bias. This study describes the relative importance of MACCE elements from a patient perspective.
A discrete choice experiment was conducted. Survey respondents were presented with a scenario that would make them eligible for the Synergy Between Percutaneous Coronary Intervention With Taxus and Cardiac Surgery (SYNTAX) trial three-vessel disease cohort. Respondents chose among pairs of procedures that differed on the 3-year probability of MACCE, potential for increased longevity, and procedure/recovery time. Conjoint analysis derived relative weights for these attributes.
In all, 224 respondents completed the survey. The attributes did not have equal weight. Risk of death was most important (relative weight 0.23), followed by stroke (0.18), potential increased longevity and recovery time (each 0.17), myocardial infarction (0.14), and risk of repeat revascularization (0.11). Applying these weights to the SYNTAX 3-year endpoints resulted in a persistent, but decreased margin of difference in MACCE favoring coronary artery bypass graft surgery compared to percutaneous coronary intervention. When labeled only as “procedure A” and “procedure B,” 87% of respondents chose coronary artery bypass graft surgery over percutaneous coronary intervention. When procedures were labeled as “coronary stent” and “coronary bypass surgery,” only 73% chose coronary artery bypass graft surgery. Procedural preference varied with demographics, sex, and familiarity with the procedures.
The MACCE elements do not carry equal weight in a composite endpoint, from a patient perspective. Using a weighted composite endpoint increases the validity of statistical analyses and trial conclusions. Patients are subject to bias by labels when considering coronary revascularization.
Randomized comparison of a polymer-free sirolimus-eluting stent versus a polymer-based paclitaxel-eluting stent in patients with diabetes mellitus: The LIPSIA yukon trial
2011, JACC: Cardiovascular Interventions
The objective of the study was to assess noninferiority of the polymer-free sirolimus-eluting Yukon Choice stent (Translumina GmbH, Hechingen, Germany) compared with the polymer-based Taxus Liberté stent (Boston Scientific, Natick, Massachusetts) with regard to the primary endpoint, in-stent late lumen loss, at 9 months in patients with diabetes mellitus.
The Yukon Choice stent has been evaluated in several randomized controlled trials before, albeit to date, there has been no trial that exclusively enrolled patients with diabetes mellitus.
Patients with diabetes mellitus undergoing percutaneous coronary intervention for clinically significant de novo coronary artery stenosis were randomized 1:1 to receive either the polymer-free sirolimus-eluting Yukon Choice stent or the polymer-based paclitaxel-eluting Taxus Liberté stent.
A total of 240 patients were randomized. Quantitative coronary angiography was available for 79% of patients. Mean in-stent late lumen loss was 0.63 ± 0.62 mm for the Yukon Choice stent and 0.45 ± 0.60 mm for the Taxus Liberté stent. Based on the pre-specified margin, the Yukon Choice stent failed to show noninferiority for the primary endpoint. During follow-up, there were no significant differences between groups regarding death, myocardial infarction, stent thrombosis, target lesion revascularization, target vessel revascularization, or nontarget vessel revascularization.
Compared with the Taxus Liberté stent, the polymer-free sirolimus-eluting Yukon Choice stent failed to show noninferiority with regard to the primary endpoint, in-stent late lumen loss, in patients with diabetes mellitus after 9-month follow-up. Both stents showed comparable clinical efficacy and safety. (Yukon Choice Versus Taxus Liberté in Diabetes Mellitus; NCT00368953)

View all citing articles on Scopus

View full text

Special ArticleMaking Sense of Noninferiority: A Clinical and Statistical Perspective on Its Application to Cardiovascular Clinical Trials

Section snippets

A Typical Noninferiority Trial

Approaches to Noninferiority Analysis

Interpretation of Composite End Points

A Composite Score for Assessment of Noninferiority

Discussion

Control Clin Trials

Am Heart J

J Am Coll Cardiol

Control Clin Trials

Am Heart J

Lancet

J Am Coll Cardiol

J Clin Epidemiol

J Am Coll Cardiol

Placebo-controlled trials and active-control trials in the evaluation of new treatments: Part 1. Ethical and scientific issues

Ann Intern Med

Placebo-controlled trials and active-control trials in the evaluation of new treatments: Part 2. Practical issues and specific case

Ann Intern Med

Some fundamental issues with noninferiority testing in active controlled trials

Stat Med

A regulatory perspective on choice of margin and statistical inference issue in noninferiority trials

Biom J

Noninferiority trials: design concepts and issues—the encounters of academic consultants in statistics

Stat Med

Alternatives for discounting in the analysis of noninferiority trials

J Biopharm Stat

Design and analysis of non-inferiority mortality trials in oncology

Stat Med

Good enough. A primer on the analysis and interpretation of noninferiority trials

Ann Intern Med

Bivalirudin and provisional glycoprotein IIb/IIIa blockade compared with heparin and planned glycoprotein IIb/IIIa blockade during percutaneous coronary intervention: REPLACE-2 randomized trial

JAMA

Special Article
Making Sense of Noninferiority: A Clinical and Statistical Perspective on Its Application to Cardiovascular Clinical Trials