Skip to main content

Main menu

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
    • Trousse média 2021
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN

User menu

Search

  • Advanced search
CMAJ
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN
CMAJ

Advanced Search

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
    • Trousse média 2021
  • Visit CMAJ on Facebook
  • Follow CMAJ on Twitter
  • Follow CMAJ on Pinterest
  • Follow CMAJ on Youtube
  • Follow CMAJ on Instagram
Review

Why randomized controlled trials fail but needn't: 1. Failure to gain "coal-face" commitment and to use the uncertainty principle

David L. Sackett
CMAJ May 02, 2000 162 (9) 1311-1314;
David L. Sackett
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Tables
  • Metrics
  • Responses
  • PDF
Loading

Abstract

From the undersideMy pager went off as I was half-dozing, half-reading the newspaper. Yuri Gagarin, born the same year as me, had just become the first man to orbit the earth; Nelson Mandela, not yet imprisoned, had gone underground; and John F. Kennedy had just signed the bill creating the Peace Corps. An hour later, back on the charity ward and feeling the effects of too much acrid coffee and too little sleep, achalasia hit me when I found the lancet-shaped diplococci in her sputum. It was 3 am and the research protocol, appearing unannounced on our workroom wall, declared that, as the front-line clinician (the British call this "working at the coal face"), I had to enter her into the trial.

From the undersideShe was single, poor, the sole supporter of 3 children and now so sick after 10 days of cough, fever and sputum that she'd risked losing her menial job by leaving it early to struggle through Chicago's filthy March slush to our emergency room. My boss was testing one of the first synthetic penicillins against the then standard penicillin G in patients with pneumococcal pneumonia, and she fit the entry criteria. But she had classic signs of hepatization and already had suffered an episode of the euphoria and cyanosis we'd been taught was characteristic of bacteremia and interlobar spread. The last patient I'd seen this sick from pneumococcal pneumonia, a strapping 18-year-old basketball player, was also the first patient on whom I'd conducted fruitless open-chest cardiac massage.

From the undersideWith the fear, hopelessness and trust I'd come to expect from my patients, she consented at once to take part in the trial. But by the time I completed her entry form I knew what I had to do. Blocking the view of the ward nurse, I took the syringe containing the study drug from the refrigerator, loaded a second syringe with penicillin G and injected her with both.

From the undersideI have never discussed this decision with anyone, nor admitted it until now. I don't know how many of my fellow house officers did the same thing for their sickest patients. I believe that my action was right in particular, wrong in general (I've never cheated since), and doubly preventable.

Effect

To the extent that other house officers doing their best to care for similar patients responded as I did, patients in both arms of this randomized controlled trial (RCT) would have been given penicillin G. As a result, it was a trial of the effect of adding synthetic penicillin to penicillin G, rather than the comparison of the 2 drugs that our boss had intended. In the absence of a negative interaction between the 2 penicillins, any lack of efficacy of synthetic penicillin when given alone would have remained undetected. The effect was a "false-equivalence" RCT.

I don't know how often this type of RCT failure occurs, and I've never encountered anyone who does know. Ken Schulz1 collected anecdotes of attempts to "break the code" to find out what treatment a patient would be assigned to if they entered an RCT, from transilluminating a sealed envelope to rifling through the files of the principal investigator.

Causes

There were 2 underlying causes for the failure (at least in my contribution to it) of this RCT. The first was failure to gain "coal-face" commitment. I was an underling who had never been party to any discussion of the study question and how it would best be answered. It was not "my trial," and I had only a general stake, not a personal one, in guaranteeing that it generated a valid answer. The second cause was failure to use the uncertainty principle, I was certain that my patient needed penicillin G, and my responsibility to her welfare was in direct conflict with my responsibility to the internal validity of the RCT. Moreover, this possibility was neither acknowledged nor accommodated in the protocol posted on the wall. I could not serve both my patient and the RCT, and in serving the former I cheated the latter because this was preferable to serving the latter and thereby cheating the former, my patient. This conflict arose from my "knowing" that my patient needed penicillin G, and such convictions are just as relevant when they are wrong as when they are right.

Principles

Coal-face commitment

Only collaborators and patients who consider an RCT "theirs" should be expected to follow its protocol. Although this principle has to do with human behaviour, not research methods, it is a key determinant of the internal validity and efficiency of every RCT. Given the increasing variety and magnitude of the competing demands placed on front-line clinicians, no one should be surprised to discover these clinicians neglecting any task they deem nonessential. The more detailed the entry form and eligibility criteria for "somebody else's" RCT, the greater the risk the criteria will be ignored, misunderstood or misapplied by distracted clinicians who regard them as further intrusions into an overfull call schedule.

Uncertainty principle

Patient entry into RCTs should be governed by the uncertainty principle. What is "ethical" in one culture or era may be "unethical" in another; therefore, in this series I will avoid pronouncements on good and evil. However, when the set of ethical principles that are in vogue at a certain place and time impinge on the design and conduct of contemporaneous RCTs, I will examine the evidence on which the principles are based and their effects on RCTs. In the example I present in this essay, we need to consider both the waning North American principle of "equipoise" and the principle of "uncertainty" that is being increasingly adopted in most other parts of the world.

Equipoise is a "state of balance or equilibrium between two alternative therapies"2 such that "there is no preference between treatments, i.e., it is thought equally likely that treatment A or B will turn out to be superior. At this point we may be said to be 'agnostic' ... we would take odds of 1:1 on a bet."3 In certain times and places, equipoise has been considered a prerequisite for an ethical RCT. And in some of these times and places, the individual clinician and patient had to be free of any "hunch" or preference ("theoretical" equipoise),4 while in others, individual clinicians and patients could have a preference as long as they "recognize that their less-favored treatment is preferred by colleagues whom they consider to be responsible and competent"5 ("clinical" equipoise).

Opponents of the equipoise construct (including me) argue that it has 3 fatal flaws. First, it is incapable of application: equipoise is lost as soon as the first pair of patients given the alternative treatments finishes the trial and the allocation code is broken. Second, it treats hunches (preferences) as point-estimates and ignores the uncertainty with which those hunches are held. Put another way, and linking the constructs of equipoise and uncertainty, equipoise demands a "confidence interval" of zero, whereas uncertainty permits and works with confidence intervals of 50%, 99%, or any other magnitude. Third, and as a result of the first 2, equipoise is almost never possessed by trialists or explored by ethics committees. For example, in the great majority of the more than 200 RCTs in which I have played a role, neither I nor my patients nor my collaborators nor the nonparticipants we encountered from the relevant profession were in equipoise, and our hunches frequently were strong ones (indeed, some potential clinical collaborators were so convinced that the experimental treatment was efficacious or useless that they refused to have anything to do with a trial of it). It isn't that we have no hunches and are indifferent to the alternative treatments in our RCTs, it's that we are uncertain about whether our hunches are correct.

The "uncertainty" construct rejects the indifference of equipoise and builds on the notion that clinicians and patients are often uncertain whether their hunches about a treatment's effectiveness are true. That is, although their hunch about a specific treatment may be that it is probably effective, the boundaries ("confidence interval") around that hunch may run all the way from extremely effective (a wonder drug), across zero (ineffective) and into the realm of frank harm. When the uncertainty boundaries of a group of clinicians and patients include or cross zero (such that they recognize that the treatment they prefer might, in fact, be useless or even harmful), it is time for a trial, and that trial is ethical. Similarly, in equivalence trials such as the one that opened this essay, uncertainty exists as long as the confidence interval around the hunch that one of the treatments is actually superior includes or crosses zero. As I'll show you in the following section, this uncertainty principle helps decision-making not only by individual clinicians and individual patients, but also by trialists and trial monitors.

Preventive strategies

Achieve commitment to the trial in the front lines

Application of the "coal-face" principle (only clinicians and patients who consider an RCT "theirs" should be expected to follow its protocol) should begin with the first draft of the question to be answered by the RCT (fighting through the question to be posed by an RCT deserves much of the total effort expended on it, and will be discussed in greater detail in future essays in this series). The question should be shown, discussed and argued over with an ever-widening array of clinical and methodological collaborators, and ultimately with the front-line physicians, research assistants and study nurses who will effect its success or failure. In multicentre RCTs, part of each centre's responsibility should be to educate and involve those who are admitting and following the study patients and responding to their questions and concerns throughout the trial. The benefits of this time-consuming activity are 4. First, the attendant discussion and debate improves the specificity and clinical usefulness of the question. Second, when this process draws in other scientists, including bench researchers, it improves the science used to answer the question (and explain it). Third, these discussions permit collaborators and front-line participants to "buy in" to the trial and develop both the ownership and commitment that are essential for the successful assembly, care and follow-up of study patients and for adherence to the study protocol. Finally, the discussions provide the forum in which to understand the uncertainty principle and to gain confidence in the front lines in its ability to preserve patient choice and clinical judgement while protecting study validity.

Apply the uncertainty principle when entering patients into RCTs

My contribution to the failure of the penicillin trial would have been prevented if its eligibility criteria had incorporated the general principle of uncertainty as it applies to the individual patient. I find this incorporation best articulated by Richard Peto and Colin Baigent:6

A patient should not be entered if the responsible clinician or the patient are for any medical or non-medical reasons reasonably certain that one of the treatments that might be allocated would be inappropriate for this particular individual (in comparison with either no treatment or some other treatment that could be offered to the patient in or outside the trial).

I was reasonably certain that the synthetic penicillin was inappropriate for my patient. Had the uncertainty principle been in effect, my patient would never have entered the trial, and its internal validity would have been protected. Yes, one less patient would have entered the trial. But the consequent loss in the study's precision could have been made up by prolonging its recruitment phase, while the loss in its validity was irreparable.

There are several supplemental benefits to this preventive strategy. First, making patients equal partners in the application of the uncertainty principle legitimizes their hunches, respects their autonomy and reinforces the need for their informed consent. Second, its application can reduce complexity, confusion and waste in the generation and application of eligibility and ineligibility criteria. When exclusion criteria try to anticipate all of the real-world situations in which a reasonable clinician might not want to invite an eligible patient to join an RCT, they swell in size and complexity, confound the patient's risk and responsiveness with the clinician's responsibility and can result in unnecessarily strict entry criteria (e.g., age) and decreased patient numbers. Third, the application of the uncertainty principle to individual patients acts synergistically with the deliberations of the trial monitors, who examine the accumulating unblinded results. Their prime duty is to monitor the boundaries of uncertainty at the group level, alerting the trialists when it shrinks (for all patients or sensible subgroups) to the point where the more effective treatment becomes clear (and if this clarity emerges for some prespecified subgroups but not others, the trial can be stopped for the former but continued for the latter, all on the basis of the uncertainty principle). Finally, while respecting the clinical judgement that keeps some patients and clinicians from entering RCTs, the uncertainty principle is impervious to the validity of their hunches; even when the hunches are wildly wrong, acting on them will not damage the internal validity of the trial result.

The uncertainty principle acknowledges that most clinicians and patients do have hunches about a treatment's effectiveness but that the boundaries ("confidence interval") around their hunches may run all the way from extremely effective (a wonder drug), across zero (ineffective) and into the realm of frank harm.

Equipoise is a "state of balance or equilibrium between two alternative therapies" such that "there is no preference between treatments, i.e., it is thought equally likely that treatment A or B will turn out to be superior." Some definitions require each individual clinician and patient to be free of any "hunch" or preference ("theoretical" equipoise), while others permit individual clinicians and patients to have hunches as long as they "recognize that their less-favored treatment is preferred by colleagues whom they consider responsible and competent" ("clinical" equipoise).

Preventing RCT failure

When the success or failure of the RCT depends on the recruitment of patients who are appropriate in type and number, achieve commitment to the trial in the front lines (the "coal face"). This will

  • improve adherence to the protocol,

  • improve the study question,

  • improve the science and

  • provide a forum for incorporating values into research.

When clinicians and patients have hunches about efficacy (i.e., always), apply the uncertainty principle when entering patients into RCTs. This will

  • recognize and reinforce clinical judgement and patients' values and autonomy,

  • reduce complex eligibility criteria and paper work,

  • act synergistically with the deliberations of the trial monitors and

  • protect internal validity regardless of whether hunches are correct or wildly wrong.

I thank 22 colleagues who offered encouragement and suggestions on drafts of this essay.

Competing interests: A detailed statement appears at the end of the commentary that introduces this series (page 1301).

Acknowledgments

CMAJ will award prizes for the best essays on any health-related subject submitted during calendar year 2000. A $2000 prize will be awarded for the best entry submitted by a medical student or resident. There is also a $2000 prize for the best entry submitted by any author. These new contests replace the Logie Medical Ethics Essay Contest for medical students.

We are looking for reflective essays of up to 1500 words. Manuscripts must be original and must be submitted only to CMAJ. Winners will be selected by a committee appointed from the CMAJ Editorial Board. Winning entries will be selected based upon originality, quality of writing and relevance to health or health care. To win, a manuscript must be suitable for publication. If suitable entries are not received, prizes will not be awarded. All papers submitted will be considered for publication in CMAJ.

Authors should submit their papers with a covering letter stating that they wish the manuscript to be considered for the essay prize, and should indicate their status regarding training. Send entries and queries to: Dr. John Hoey, 1867 Alta Vista Dr., Ottawa ON K1G 3Y6; hoeyj{at}cma.ca

Association médicale canadienne

Canadian Medical Association

Footnotes

  • This article has been peer reviewed.

    SYMBOL See related article page 1301

    Figure1
    • Download figure
    • Open in new tab
    • Download powerpoint

    Symbol. No caption available.

    Reprint requests to: Dr. David L. Sackett, Trout Research and Education Centre at Irish Lake, RR 1, Markdale ON N0C 1H0; sackett@bmts.com

References

  1. 1.↵
    Schulz KF. Subverting randomization in controlled trials. JAMA 1995;274:1456-8.
    OpenUrlCrossRefPubMed
  2. 2.↵
    Singer PA, Lantos JD, Whitington PF, Broelsch CE, Siegler M. Equipoise and the ethics of segmental liver resection. Clin Res 1988;36:539-45.
    OpenUrlPubMed
  3. 3.↵
    Lilford RJ, Jackson J. Equipoise and the ethics of randomisation. J R Soc Med 1995;88:552-9.
    OpenUrlFREE Full Text
  4. 4.↵
    Hellman S, Hellman DS. Of mice but not men - problems of the randomized trial. N Engl J Med 1991;324:1585-9.
    OpenUrlPubMed
  5. 5.↵
    Freedman B. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-5.
    OpenUrlCrossRefPubMed
  6. 6.↵
    Peto R, Baigent C. Trials: the next 50 years. BMJ 1998;317:1170-1.
    OpenUrlFREE Full Text
PreviousNext
Back to top

In this issue

CMAJ
Vol. 162, Issue 9
2 May 2000
  • Table of Contents
  • Index by author

Article tools

Respond to this article
Print
Download PDF
Article Alerts
To sign up for email alerts or to access your current email alerts, enter your email address below:
Email Article

Thank you for your interest in spreading the word on CMAJ.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Why randomized controlled trials fail but needn't: 1. Failure to gain "coal-face" commitment and to use the uncertainty principle
(Your Name) has sent you a message from CMAJ
(Your Name) thought you would like to see the CMAJ web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Why randomized controlled trials fail but needn't: 1. Failure to gain "coal-face" commitment and to use the uncertainty principle
David L. Sackett
CMAJ May 2000, 162 (9) 1311-1314;

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
‍ Request Permissions
Share
Why randomized controlled trials fail but needn't: 1. Failure to gain "coal-face" commitment and to use the uncertainty principle
David L. Sackett
CMAJ May 2000, 162 (9) 1311-1314;
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • Effect
    • Causes
    • Principles
    • Preventive strategies
    • Preventing RCT failure
    • Acknowledgments
    • Footnotes
    • References
  • Figures & Tables
  • Responses
  • Metrics
  • PDF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Cited By...

  • Treating hypotension in extremely preterm infants. The pressure is mounting
  • Why randomized controlled trials fail but needn't: 2. Failure to employ physiological statistics, or the only formula a clinician-trialist is ever likely to need (or understand!)
  • Uncertainty about clinical equipoise
  • Uncertainty about clinical equipoise
  • Uncertainty and equipoise
  • Why Sackett's analysis of randomized controlled trials fails, but needn't
  • For and against: Clinical equipoise and not the uncertainty principle is the moral underpinning of the randomised controlled trial FOR AGAINST
  • Google Scholar

More in this TOC Section

  • Virtual care for patients with Alzheimer disease and related dementias during the COVID-19 era and beyond
  • The efficacy and safety of sunscreen use for the prevention of skin cancer
  • Prone positioning for patients with hypoxic respiratory failure related to COVID-19
Show more Review

Similar Articles

Collections

  • Topics
    • Research methods & statistics
    • Journalology & publication ethics

Content

  • Current issue
  • Past issues
  • Collections
  • Sections
  • Blog
  • Podcasts
  • Alerts
  • RSS
  • Early releases

Information for

  • Advertisers
  • Authors
  • Reviewers
  • CMA Members
  • Media
  • Reprint requests
  • Subscribers

About

  • General Information
  • Journal staff
  • Editorial Board
  • Governance Council
  • Journal Oversight
  • Careers
  • Contact
  • Copyright and Permissions

Copyright 2021, CMA Joule Inc. or its licensors. All rights reserved. ISSN 1488-2329 (e) 0820-3946 (p)

All editorial matter in CMAJ represents the opinions of the authors and not necessarily those of the Canadian Medical Association or its subsidiaries.

To receive any of the resources on this site in an accessible format, please contact us at cmajgroup@cmaj.ca.

Powered by HighWire