Table 1:

Limitations of RCTs and whether they can be amended by RCD studies

Potential limitations of RCTsWhat RCD can offerChallenges and remaining caveats in RCD studies
Substantial improvement by an amended RCT agenda
Understudied health care questionsNo direct comparison of relevant treatments or use of pragmatic patient-important outcomes (i.e., mortality); possible and feasible to conduct but not prioritizedSelection of almost any research topic and treatment comparison and of many patient-important clinical events and mortalitySome outcomes typically require deviation from routine care (e.g., evaluation of patient-reported outcomes, such as pain or quality of life, by surveying patients) and are often unavailable.
Data access/publication biasData often generated and collected by industrial sponsors without sharing raw data; reproduction often impossible without infrastructureUnknownAccess issues and publication bias likely do not improve with studies using RCD when compared with RCTs.
Considerable improvement by an amended RCT agenda
Generalizability and real-world relevanceStudy populations differ from real-world target population because inclusion and exclusion criteria are too strict; treatment circumstances differ from routine care because of trial setting.Liberal inclusion criteria; real-world data with high external validity; no interference with routine careSome outcomes typically require deviation from routine care. External validity not necessarily high when collection of data depends on other factors (e.g., collected in tertiary centres, for patients with certain insurance plans).
Specific conditions/subgroup effectsPatients from specific demographic populations or patients with complex conditions are often underrepresented.Large populations; liberal inclusion criteriaImportance of subgroup claims and consequences are frequently unclear and might be overrated; high risk of false-positive findings
Conflicts of interest/sponsorship biasEvidence generated, analyzed and published by researchers or trial sponsors who have an economic conflict of interest; almost always for novel drugsUnknown, often fewer conflicts and nonconflicted sponsorsFinancial and scientific conflicts due to strong beliefs or preconceived hypotheses may be prominent even for analyses using RCD.
Modest improvement by an amended RCT agenda
CostsLogistic costs, and efforts for data generation and collectionMuch lower costs for data generation and collectionHigh investments in data infrastructures and maintenance, although some are not directly research-related (e.g., for electronic health records). Nonstandardized efforts that are often fragmented across teams waste resources, increase costs and create false leads that further waste resources.
SpeedTime needed for planning, protocol development, regulatory issues, and patient recruitment; time of follow-up until outcomes are observedNo need to wait for outcomes in analyses of existing data; time for prespecification not required for exploratory analyses; analyses can be run by small teams or one investigatorLack of prespecification and protocols may reduce validity because of increased risk of findings that are false-positive or false-negative and bias (e.g., selective reporting bias and modelling biases). Thorough reflections about research and involvement of larger teams may improve initial research plans and provide a wider perspective and increase research usefulness and value.
RegulationsRandomization requires ethical and regulatory approval and the process can be cumbersome.No or fewer requirements for ethical and/or regulatory approvalLess oversight; more opportunities for unnoticed errors and biases
Late outcomesLength of follow-up too short for detecting long-term effectsLong observation periodsMissing data; no consistent outcome ascertainment across patients; crossover; poor adherence common with long-term follow-up
Modest improvement by an amended RCT agenda
Uncommon conditionsTrial populations small; recruitment difficultRecruitment usually not difficult; very diverse treatment settings because of liberal inclusion criteria and large populationsConfounding by indication; referral biases
Minimal improvement by an amended RCT agenda
Uncommon outcomesInsufficient statistical power for detecting effects on uncommon outcomes because trial populations are too small or follow-up is too short.Many events because of large populations and long observation periodsSpurious findings and significant findings that are false-positive because of overpowered studies and lack of analytical safeguards. High risk of confounding could also lead to spurious nullification of true treatment effects (false-negative results).
Superseded/outdated/unusual treatment setting or unfeasible conductOutdated or unusual circumstances under which existing RCTs were conducted (e.g., no modern background treatments); new trials can be done, but they would be expensive and take a long time. Perceived disadvantage of one treatment making recruitment of patients difficult.Gathering relevant data on very diverse treatment settings is feasible because of liberal inclusion criteria and large populations.High risk for confounding by indication (i.e., strong indications required for treatments that are superseded or perceived as inferior); generalizability limited to settings with similar circumstances
No improvement by an amended RCT agenda
Unethical conductProven disadvantage or anticipated harm with one treatment (lack of equipoise), making randomization unethical18Size of the disadvantage or harm can be documentedIf the treatment is clearly inferior, maybe it may not have been used even in RCD settings, and it would be of no or little clinical relevance.
  • Note: RCD = routinely collected data, RCT = randomized controlled trial.