Original Article
A capture-recapture analysis demonstrated that randomized controlled trials evaluating the impact of diagnostic tests on patient outcomes are rare

https://doi.org/10.1016/j.jclinepi.2011.07.003Get rights and content

Abstract

Objective

To estimate the number of randomized controlled trials (RCTs) published annually that evaluate the impact of diagnostic tests on patient outcomes to gauge the extent of available randomized evidence assessing the effectiveness of diagnostic tests.

Study Design and Setting

Relevant RCTs published in 2004–2007 were identified from electronic searches of the Cochrane Central Register of Controlled Trials (CENTRAL). Two search strategies were developed, one using diagnostic methodological terms and one using test names. Potentially relevant RCTs were identified by screening titles and abstracts. Final inclusion decisions were based on full-text review. A random 10% sample of all citations was independently screened by a second reviewer. Capture-recapture methodology was used to estimate the number of relevant RCTs missed by both searches.

Results

One hundred thirty-five relevant RCTs were identified from the 23,888 records retrieved. Interobserver agreement was substantial. Capture-recapture methodology estimated that 148 (95% confidence interval: 140, 160) relevant RCTs were published in the 4-year period, an average of only 37 publications per year.

Conclusion

RCTs of diagnostic tests that evaluate patient outcomes are rare. Consequently recommendations on the use of diagnostic tests can rarely be made on the basis of randomized comparisons, lower grade evidence frequently being the best available.

Introduction

What is new?

  • Test-treatment randomized controlled trials of diagnostic tests are rare: we estimate that only 37 trials were published per year for 2004–2007.

  • Randomized evidence of effectiveness of diagnostic tests is unlikely to be available for many tests in specific clinical settings.

  • Policy and decision makers frequently need to resort to lower grade evidence, such as decision models to provide guidance on test selection and use.

It is recommended that decisions on diagnostic test use should be based on evidence that they do more good than harm in terms of improving patient outcomes. Test-treatment randomized controlled trials (RCTs) are reputed to provide the highest quality evidence for these decisions [1], [2], [3]: patients presenting with a clinical problem are randomized between two or more testing strategies, followed through subsequent diagnosis, management planning, and treatment and evaluated after all interventions have been undertaken.

Guideline developers and systematic reviewers often fail to locate RCTs of diagnostic tests [4], [5], [6], [7], [8], [9], implying that they are rarely conducted [10] or difficult to find, potentially because of the absence or inconsistent use of diagnostic method terms [11]. Search strategies developed to locate diagnostic accuracy studies are known to suffer from the absence of index terms that sort diagnostic evaluations by study type [11]. However, no strategies have yet been designed to locate test-treatment RCTs.

This study aims to estimate the number of test-treatment RCTs published annually.

Section snippets

Methods

The number of published RCTs was estimated by undertaking two different electronic searches and applying capture-recapture methods to estimate the number of studies missed by both searches. The capture-mark-recapture method was developed by ecologists to estimate the size of animal populations that may be difficult to count directly and completely. It has since been applied in health care to estimate population prevalence using multiple incomplete sources [12] to assess publication bias [13]

Results

After elimination of duplicates, the IT and MT strategies, respectively, yielded 10,393 and 13,495 citations, of which 221 and 318 were judged potentially eligible after review of titles and abstracts. In total, 135 test-treatment RCTs were included after full-text review, of which 85 evaluated one of the five selected imaging tests (Table 3). Sixty-two of these were found by both strategies. The MT strategy yielded a higher overall number of relevant test-treatment RCTs than the IT strategy

Discussion

The capture-recapture analysis estimated that only 37 (95% CI: 35, 40) randomized trials that evaluate the impact of diagnostic test strategies on patient outcomes are published per year. This is astoundingly low when compared with the approximate rate of 21,949 per year for all RCTs indexed in CENTRAL (n = 87,794 for 2004–2007). It is therefore unlikely that randomized trial evidence of the impact of tests on patient health is available to inform guidelines on the use of diagnostic tests in many

Acknowledgments

This work was carried out as part of a Methodology Programme Grant (number 78381) awarded to J.D. by the Medical Research Council, UK. The authors thank Sue Bayliss and Anne Fry-Smith of the University of Birmingham for their contributions to developing the MT search strategy.

References (23)

  • S.M. Dyer et al.

    Systematic review of the impact of endoscopic ultrasound on the management of patients with esophageal cancer

    Int J Technol Assess Health Care

    (2008)
  • Cited by (0)

    View full text