Skip to main content

Main menu

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
    • Classified ads
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN

User menu

Search

  • Advanced search
CMAJ
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN
CMAJ

Advanced Search

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
    • Classified ads
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
  • Visit CMAJ on Facebook
  • Follow CMAJ on Twitter
  • Follow CMAJ on Pinterest
  • Follow CMAJ on Youtube
  • Follow CMAJ on Instagram
Commentary

Sources of bias in diagnostic accuracy studies and the diagnostic process

Toshi A. Furukawa and Gordon H. Guyatt
CMAJ February 14, 2006 174 (4) 481-482; DOI: https://doi.org/10.1503/cmaj.060014
Toshi A. Furukawa
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Gordon H. Guyatt
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Tables
  • Related Content
  • Responses
  • Metrics
  • PDF
Loading

Making accurate diagnoses requires knowing the performance characteristics of the tests we use. How can we know whether to trust studies that purport to establish those performance characteristics?

The fundamental design of studies assessing the accuracy of diagnostic tests involves a comparison of the test under investigation with a „gold” or reference standard. In many cases, the gold standard is impractical in the clinical setting because it is too invasive (e.g., pulmonary angiography), is too costly or time-consuming (e.g., long-term follow-up for diagnosis of seizures or syncope) or is possible only after the patient is dead (neuropathological examination for Alzheimer's disease). The test under investigation can be any method of obtaining information from the patient: clinical history taking, physical examination, laboratory investigation, imaging, questionnaire or pathological examination. Investigators assess the accuracy of the test by the extent of its agreement with the gold standard. One can best express this agreement in the form of likelihood ratios, although a number of less satisfactory alternatives (sensitivity and specificity, diagnostic odds ratios, or areas under ROC [receiver-operating-characteristic] curves) are also available.

There are many variations possible to this prototypical diagnostic accuracy study, which may or may not introduce biases (i.e., systematic over-or underestimation of the true accuracy of a diagnostic test). Critical examination of diagnostic accuracy studies started around 1980, and we now have some 90 quality-assessment tools,1 the most notable of which include the STARD (Standards for Reporting of Diagnostic Accuracy)2 and the QUADAS (Quality Assessment of Diagnostic Accuracy Studies).3 The STARD is a checklist of 25 items and flow diagram to improve the reporting of original studies, whereas the QUADAS is a list of 14 checkpoints for assessing the quality of original studies to include them in systematic reviews. The authors of both checklists explicitly acknowledge that our understanding of the determinants of the quality of diagnostic accuracy studies is growing, and they intend to update their products accordingly.

In this issue (page 469), Rutjes and colleagues report on their investigation of 15 potential sources of bias in diagnostic accuracy studies by empirically examining their association with estimates of diagnostic accuracy in each study.4 Their work complements previous empirical works by groups led by Lijmer5 and Whiting.6 Both Rutjes and colleagues4 and Lijmer and associates5 sampled primary studies from systematic reviews, a sampling frame that is likely to be systematically different from the total pool of diagnostic accuracy studies (and may include stronger or weaker original investigations). Whiting and colleagues,6 on the other hand, systematically searched for both primary and secondary studies that examined influences of design features on estimates of diagnostic accuracy; the limitation of their work is that they provided narrative, rather than quantitative, summaries of their results.

The 3 studies have identified a number of design features that consistently appear to introduce bias (see online Appendix 1 at www.cmaj.ca/cgi/content/full/cmaj.060014/DC1). These include poor selection of patients with and without the target diagnosis (spectrum bias); failure to use the same gold standard on all patients, irrespective of their test results (verification bias); and awareness of the results of the gold standard or test by those interpreting the results of the other (lack of blinding). Clinicians should pay special attention to these aspects of diagnostic accuracy studies before deciding to apply the results to their own practice, regardless of whether they are reviewing a single primary study or a systematic review of many such studies.1

The limitations of the 3 reviews suggest that we should not ignore potential sources of bias even when supporting empirical data are lacking. For instance, empirical support for bias associated with a poor choice of gold standard is limited (see online Appendix 1 at www.cmaj.ca/cgi/content/full/cmaj.060014/DC1). Nevertheless, common sense tells us that, if the gold standard lacks reliability or validity, assessments of test properties will be pointless. Malignant melanoma provides a recent example of the unreliability of the accepted gold standard: a nationwide sample of practising pathologists in the United Kingdom agreed only modestly in the diagnosis of malignant melanoma without the use of standardized diagnostic criteria (kappa = 0.45). Satisfactory agreement was achieved when standardized diagnostic criteria were used and some disease categories were modified.7

To establish that studies have provided unbiased estimates of test properties, and that a particular test is indeed accurate, is but one step in the diagnostic process. In order to better serve our patients, we need to generate appropriate differential diagnoses with a sense of the relative likelihood of competing alternatives. Ideally, we will think in quantitative terms, generating estimates of pretest probabilities. In addition, we must understand the sequential application of each diagnostic test. In practice, the diagnostic process proceeds in a stepwise fashion, starting with a particular set of signs and symptoms. Clinical prediction rules or guides, which simultaneously consider a range of diagnostic information, provide one solution to incorporating the results of multiple tests to arrive at an accurate diagnosis.8

@ See related article page 469

Footnotes

  • Competing interests: None declared.

REFERENCES

  1. 1.↵
    Whiting P, Rutjes AW, Dinnes J, et al. A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol 2005;58:1-12.
    OpenUrlCrossRefPubMed
  2. 2.↵
    Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003;49:7-18.
    OpenUrlAbstract/FREE Full Text
  3. 3.↵
    Whiting P, Rutjes AW, Reitsma JB, et al. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 2003;3:25.
    OpenUrlCrossRefPubMed
  4. 4.↵
    Rutjes AWS, Reitsma JB, Di Nisio M, et al. Evidence of bias and variation in diagnostic accuracy studies. CMAJ 2006;174(4):469-76.
    OpenUrlAbstract/FREE Full Text
  5. 5.↵
    Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999;282:1061-6.
    OpenUrlCrossRefPubMed
  6. 6.↵
    Whiting P, Rutjes AW, Reitsma JB, et al. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med 2004;140:189-202.
    OpenUrlCrossRefPubMed
  7. 7.↵
    A nationwide survey of observer variation in the diagnosis of thin cutaneous malignant melanoma including the MIN terminology. CRC Melanoma Pathology Panel. J Clin Pathol 1997;50:202-5.
    OpenUrlAbstract/FREE Full Text
  8. 8.↵
    McGinn T, Guyatt G, Wyer P, et al. Clinical prediction rules. In: Guyatt G, Rennie D, editors. Users' guides to the medical literature: a manual for evidence-based clinical practice. Chicago: AMA Press; 2002.
PreviousNext
Back to top

In this issue

Canadian Medical Association Journal: 174 (4)
CMAJ
Vol. 174, Issue 4
14 Feb 2006
  • Table of Contents
  • Index by author

Article tools

Respond to this article
Print
Download PDF
Article Alerts
To sign up for email alerts or to access your current email alerts, enter your email address below:
Email Article

Thank you for your interest in spreading the word on CMAJ.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Sources of bias in diagnostic accuracy studies and the diagnostic process
(Your Name) has sent you a message from CMAJ
(Your Name) thought you would like to see the CMAJ web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Sources of bias in diagnostic accuracy studies and the diagnostic process
Toshi A. Furukawa, Gordon H. Guyatt
CMAJ Feb 2006, 174 (4) 481-482; DOI: 10.1503/cmaj.060014

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
‍ Request Permissions
Share
Sources of bias in diagnostic accuracy studies and the diagnostic process
Toshi A. Furukawa, Gordon H. Guyatt
CMAJ Feb 2006, 174 (4) 481-482; DOI: 10.1503/cmaj.060014
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Footnotes
    • REFERENCES
  • Figures & Tables
  • Related Content
  • Responses
  • Metrics
  • PDF

Related Articles

  • Highlights of this issue
  • Dans ce numéro
  • Evidence of bias and variation in diagnostic accuracy studies
  • PubMed
  • Google Scholar

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • A “No More Waves” strategy for COVID-19 in Canada
  • The case for relaxing no-visitor policies in hospitals during the ongoing COVID-19 pandemic
  • Transforming race-based health research in Canada
Show more Commentary

Similar Articles

Collections

  • Topics
    • Screening & diagnostic tests
    • Research methods & statistics

Content

  • Current issue
  • Past issues
  • Collections
  • Sections
  • Blog
  • Podcasts
  • Alerts
  • RSS
  • Early releases

Information for

  • Advertisers
  • Authors
  • Reviewers
  • CMA Members
  • Media
  • Reprint requests
  • Subscribers

About

  • General Information
  • Journal staff
  • Editorial Board
  • Governance Council
  • Journal Oversight
  • Careers
  • Contact
  • Copyright and Permissions

Copyright 2021, Joule Inc. or its licensors. All rights reserved. ISSN 1488-2329 (e) 0820-3946 (p)

All editorial matter in CMAJ represents the opinions of the authors and not necessarily those of the Canadian Medical Association or its subsidiaries.

Powered by HighWire