Advertisement
ADVANCED SEARCH

This Article Figures Only Full Text Full Text (PDF) Abridged Version Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Rutjes, A. W.S. Articles by Bossuyt, P. M.M. Search for Related Content PubMed PubMed Citation Articles by Rutjes, A. W.S. Articles by Bossuyt, P. M.M. Related Collections Statistics & Research Methods Diagnostic tests Related Article

Evidence of bias and variation in diagnostic accuracy studies

From the Deptartment of Clinical Epidemiology & Biostatistics(Rutjes, Reitsma, van Rijn, Bossuyt), Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands ; the Department of Medicine and Aging, School of Medicine, and Aging Research Center (Nisio), Ce.S.I., Gabriele D'Annunzio University Foundation, Chieti-Pescara, Via dei Vestini Chieti-Pescara, Italy; and the Institute for Research in Extramural Medicine (Smidt), VU University Medical Center, Amsterdam, Amsterdam, the Netherlands.

Correspondence to: Dr. Anne W.S. Rutjes, Department of Clinical Pharmacology and Epidemiology, Consorzio Mario Negri Sud, Via Nazionale 8, 66030 Santa Maria Imbaro, Chieti, Italy; fax +39 087 2570206

Background: Studies with methodologic shortcomings can overestimate the accuracy of a medical test. We sought to determine and compare the direction and magnitude of the effects of a number of potential sources of bias and variation in studies on estimates of diagnostic accuracy.

Methods: We identified meta-analyses of the diagnostic accuracy of tests through an electronic search of the databases MEDLINE, EMBASE, DARE and MEDION (1999–2002). We included meta-analyses with at least 10 primary studies without preselection based on design features. Pairs of reviewers independently extracted study characteristics and original data from the primary studies. We used a multivariable meta-epidemiologic regression model to investigate the direction and strength of the association between 15 study features on estimates of diagnostic accuracy.

Results: We selected 31 meta-analyses with 487 primary studies of test evaluations. Only 1 study had no design deficiencies. The quality of reporting was poor in most of the studies. We found significantly higher estimates of diagnostic accuracy in studies with nonconsecutive inclusion of patients (relative diagnostic odds ratio [RDOR] 1.5, 95% confidence interval [CI] 1.0–2.1) and retrospective data collection (RDOR 1.6, 95% CI 1.1–2.2). The estimates were highest in studies that had severe cases and healthy controls (RDOR 4.9, 95% CI 0.6–37.3). Studies that selected patients based on whether they had been referred for the index test, rather than on clinical symptoms, produced significantly lower estimates of diagnostic accuracy (RDOR 0.5, 95% CI 0.3–0.9). The variance between meta-analyses of the effect of design features was large to moderate for type of design (cohort v. case–control), the use of composite reference standards and the use of differential verification; the variance was close to zero for the other design features.

Interpretation: Shortcomings in study design can affect estimates of diagnostic accuracy, but the magnitude of the effect may vary from one situation to another. Design features and clinical characteristics of patient groups should be carefully considered by researchers when designing new studies and by readers when appraising the results of such studies. Unfortunately, incomplete reporting hampers the evaluation of potential sources of bias in diagnostic accuracy studies.

This article has been cited by other articles:

				B. B. Meserve, J. A. Cleland, and T. R. Boucher A Meta-analysis Examining Clinical Test Utility for Assessing Superior Labral Anterior Posterior Lesions Am. J. Sports Med., November 1, 2009; 37(11): 2252 - 2258. [Abstract] [Full Text] [PDF]

				J. P.A. Ioannidis Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses Can. Med. Assoc. J., October 13, 2009; 181(8): 488 - 493. [Full Text] [PDF]

				T. A. Trikalinos, U. Siebert, and J. Lau Decision-Analytic Modeling to Evaluate Benefits and Harms of Medical Tests: Uses and Limitations Med Decis Making, September 1, 2009; 29(5): E22 - E29. [Abstract] [PDF]

				M. M.G. Leeflang, J. J. Deeks, C. Gatsonis, P. M.M. Bossuyt, and on behalf of the Cochrane Diagnostic Test Accuracy Systematic Reviews of Diagnostic Test Accuracy Ann Intern Med, December 16, 2008; 149(12): 889 - 897. [Abstract] [Full Text] [PDF]

				H. J Schunemann, A. D Oxman, J. Brozek, P. Glasziou, P. Bossuyt, S. Chang, P. Muti, R. Jaeschke, and G. H Guyatt GRADE: assessing the quality of evidence for diagnostic recommendations Evid. Based Med., December 1, 2008; 13(6): 162 - 163. [Full Text] [PDF]

				G. Mowatt, J. M. Burr, J. A. Cook, M. A. R. Siddiqui, C. Ramsay, C. Fraser, A. Azuara-Blanco, J. J. Deeks, and for the OAG Screening Project Screening Tests for Detecting Open-Angle Glaucoma: Systematic Review and Meta-analysis Invest. Ophthalmol. Vis. Sci., December 1, 2008; 49(12): 5373 - 5385. [Abstract] [Full Text] [PDF]

				D. I. Ling, A. A. Zwerling, and M. Pai GenoType MTBDR assays for the diagnosis of multidrug-resistant tuberculosis: a meta-analysis Eur. Respir. J., November 1, 2008; 32(5): 1165 - 1174. [Abstract] [Full Text] [PDF]

				B. H Willis Spectrum bias--why clinicians need to be cautious when applying diagnostic test studies Fam. Pract., October 1, 2008; 25(5): 390 - 396. [Abstract] [Full Text] [PDF]

				A. van Randen, S. Bipat, A. H. Zwinderman, D. T. Ubbink, J. Stoker, and M. A. Boermeester Acute Appendicitis: Meta-Analysis of Diagnostic Performance of CT and Graded Compression US Related to Prevalence of Disease Radiology, October 1, 2008; 249(1): 97 - 106. [Abstract] [Full Text] [PDF]

				N. L. Wilczynski Quality of Reporting of Diagnostic Accuracy Studies: No Change Since STARD Statement Publication--Before-and-after Study Radiology, September 1, 2008; 248(3): 817 - 823. [Abstract] [Full Text] [PDF]

				P. A. Kyzas, E. Evangelou, D. Denaxa-Kyza, and J. P. A. Ioannidis 18F-Fluorodeoxyglucose Positron Emission Tomography to Evaluate Cervical Node Metastases in Patients With Head and Neck Squamous Cell Carcinoma: A Meta-analysis J Natl Cancer Inst, May 21, 2008; 100(10): 712 - 720. [Abstract] [Full Text] [PDF]

				H. J Schunemann, A. D Oxman, J. Brozek, P. Glasziou, R. Jaeschke, G. E Vist, J. W Williams Jr, R. Kunz, J. Craig, V. M Montori, et al. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies BMJ, May 17, 2008; 336(7653): 1106 - 1110. [Full Text] [PDF]

				J. S Cnossen, K. C Vollebregt, N. d. Vrieze, G. t. Riet, B. W J Mol, A. Franx, K. S Khan, and J. A M v. d. Post Accuracy of mean arterial pressure and blood pressure measurements in predicting pre-eclampsia: systematic review and meta-analysis BMJ, May 17, 2008; 336(7653): 1117 - 1120. [Abstract] [Full Text] [PDF]

				K. Horsthuis, S. Bipat, R. J. Bennink, and J. Stoker Inflammatory Bowel Disease Diagnosed with US, MR, Scintigraphy, and CT: Meta-analysis of Prospective Studies Radiology, April 1, 2008; 247(1): 64 - 79. [Abstract] [Full Text] [PDF]

				T. J. Selman, C. Mann MD, J. Zamora PhD, T.-L. Appleyard MBBS, and K. Khan MSc Diagnostic accuracy of tests for lymph node status in primary cervical cancer: a systematic review and meta-analysis Can. Med. Assoc. J., March 25, 2008; 178(7): 855 - 862. [Abstract] [Full Text] [PDF]

				J. S. Cnossen MD, R. K. Morris MD, G. ter Riet MD PhD, B. W.J. Mol MD PhD, J. A.M. van der Post MD PhD, A. Coomarasamy MD, A. H. Zwinderman MSc PhD, S. C. Robson MD, P. J.E. Bindels MD PhD, J. Kleijnen MD PhD, et al. Use of uterine artery Doppler ultrasonography to predict pre-eclampsia and intrauterine growth restriction: a systematic review and bivariable meta-analysis Can. Med. Assoc. J., March 11, 2008; 178(6): 701 - 711. [Abstract] [Full Text] [PDF]

				B. B. Meserve, J. A. Cleland, and T. R. Boucher A meta-analysis examining clinical test utilities for assessing meniscal injury Clinical Rehabilitation, February 1, 2008; 22(2): 143 - 161. [Abstract] [PDF]

				E J Hegedus, A Goode, S Campbell, A Morin, M Tamaddoni, C T Moorman III, and C Cook Physical examination tests of the shoulder: a systematic review with meta-analysis of individual tests Br. J. Sports Med., February 1, 2008; 42(2): 80 - 92. [Abstract] [Full Text] [PDF]

				M. Leeflang, J. Reitsma, R. Scholten, A. Rutjes, M. Di Nisio, J. Deeks, and P. Bossuyt Impact of Adjustment for Quality on Results of Metaanalyses of Diagnostic Accuracy Clin. Chem., February 1, 2007; 53(2): 164 - 172. [Abstract] [Full Text] [PDF]

				S. Mallett, J. J Deeks, S. Halligan, S. Hopewell, V. Cornelius, and D. G Altman Systematic reviews of diagnostic tests in cancer: review of methods and reporting BMJ, August 26, 2006; 333(7565): 413. [Abstract] [Full Text] [PDF]

				T. A. Furukawa and G. H. Guyatt Sources of bias in diagnostic accuracy studies and the diagnostic process. Can. Med. Assoc. J., February 14, 2006; 174(4): 481 - 482. [Full Text] [PDF]

HOME	CURRENT ISSUE	PAST ISSUES	COLLECTIONS	HELP	SEARCH
Copyright 1995-2006, Canadian Medical Association. All rights reserved. ISSN 1488-2329 (e) 0820-3946 (p) All editorial matter in CMAJ represents the opinions of the authors and not necessarily those of the Canadian Medical Association.