CommentaryDiagnostic research on routine care data: Prospects and problems
Introduction
Diagnostic practice is a sequential, stepwise process starting with a patient with a particular set of signs and symptoms. To ascertain or rule out a diagnosis, the physician decides upon additional tests based on his findings in previous steps, to increase or decrease the probability of a particular disease (target disease). Hence, to serve practice, diagnostic research should select patients conform practice, follow the sequential process of making a diagnosis in practice, and should aim to quantify the added value of a test to clinical information that is available before the test would be applied. Although this has been recognized for years [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], at present, the majority of diagnostic studies still include single test studies, aiming to estimate a test's sensitivity and specificity, without considering other (previous) patient information and quantifying the tests' added value [16], [17], [18]. Routine care databases or electronic patient records commonly include all information that is related to patient care. Hence, they include all patient information that is considered relevant to ascertain a diagnosis in routine practice. This makes routinely documented data very well suitable for quantifying the value of a particular test additional to other (previous) information. It is has widely been discussed and illustrated (e.g., in studies on the value of exercise stress testing in diagnosis of coronary artery disease), that studies using retrospective patient data as obtained from routine care, provide invalid results. This particularly includes the problem of selection (or also called referral, workup or verification) bias [2], [4], [5], [7], [13], [17], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31]. However, the use of large electronic databases or patient records in medical practice is still increasing, notably in general practice [32], [33]. We, therefore, believe it is time to reconsider the methodologic disadvantages of diagnostic research based on routine care data and potential solutions to overcome them.
In this article, we start from the nature of diagnosis in practice and the preferred design for quantification of (added) value of diagnostic tests. Subsequently, we summarize the well-known problems that can be encountered in diagnostic research using routine care data. Per problem, we discuss the currently available and sometimes novel methods that may (partly) solve the problem. We believe that these methods could contribute to more frequent and appropriate use of routine care data in diagnostic research. For illustration purposes, we will use throughout an example study that aimed to quantify the (added) value of various diagnostic tests in children suspected of having bacterial meningitis [34], [35].
Section snippets
Example study
At the emergency department of the Sophia Children's Hospital, Rotterdam, The Netherlands, we performed a study on children visiting the emergency department because of meningeal signs [34], [35]. These children pose a diagnostic dilemma for the physician, because they are at risk of bacterial meningitis (target disease [23]), but they may have self-limiting diseases in 50%–60% [35], [36], [37]. The question is in which of these children a lumbar puncture should be performed and empirical
Diagnostic practice and diagnostic research
In practice, a diagnosis starts with a patient with a clinical problem (symptoms or signs) suspected of having a particular disease, the so-called target disease [2], [6], [9], [16], [23]. As shown in Fig. 1, the physician commonly follows a phased workup, starting with patient history and physical examination. Subsequent steps may include additional laboratory tests, imaging, and finally so-called gold standard or reference tests such as arthroscopy, angiography or, as in our example, a lumbar
Selection of the proper patient population
In scientific diagnostic studies or test evaluations, patients are selected on the established presence or absence of a particular disease, that is, patients that have undergone the so-called gold standard or reference test in routine care are selected for the study. Similarly, studies aiming to quantify the diagnostic accuracy of a particular test often include only those patients that have undergone that test in routine care. Both types of patient selection for diagnostic studies, however,
Conclusions
The described problems encountered in diagnostic studies using routine care data and the methods to handle them are not limited to retrospective diagnostic studies only. They may as well apply to diagnostic research based on prospectively documented data. As diagnostic research is pragmatic prediction research, it should always reflect the sequential diagnostic workup of medical practice to allow for clinically meaningful inferences. It should start with patients selected on their disease
Acknowledgements
We gratefully acknowledge the reviewers for their comments, which made a great contribution to the merits of this article. All authors worked together within a large research project on the evaluation of diagnostic procedures in pediatrics, which was completely financed by a grant from the Health Care Insurance Counsel of The Netherlands.
References (66)
Selection bias and the evaluation of diagnostic tests: a metadissent
J Chronic Dis
(1986)- et al.
Clinical epidemiology introduction to the discipline
Neth J Med
(1995) - et al.
Measuring the effects of imaging: an evaluative framework
Clin Radiol
(1995) - et al.
Problems in selecting the adequate patient population from existing data files for assessment studies of new diagnostic tests
J Clin Epidemiol
(1995) - et al.
Factors affecting sensitivity and specificity of exercise electrocardiography. Multivariable analysis
Am J Med
(1984) - et al.
A model for assessing the sensitivity and specificity of tests subject to selection bias. Application to exercise radionuclide ventriculography for diagnosis of coronary artery disease
J Chronic Dis
(1986) - et al.
Factors affecting sensitivity and specificity of a diagnostic test: the exercise thallium scintigram
Am J Med
(1988) - et al.
Evaluation of diagnostic imaging tests: diagnostic probability estimation
J Clin Epidemiol
(1998) - et al.
Verification bias in pediatric studies evaluating diagnostic tests
J Pediatr
(1993) - et al.
A simulation study of the number of events per variable in logistic regression analysis
J Clin Epidemiol
(1996)
Problems of spectrum and bias in evaluating the efficacy of diagnostic tests
N Engl J Med
Clinical epidemiology: the architecture of clinical research
Clinical prediction rules. Applications and methodological standards
N Engl J Med
Biases in the assessment of diagnostic tests
Stat Med
The efficacy of diagnostic imaging
Med Decis Making
The influence of referral patterns on the characteristics of diagnostic tests
J Clin Epidemiol
Application of logistic regression to the analysis of diagnostic data: exact modeling of a probability tree of multiple binary variables
Med Decis Making
A primer on the precision and accuracy of the clinical examination
JAMA
Users' guides to the medical literatureIII: How to use an article about a diagnostic tests. A. Are the results of the study valid?
JAMA
An overview of cost-effective radiology
Eur Radiol
Limitations of sensitivity, specificity, likelihood ratio, and Bayes' theorem in assessing diagnostic probabilities: a clinical example
Epidemiology
Reduncancy of single diagnostic test evaluation
Epidemiology
Use of methodological standards in diagnostic test research. Getting better but still not good
JAMA
Empirical evidence of design-related bias in studies of diagnostic tests
JAMA
Assessment of diagnostic tests when disease verification is subject to selection bias
Biometrics
Patient-oriented performance measures of diagnostic tests1. Tools for prospective evaluation of test order decisions
Med Decis Making
Patient-oriented performance measures of diagnostic tests2. Assignment potential and assignment strength
Med Decis Making
Clinical epidemiology; a basic science for clinical medicine
Statistical methods in medical diagnosis
Crit Rev Med Inform
Carcinoembryonic antigen
Ann Intern Med
The effect of disease verification and referral on the relationship between symptoms and diseases
Med Decis Making
Workup bias in prediction research
Med Decis Making
The logistic modelling of sensitivity, specificity and predictive value of a diagnostic test
J Clin Epidemiol
Cited by (47)
The question
2023, Translational Sports MedicineDiagnostic accuracy of smart gadgets/wearable devices in detecting atrial fibrillation: A systematic review and meta-analysis
2021, Archives of Cardiovascular DiseasesCitation Excerpt :For example, different population groups included in each type of study may play an important role, conferring variable accuracy in AF diagnosis. Overestimation of diagnostic performance can be compounded by retrospective data collection, as ambiguous inclusion criteria may be inadvertently exploited [40]. In a case-control study type, overestimation of diagnostic performance may ensue largely as a result of the selection process, which includes severe cases and a healthy population, thereby increasing specificity [41].
It is time to harmonize clinical MEG practice internationally
2020, Clinical NeurophysiologyDevelopment of practical recommendations for diagnostic accuracy studies in low-prevalence situations
2019, Journal of Clinical EpidemiologyCitation Excerpt :Finally, routine care data do not allow the evaluation of innovative or experimental tests that are not part of routine clinical practice. For evaluating medical tests, these issues could lead to verification bias, information bias, and applicability concerns [37]. Assumptions and clinical judgment will be required to judge the impact of these issues and select appropriate methods to deal with them.
Linking NHS data for pediatric pharmacovigilance: Results of a Delphi survey
2016, Research in Social and Administrative PharmacyCitation Excerpt :Attempts to implement new initiatives for identification or reporting of ADRs are likely to fail if potential barriers are not addressed.21 Whilst data linkage for pharmacovigilance would not contravene current ethical, legal and practical guidance,22–25 it is unclear whether frontline HCPs would support the use of pediatric clinical data for pharmacovigilance. Two qualitative studies conducted with HCPs prior to the work reported in this paper26,27 found that in general views of the proposed data linkage were positive, but there were also many potential concerns such as confidentiality and access to data.
The comprehensive diagnostic study: A new solution to old problems?
2014, Journal of Clinical Epidemiology