Commentary
Diagnostic research on routine care data: Prospects and problems

https://doi.org/10.1016/S0895-4356(03)00080-5Get rights and content

Abstract

A diagnosis in practice is a sequential process starting with a patient with a particular set of signs and symptoms. To serve practice, diagnostic research should aim to quantify the added value of a test to clinical information that is commonly available before the test will be applied. Routine care databases commonly include all documented patient information, and therefore seem to be suitable to quantify a tests' added value to prior information. It is well known, however, that retrospective use of routine care data in diagnostic research may cause various methodologic problems. But, given the increased attention of electronic patient records including data from routine patient care, we believe it is time to reconsider these problems. We discuss four problems related to routine care databases. First, most databases do not label patients by their symptoms or signs but by their final diagnosis. Second, in routine care the diagnostic workup of a patient is by definition determined by previous diagnostic (test) results. Therefore, routinely documented data are subject to so-called workup bias. Third, in practice, the reference test is always interpreted with knowledge of the preceding test information, such that in scientific studies using routine data the diagnostic value of a test under evaluation is commonly overestimated. Fourth, routinely documented databases are likely to suffer from missing data. Per problem we discuss methods that are presently available and may (partly) overcome each problem. All this could contribute to more frequent and appropriate use of routine care data in diagnostic research. The discussed methods to overcome the above problems may well be similarly useful to prospective diagnostic studies.

Introduction

Diagnostic practice is a sequential, stepwise process starting with a patient with a particular set of signs and symptoms. To ascertain or rule out a diagnosis, the physician decides upon additional tests based on his findings in previous steps, to increase or decrease the probability of a particular disease (target disease). Hence, to serve practice, diagnostic research should select patients conform practice, follow the sequential process of making a diagnosis in practice, and should aim to quantify the added value of a test to clinical information that is available before the test would be applied. Although this has been recognized for years [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], at present, the majority of diagnostic studies still include single test studies, aiming to estimate a test's sensitivity and specificity, without considering other (previous) patient information and quantifying the tests' added value [16], [17], [18]. Routine care databases or electronic patient records commonly include all information that is related to patient care. Hence, they include all patient information that is considered relevant to ascertain a diagnosis in routine practice. This makes routinely documented data very well suitable for quantifying the value of a particular test additional to other (previous) information. It is has widely been discussed and illustrated (e.g., in studies on the value of exercise stress testing in diagnosis of coronary artery disease), that studies using retrospective patient data as obtained from routine care, provide invalid results. This particularly includes the problem of selection (or also called referral, workup or verification) bias [2], [4], [5], [7], [13], [17], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31]. However, the use of large electronic databases or patient records in medical practice is still increasing, notably in general practice [32], [33]. We, therefore, believe it is time to reconsider the methodologic disadvantages of diagnostic research based on routine care data and potential solutions to overcome them.

In this article, we start from the nature of diagnosis in practice and the preferred design for quantification of (added) value of diagnostic tests. Subsequently, we summarize the well-known problems that can be encountered in diagnostic research using routine care data. Per problem, we discuss the currently available and sometimes novel methods that may (partly) solve the problem. We believe that these methods could contribute to more frequent and appropriate use of routine care data in diagnostic research. For illustration purposes, we will use throughout an example study that aimed to quantify the (added) value of various diagnostic tests in children suspected of having bacterial meningitis [34], [35].

Section snippets

Example study

At the emergency department of the Sophia Children's Hospital, Rotterdam, The Netherlands, we performed a study on children visiting the emergency department because of meningeal signs [34], [35]. These children pose a diagnostic dilemma for the physician, because they are at risk of bacterial meningitis (target disease [23]), but they may have self-limiting diseases in 50%–60% [35], [36], [37]. The question is in which of these children a lumbar puncture should be performed and empirical

Diagnostic practice and diagnostic research

In practice, a diagnosis starts with a patient with a clinical problem (symptoms or signs) suspected of having a particular disease, the so-called target disease [2], [6], [9], [16], [23]. As shown in Fig. 1, the physician commonly follows a phased workup, starting with patient history and physical examination. Subsequent steps may include additional laboratory tests, imaging, and finally so-called gold standard or reference tests such as arthroscopy, angiography or, as in our example, a lumbar

Selection of the proper patient population

In scientific diagnostic studies or test evaluations, patients are selected on the established presence or absence of a particular disease, that is, patients that have undergone the so-called gold standard or reference test in routine care are selected for the study. Similarly, studies aiming to quantify the diagnostic accuracy of a particular test often include only those patients that have undergone that test in routine care. Both types of patient selection for diagnostic studies, however,

Conclusions

The described problems encountered in diagnostic studies using routine care data and the methods to handle them are not limited to retrospective diagnostic studies only. They may as well apply to diagnostic research based on prospectively documented data. As diagnostic research is pragmatic prediction research, it should always reflect the sequential diagnostic workup of medical practice to allow for clinically meaningful inferences. It should start with patients selected on their disease

Acknowledgements

We gratefully acknowledge the reviewers for their comments, which made a great contribution to the merits of this article. All authors worked together within a large research project on the evaluation of diagnostic procedures in pediatrics, which was completely financed by a grant from the Health Care Insurance Counsel of The Netherlands.

References (66)

  • D.F Ransohoff et al.

    Problems of spectrum and bias in evaluating the efficacy of diagnostic tests

    N Engl J Med

    (1978)
  • A.R Feinstein

    Clinical epidemiology: the architecture of clinical research

    (1985)
  • J.H Wasson et al.

    Clinical prediction rules. Applications and methodological standards

    N Engl J Med

    (1985)
  • C.B Begg

    Biases in the assessment of diagnostic tests

    Stat Med

    (1987)
  • D.G Fryback et al.

    The efficacy of diagnostic imaging

    Med Decis Making

    (1991)
  • J.A Knottnerus et al.

    The influence of referral patterns on the characteristics of diagnostic tests

    J Clin Epidemiol

    (1992)
  • J.A Knottnerus

    Application of logistic regression to the analysis of diagnostic data: exact modeling of a probability tree of multiple binary variables

    Med Decis Making

    (1992)
  • D.L Sackett

    A primer on the precision and accuracy of the clinical examination

    JAMA

    (1992)
  • R Jaeschke et al.

    Users' guides to the medical literatureIII: How to use an article about a diagnostic tests. A. Are the results of the study valid?

    JAMA

    (1994)
  • L Dalla-Palma et al.

    An overview of cost-effective radiology

    Eur Radiol

    (1997)
  • K.G.M Moons et al.

    Limitations of sensitivity, specificity, likelihood ratio, and Bayes' theorem in assessing diagnostic probabilities: a clinical example

    Epidemiology

    (1997)
  • K.G.M Moons et al.

    Reduncancy of single diagnostic test evaluation

    Epidemiology

    (1999)
  • M.C Reid et al.

    Use of methodological standards in diagnostic test research. Getting better but still not good

    JAMA

    (1995)
  • J.G Lijmer et al.

    Empirical evidence of design-related bias in studies of diagnostic tests

    JAMA

    (1999)
  • C.B Begg et al.

    Assessment of diagnostic tests when disease verification is subject to selection bias

    Biometrics

    (1983)
  • R.A Greenes et al.

    Patient-oriented performance measures of diagnostic tests1. Tools for prospective evaluation of test order decisions

    Med Decis Making

    (1984)
  • R.A Greenes et al.

    Patient-oriented performance measures of diagnostic tests2. Assignment potential and assignment strength

    Med Decis Making

    (1984)
  • D.L Sackett et al.

    Clinical epidemiology; a basic science for clinical medicine

    (1985)
  • C.B Begg

    Statistical methods in medical diagnosis

    Crit Rev Med Inform

    (1986)
  • R.H Fletcher

    Carcinoembryonic antigen

    Ann Intern Med

    (1986)
  • J.A Knottnerus

    The effect of disease verification and referral on the relationship between symptoms and diseases

    Med Decis Making

    (1987)
  • R.J Panzer et al.

    Workup bias in prediction research

    Med Decis Making

    (1987)
  • S.S Coughlin et al.

    The logistic modelling of sensitivity, specificity and predictive value of a diagnostic test

    J Clin Epidemiol

    (1992)
  • Cited by (47)

    • The question

      2023, Translational Sports Medicine
    • Diagnostic accuracy of smart gadgets/wearable devices in detecting atrial fibrillation: A systematic review and meta-analysis

      2021, Archives of Cardiovascular Diseases
      Citation Excerpt :

      For example, different population groups included in each type of study may play an important role, conferring variable accuracy in AF diagnosis. Overestimation of diagnostic performance can be compounded by retrospective data collection, as ambiguous inclusion criteria may be inadvertently exploited [40]. In a case-control study type, overestimation of diagnostic performance may ensue largely as a result of the selection process, which includes severe cases and a healthy population, thereby increasing specificity [41].

    • Development of practical recommendations for diagnostic accuracy studies in low-prevalence situations

      2019, Journal of Clinical Epidemiology
      Citation Excerpt :

      Finally, routine care data do not allow the evaluation of innovative or experimental tests that are not part of routine clinical practice. For evaluating medical tests, these issues could lead to verification bias, information bias, and applicability concerns [37]. Assumptions and clinical judgment will be required to judge the impact of these issues and select appropriate methods to deal with them.

    • Linking NHS data for pediatric pharmacovigilance: Results of a Delphi survey

      2016, Research in Social and Administrative Pharmacy
      Citation Excerpt :

      Attempts to implement new initiatives for identification or reporting of ADRs are likely to fail if potential barriers are not addressed.21 Whilst data linkage for pharmacovigilance would not contravene current ethical, legal and practical guidance,22–25 it is unclear whether frontline HCPs would support the use of pediatric clinical data for pharmacovigilance. Two qualitative studies conducted with HCPs prior to the work reported in this paper26,27 found that in general views of the proposed data linkage were positive, but there were also many potential concerns such as confidentiality and access to data.

    View all citing articles on Scopus
    View full text