METHODS

There were four phases to the work:

• establishment of criteria for inclusion of clinical databases;

• development of a checklist for assessing the quality of clinical databases;

• empirical testing of the checklist; and

• design of a website.

The establishment of criteria to determine which clinical databases would be included was undertaken by an expert group comprising health services epidemiologists and statisticians, clinicians (from intensive care, nephrology, cardiac surgery, obstetrics and gynaecology, haematology, general surgery, spinal injuries), research funders, and information specialists (see Acknowledgements).

The development of the quality assessment checklist was based on the approach used to develop CONSORT, the checklist assessing the quality of randomised trials.⁸ Consensus was sought through a series of meetings of the expert group during 1999 and 2000. Agreement within this diverse group would be an indication of the content validity of the draft instrument. The checklist had to cover all methodological threats to the reliability, internal validity, and generalisability of a database. It also had to be straightforward enough to be of practical value and applicable to a wide variety of databases.

During spring 2001 the instrument was tested by a Masters level graduate in epidemiology on a sample of 20 clinical databases selected to represent the range of those available. Our objectives were to assess the face validity, its acceptability to database custodians, and the existence of floor and ceiling effects resulting from our definitions of the four levels of each item. Following modifications the instrument was retested by the same person reapplying it to the 20 databases.

Finally, a website was designed that described each clinical database and its management and provided an independent assessment of its quality (using the checklist).

RESULTS

Scope of databases included

The scope of the databases that would be eligible for inclusion was restricted to those which included information on the recipients of health care. We excluded databases in which information was limited to the provision of resources or services such as a register of hospital bed provision, useful though such data are in studying and managing health services. We included databases—based on retrospectively and prospectively collected data—that met the following criteria:

• provision of individual level data (whether or not users of the database are permitted to know the identity of individuals);

• inclusion in the database is defined by a common circumstance (for example, individual’s condition, intervention required or undergone (which might be a diagnostic test, treatment, or a collection of interventions such as intensive care)), by an administrative arrangement (for example, registered with a general practitioner, target for immunisation, subscriber to health insurance), or by an adverse outcome (such as maternal death);

• inclusion of data from more than one provider of health care (usually many providers in a region or country).

As our concern was health services, we excluded databases of cohorts defined by an environmental exposure, although such data are useful for aetiological research.

Methodological considerations for assessing quality

Two main methodological concerns were identified by the expert group when selecting a database—its coverage and its accuracy.

A potential user needs to know about four aspects of coverage to determine whether or not a database is suitable for the proposed task:

• how representative it is of the country as a whole;

• the extent to which all eligible individuals have been included;

• the extent of the dataset on each individual; and

• the completeness of data collection.

Similarly, accuracy can be defined in terms of six aspects:

• the form in which quantitative data are collected;

• the use of explicit definitions of variables;

• the use of explicit rules for data collection;

• reliability of data coding;

• the independence of observations of outcomes; and

• the extent to which data are validated.

For each of these 10 items we defined four possible levels of attainment: level 1 describes the simplest and least rigorous, level 4 the most sophisticated and most rigorous.

Empirical testing and modification

The draft instrument was tested by interviewing the custodians of 20 databases (representing a range of areas of health care and geographical coverage). As a result, the face validity was improved. Several terms had to be clarified such as the distinction between immediate and long term outcomes. The attempt to distinguish between mandatory and optional variables included in the database had to be abandoned in the face of lack of clarity on the part of database custodians. For some items the boundaries between the four levels had to be altered to increase the sensitivity. Only one item displayed floor and ceiling effects—the reliability of coding of conditions and interventions was either “not tested” or, if tested, was “good”.

The modified instrument (table 1) was retested by reapplying to the same 20 databases and was found to be less ambiguous and therefore easier to administer. In general, the spread of results was better although the mean level for each item was largely unchanged. To ensure improvements in face validity were maintained, an instruction manual was written containing clear definitions of all terms and explicit rules for determining the level of each item.

Creation of the website

The website (www.docdat.org) was successfully created with database users’ needs in mind. It allows visitors to search for and identify databases that may be suitable for their need, whether that be evaluative research, clinical audit, supporting shared decision making models, or strategic planning of services. A facility exists to search on the basis of one or more of: body system; pathogenesis; health care intervention; age group; and geographical area. Each database entry is designed to achieve three objectives: (1) to inform potential users as to the scope and management of a database (inclusion criteria, geographical area and time period covered, dataset, management of security and confidentiality, outputs); (2) how it can be accessed (contact details of custodian); and (3) its methodological quality (using the new instrument). All of this information has to be obtained by a trained interviewer to ensure an independent assessment is obtained.

The information provided on the coverage and accuracy of the identified databases will enable an assessment to be made as to their suitability—for example, evaluative research might require information on confounders to be included in a database to allow adequate risk adjustment of clinical outcomes. In contrast, a simpler database would be adequate to compare the rates of use of a particular intervention in different populations. Examples of possible applications are shown in box 1. The website enables a user to compare potentially relevant databases by means of “at a glance” histograms (fig 1).

DoCDat only provides an overview of each clinical database. Clearly, despite careful enquiries by the assessor, the validity of the information reported on the website is dependent on the knowledge and views of the custodian of the database. A warning to this effect is included, together with a request for users of DoCDat to let us know of any concerns they have about the accuracy of our assessments. If users wish to delve deeper, it is necessary to find out more from the database custodian whose contact details are provided in the DoCDat entry.

The only feasible means of judging the initial impact of the website is by a process measure—the frequency of visitors to the site. The site was launched in July 2001. Although there has been little effort to market the site, by November 2002 there were around 3500 visits a month.

DISCUSSION

An instrument for describing the quality of a clinical database was successfully developed. For the first time a practical standardised method exists for summarising database quality and this, in turn, enables meaningful comparisons to be made. In addition, a website has been established that provides a simple free means for potential database users to search for and select suitable sources to meet their needs.

The number of databases included in DoCDat will be expanded over the next few years. Following the deliberate policy of initially including databases from a wide variety of clinical settings during the initial development phase, we have focused on areas of national priority in the UK commencing with coronary heart disease and cancer. By February 2003 over 90 databases had been included (see table 2 on the QSHC website www.qshc.com/supplemental). In parallel, it is hoped to include databases from outside the UK, following interest from other countries. Alternatively, people may wish to collaborate and establish DoCDat in their own country. An international network could then be established to link the websites.

While adding more databases is the priority, it is also essential to update and maintain all the entries. This is being done by requesting information of changes from database custodians as they are instituted and by an annual enquiry initiated by DoCDat staff.

A further development underway is to modify the website for local use (so called “Local DoCDat”). Hospital managers are often aware that their clinicians have established their own clinical databases but are unsure of the extent, content, and nature of such activity. Given the costs involved and increasing concern about data confidentiality,¹² a central directory of all the clinical databases in a hospital might prove very helpful for managers.

DoCDat is being evaluated in three ways. Firstly, the level of use is being monitored using the number of website “hits”. This somewhat limited information is being supplemented by asking visitors to register their interest by recording their e-mail address. While this will not be comprehensive (as registration is voluntary), it will provide some insight into the background of users. Secondly, visitors are invited to comment and make suggestions as to how DoCDat might be improved. Thirdly, ad hoc surveys will be conducted of registered users, database custodians, relevant statutory bodies (such as the Commission for Health Improvement and the Department of Health), and those involved in funding, assessing and publishing research and audit results.

While enabling greater access and use of existing clinical databases is the immediate aim of DoCDat, its other aim is to improve their quality. Our experience suggests that some database custodians have rather limited knowledge and understanding of the methodological issues relating to database quality. DoCDat aims to advise, where appropriate, how quality can be improved. This is being facilitated by establishing a network of database custodians and putting them in contact with one another to enable practical experiences to be shared.