• patient safety
• adverse events
• measurement methodology

The Harvard Medical Practice Study (HMPS)^1,2 was not the first study to examine adverse events in healthcare organizations, but it established the standard by which adverse events are measured and laid the groundwork for policy discussions on patient safety in several countries. This commentary examines the impact of the study on research and policy in the US and elsewhere.

The methods used in the HMPS were based on the 1977 California medical insurance feasibility study.³ The refining and rigorous application of these methods to a random sample of patients and hospitals offered one of the first large sample estimates of adverse events in the health services research literature.

Today the HMPS is best known for the methods developed to identify adverse events and estimate their incidence. Yet this was only one of the investigators’ goals. Defining the incidence of adverse events was necessary for evaluating whether the tort system was effective in rewarding those who are injured as a result of their care in hospitals and assessing the economic consequences of such injuries. The dramatic finding that adverse events were a common component of hospital care has largely overshadowed the attention given to the evaluation of the tort system and assessment of costs.

The HMPS method for identifying adverse events is based on a two stage chart review. The first stage is carried out by nurses to screen patient records that are likely to include an adverse event. Selected charts are then reviewed in more detail by physicians to confirm the presence of adverse events and to assess the extent to which these events indicate substandard care. This review process has become the benchmark method for research on adverse events in hospitals. However, it should be noted that the methods have drawn criticism for several reasons. Firstly, the documentation in patient records may be incomplete allowing some adverse events to escape notice and, secondly, it is often difficult to untangle the contribution of medical intervention from the underlying disease processes. Thus, even with the carefully structured review process created in the Harvard study, there is substantial variation in the judgments of physician reviewers in that study and others who have used this method. Reliability estimates on the assessment of adverse events are only moderate; those relating to negligence and the degree of impairment attributable to the adverse event are even lower. Other methods, including direct observation and stimulated recall, yield higher numbers of adverse events. Detection using administrative data systems, computer screens, and error reporting systems are less sensitive, but also less costly.⁴ Regardless of this, chart review—perhaps because it relies upon the written history of patients’ experiences and provides a longitudinal view not available through any other method (except for computerized records)—is often considered the best method for identifying adverse events.

Despite the considerable weight of its findings, the full impact of the HMPS was not felt until the release of the Institute of Medicine (IOM) report, To Err is Human,⁵ in late 1999. The authors of this report developed population estimates of the numbers of Americans who die in hospitals as a result of preventable adverse events based on extrapolations from the HMPS and the more recent Utah-Colorado study.⁶ Before publication of the IOM report patient safety was a hidden issue in American health care, but following its publication patient safety became a focal point for reform. The HMPS study contributed important evidence to the ensuing policy debates on the steps needed to assess patient safety and reduce the injury burden.

Another important impact of the HMPS is the use of these methods by researchers in other countries. The HMPS results stimulated interest among Australian researchers and policy makers who replicated the study in a sample of 28 hospitals in 1995. The Australians were more interested in the quality of hospital care than in the performance of the malpractice system, so they reoriented the chart review assessment from judgments of negligence (was the care substandard?) to assessments of potential improvement (could the adverse event be prevented?) This orientation, together with some alterations in the methods, yielded substantially higher results. While the Harvard study found 3.7% of hospital patients in New York State had experienced an adverse event, the Australian study reported that 16.6% of hospital admissions were associated with an adverse event.⁷ Later analyses comparing the Australian methods with those of the Utah-Colorado study reduced the magnitude of these differences.^8,9

In addition to the Australian study, the HMPS methods have been replicated in the UK,¹⁰ Denmark, and New Zealand.^11,12 A recent study in France compared the Harvard methods with other approaches,¹³ while a Canadian study of adverse events will be published shortly. The existence of benchmarks in other jurisdictions heightens the appeal of the HMPS methods as a means of assessing the status of patient safety in hospitals around the world.

The HMPS identified adverse drug events as the second most common type of event. This result helped to stimulate research on the epidemiology of adverse drug events¹⁴ and on methods to reduce them.¹⁵ The Harvard study and the more recent study in Utah and Colorado have also contributed to policy discussions about tort reform and the effectiveness of the current medical malpractice system in the US.¹⁶

The next steps for improving adverse event reporting and investigation will require flexible and efficient tools that can accurately identify patients at risk and target areas for improvement. Beyond the issues of reliability, chart review methods are limited by the retrospective nature of such reviews and the expense involved in clinical assessment of patient records. However, these limitations could be reduced if the screens used in the first stage review were computerized, or other methods were developed that identified patients with a high likelihood of adverse events. Chart review inevitably points toward individual activities rather than system problems that underlie preventable adverse events. Identifying an adverse event or a pattern of events can therefore only be the first step in creating more effective care systems. Nevertheless, the information gleaned from such reviews may help to stimulate improvement. The next generation of tools needs to be applicable at a reasonable cost and linked to ongoing reviews of patient care. Computerization of the chart review tools would extend the use of these methods from research to quality improvement.