The term “evidence-based medicine” was first coined by Sackett and colleagues as “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.”1 The key to practising evidence-based medicine is applying the best current knowledge to decisions in individual patients. Medical knowledge is continually and rapidly expanding. For clinicians to practise evidence-based medicine, they must have the skills to read and interpret the medical literature so that they can determine the validity, reliability, credibility and utility of individual articles. These skills are known as critical appraisal skills, and they require some knowledge of biostatistics, clinical epidemiology, decision analysis and economics, and clinical knowledge.
Evidence Based Reviews in Surgery (EBRS) is a program jointly sponsored by the Canadian Association of General Surgeons (CAGS) and the American College of Surgeons (ACS) and is supported by an educational grant from ETHICON and ETHICON ENDO-SURGERY, both units of Johnson & Johnson Medical Products, a division of Johnson & Johnson and ETHICON Inc. and ETHICON ENDO-SURGERY Inc., divisions of Johnson & Johnson Inc. The primary objective of EBRS is to help practising surgeons improve their critical appraisal skills. During the academic year, 8 clinical articles are chosen for review and discussion. They are selected for their clinical relevance to general surgeons and because they cover a spectrum of issues important to surgeons, including causation or risk factors for disease, natural history or prognosis of disease, how to quantify disease, diagnostic tests, early diagnosis and the effectiveness of treatment. A methodological article guides the reader in critical appraisal of the clinical article. Methodological and clinical reviews of the article are performed by experts in the relevant areas and posted on the EBRS website, where they are archived indefinitely. In addition, a listserv allows participants to discuss the monthly article. Surgeons who participate in the monthly packages can obtain Royal College of Physicians and Surgeons of Canada Maintenance of Certification credits and/or continuing medical education credits for the current article only by reading the monthly articles, participating in the listserv discussion, reading the methodological and clinical reviews and completing the monthly online evaluation and multiple choice questions.
We hope readers will find EBRS useful in improving their critical appraisal skills and in keeping abreast of new developments in general surgery. Four reviews are published in condensed versions in the Canadian Journal of Surgery and 4 are published in the Journal of the American College of Surgeons. For further information about EBRS, please refer to the CAGS or ACS websites. Questions and comments can be directed to the program administrator, Marg McKenzie, at mmckenzie{at}mtsinai.on.ca.
Selected Article
Neily J, Mills PD, Young-Xu Y, et al. Association between implementation of a medical team training program and surgical mortality. JAMA 2010;304:1693–1700.
Abstract
Objective: To determine whether an association existed between the Veterans Health Administration (VHA) medical team training and surgical outcomes. Study design: Retrospective cohort study. Setting: 108 VHA hospitals (74 with and 34 without team training). Intervention: Medical team training consisted of a 1-day session with nurses, surgeons and anesthesiologists in attendance, followed by 4 quarterly structured telephone interviews. Data sources: VHA Surgical Quality Improvement Program (VASQIP), which included 182 409 patients between 2006 and 2008. Main outcome measure: Postoperative mortality. Results: The 74 facilities in the training program experienced an 18% reduction in annual mortality (rate ratio [RR] 0.82, 95% confidence interval [CI] 0.76–0.91, p = 0.01) compared with a 7% decrease among the 34 facilities that had not yet undergone training (RR 0.93, 95% CI 0.80–1.06, p = 0.59). Risk-adjusted mortality at baseline was 17 per 1000 procedures per year for the trained and 15 per 1000 per year for the nontrained facilities. At the end of the study, the rates were 14 per 1000 per year for both groups. Propensity matching of the trained and nontrained groups demonstrated that the decline in risk-adjusted surgical mortality was about 50% greater in the trained (RR 1.49, 95% CI 1.10–2.07, p = 0.01) than in the nontrained group. A dose–response relationship for additional quarters of the training program was also demonstrated: for every quarter of the training program a reduction of 0.5 deaths per 1000 procedures occurred (95% CI 0.2–1.0, p = 0.001). Conclusion: Participation in the VHA Medical Team Training program was associated with lower surgical mortality.
Commentary
The development of Crew Resource Management, the team training curriculum used in aviation, was led by human factors expert, Robert Helmreich. As early as 1993, Helmreich recognized that the principles of communication and team coordination that improved the performance of flight crews were applicable to medicine.1 Nearly 2 decades later, these concepts of teamwork are acknowledged as important. However, the penetrance into health care institutions has been relatively limited. The stated goals of team training programs include improved patient safety and staff satisfaction. An early challenge in the implementation of these programs has been to identify metrics by which the goals can be measured.
Because airplane accidents are rare, demonstrating improved safety with Crew Resource Management proved challenging. Helmreich developed the Safety Attitudes Questionnaire as a way to measure perceived team-work as an indirect quality measure. This questionnaire was subsequently adapted for health care purposes to measure the attitudes of operating room team members. Early reports on team training used the Safety Attitudes Questionnaire as a measure of effectiveness.2 More recently, investigators have been able to link team training programs to improved team performance in terms of efficiency and compliance with performance measures, such as preoperative antibiotic administration.3 This article by Neily and colleagues adds to their previous report that demonstrated improved patient outcomes following implementation of a team training curriculum in the VHA system.4
In the present study, the authors examined a retrospective health services cohort using a contemporaneous control group. At VHA hospitals where team training was implemented, investigators compared surgical mortality following the intervention with the centre’s own baseline surgical mortality and with that of facilities where no intervention had been implemented. Of the 108 facilities included in the study, 74 had participated in team training between 2006 and 2008, and 34 had not yet received the training. The intervention involved training clinicians to work as a team; challenge each other when they identify safety risks; conduct checklist-guided preoperative briefings and postoperative debriefings; and implement other communication strategies, such as recognizing red flags, establishing rules of conduct for communication, stepping back to reassess a situation, and communicating effectively during care transitions between clinicians. Data on the covariates and outcomes were collected from the VASQIP data. The main outcome measures were observed and risk-adjusted 30-day mortality. Observed mortality was defined as any-cause death in the 30 days after a major noncardiac surgery performed at a VA medical centre. Risk-adjusted mortality was defined as observed mortality adjusted for each patient’s risk profile (surgical complexity, patient comorbidity and socio-demographic characteristics). Patients in the intervention versus the nonintervention groups were stratified using propensity scores and matched to assess the effect of the intervention/exposure.
Neily and colleagues noted that, when comparing the before and after data at the 74 trained facilities, there was an 18% decrease in the annual observed mortality (RR 0.82, p = 0.01). At the 34 untrained facilities there was a 7% decrease in mortality (RR 0.93, p = 0.59). Propensity score matched analysis showed a 50% greater decrease in annual mortality in the trained (RR 1.49, p = 0.01) compared with the nontrained group. Interestingly, the investigators found a dose-dependent relationship for increasing quarters of training. Mortality decreased by 0.5 deaths per 1000 procedures for every quarter (p = 0.001).
Two other before and after studies evaluating the effect of using a surgical safety checklist on postoperative mortality found that use of a checklist alone could reduce surgical mortality by 50%.5,6 By comparison, Neily and colleagues showed that medical team training resulted in an 18% reduction in postoperative mortality in the trained cohort. This reduction in postoperative mortality was 50% greater than that in the nontrained cohort in the propensity-matched analyses. While different studies need to be compared cautiously, it is interesting to note that the much more simple, quick and inexpensive intervention of a surgical safety checklist appears to be as or more effective at reducing postoperative deaths than the multidimensional intervention team-training evaluated in the present study. As such, decision-makers may be less likely to be persuaded to adopt medical team training as a method to improve surgical outcomes.
A major shortcoming of the present study was its retrospective nature. No randomized controlled trial has demonstrated a benefit of perioperative communication tools (e.g., surgical safety checklists) or behavioural interventions (e.g., medical team training). The evidence on the effectiveness of these interventions comes solely from before and after studies, which are subject to selection bias and confounding. The gold standard for ensuring that 2 groups are similar in a comparative study is randomization. The rationale behind randomization is that identical confounding variables should be similar in both the intervention and the control group, thus allowing the investigators to conclude that an observed difference is due to the intervention. In most real-life studies, randomization is not possible. Propensity score matching is a method that provides investigators of a nonrandomized study a certain degree of confidence that the intervention and control arms are comparable at baseline. Although propensity-score matching can control for many potential confounders, it cannot remove all potential bias; therefore, the training assignment in the present study cannot truly be considered random.
Notwithstanding these limitations, the results are provocative and may be clinically important. Lower short-term mortality for major surgical procedures has been an important issue in patient safety. Concrete evidence to support suitable and focused interventions toward this goal has been slow to come by. Importantly, the study by Neily and colleagues was able to demonstrate reduction in mortality across a nationwide system of health care facilities. These data seemingly provide evidence to support the implementation of such training and follow-up activities at surgical facilities across the United States. It is important to note, however, that the study was conducted exclusively in the US Veterans Affairs system, which differs substantially from many other health care environments. Most patients in the VA system are men, older and may have more comorbidities than the average patient. The VA hospitals also have different organizational structures and operating room staffing models than non-VA hospitals. In many hospitals, health care personnel do not work with the same people on a consistent basis, limiting their ability to develop team behaviours since there really may be no “team.” Furthermore, the intervention tested in the study included a 1-day training session using lectures, group interactions and videos. Although it may not seem like a major logistical feat, in many typical hospital settings it would be difficult to organize a full-day intervention involving all the appropriate health care providers. As such, while the results of the study may potentially be generalizable to non-VA hospitals, it remains unclear if such an intervention has broader applicability.
Furthermore, explanations of the mechanisms of quality improvement from the team training intervention were not particularly compelling. For example, stories shared by surgical teams in the study about how medical team training improved surgical performance included anecdotes about discovering during a briefing that a patient was anticoagulated or required cardiac clearance. It is not clear why a team briefing would identify these potential problems any better than a routine preoperative assessment by a health care provider. Until there is experimental evidence supporting the effectiveness of perioperative behavioural interventions, it cannot be assumed that they will result in the improvements in perioperative outcomes suggested by the existing literature.
One of the most important learning points in this article is the months of preparation and planning that occurred before the actual team training and the 1-year follow-up to provide support and coaching. Hamman appropriately cautions that medical team training “is not a one-time immunization given in isolation to a professional in a 2-day seminar.7” The success of a team training program is dependent on an institution’s long-term commitment to the principles covered, such as using checklists, flattening hierarchy and adopting rules of conduct for communication. These programs require extensive institutional investment of resources for curriculum infrastructure and time away from work for the participants. In addition, ongoing coaching, encouragement of constructive behaviour and repercussions for individuals who refuse to participate are needed. Armour and colleagues reported that the improved outcomes after team training disappeared after several months, reflecting cultural deterioration.8
In theory, the new behaviours taught in team training courses could become so embedded in the culture of an institution that a separate team training curriculum would no longer be required. The principles of respect, appropriate assertiveness and mutual support would be woven into the education of trainees at all levels. More importantly, more junior members of the team would learn by having exemplary role models. At this time, additional studies that demonstrate improved outcomes with team training are warranted to encourage more institutions to develop such programs.
Footnotes
↵* The CAGS/ACS Evidence Based Reviews in Surgery Group comprises N.N. Baxter, K.J. Brasel, C.J. Brown, P. Chaudhury, C.S. Cutter, C.M. Divino, E. Dixon, L. Dubois, G.W.N. Fitzgerald, H.J.A. Henteleff, A.W. Kirkpatrick, S. Latosinsky, A.R. MacLean, T.M. Mastracci, R.S. McLeod, A.M. Morris, L.A. Neumayer, L.K. Temple and M.E. McKenzie.
Competing interests: None declared.