Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat
BMJ 2002; 325 doi: https://doi.org/10.1136/bmj.325.7360.367 (Published 17 August 2002) Cite this as: BMJ 2002;325:367
- Signe Flottorp, researcher (signe.flottorp{at}shdir.no)a,
- Andrew D Oxman, directora,
- Kari Håvelsrud, research associatea,
- Shaun Treweek, researchera,
- Jeph Herrin, esearch analystb
- aDepartment of Health Services Research, Norwegian Directorate for Health and Social Welfare, PO Box 8054 Dep, N-0031 Oslo, Norway
- b Flying Buttress Associates, PO Box 2254, Charlottesville VA 22902, USA
- Correspondence to: S Flottorp
- Accepted 24 May 2002
Abstract
Objective:To assess the effectiveness of tailored interventions to implement guidelines for urinary tract infections in women and sore throat
Design:Unblinded, cluster randomised pretest-post-test trial
Setting:142 general practices in Norway
Participants:72 practices received interventions to implement guidelines for urinary tract infection and 70 practices received interventions to implement guidelines for sore throat, serving as controls for each other. 59 practices in the urinary tract infection group and 61 practices in the sore throat group completed the study. Outcomes were measured in 16 939 consultations for sore throat and 9887 consultations for urinary tract infection.
Interventions:Interventions were developed to overcome identified barriers to implementing the guidelines. The main components of the tailored interventions were patient educational material, computer based decision support and reminders, an increase in the fee for telephone consultations, and interactive courses for general practitioners and practice assistants
Main outcome measures:Changes in rates of use of antibiotics, laboratory tests, and telephone consultations
Results:Patients in the sore throat group were 3% less likely to receive antibiotics after the intervention. Women with symptoms of urinary tract infection in the intervention group were 5.1% less likely to have a laboratory test ordered. No significant differences were found between the groups for the other outcomes. Large variation was found across the included practicesin the rates of antibiotic prescription, use of laboratory tests and telephone consultations, and in the extent of change for all three outcome measures
Conclusions:Passively delivered, complex interventions targeted at identified barriers to change had little effect in changing practice
What is already known on this topic
Interventions to change professional practice have small to moderate effects at best
Multifaceted interventions targeted at identified barriers to change are more likely to be effective for implementing guidelines than a simple intervention selected by chance
What this study adds
Large variation exists in the extent of change before and after the delivery of tailored interventions to support implementing guidelines
Rigorous methods are needed to evaluate interventions to change practice
With passively delivered, complex interventions targeted at identified barriers there was only a 3% decrease in antibiotic prescribing for sore throat and a 5% decrease in test use for urinary tract infection in women
Introduction
Many theories but limited evidence exist on the effectiveness of interventions to change professional practice. 1 2 Although more complex interventions tend to be most effective, their effectiveness varies, they require more resources, and it is difficult to know which interventions to use. Identifying barriersto change and tailoring interventions to address these is a logical approach to selecting appropriate interventions.3 The effectiveness of tailored interventions remains uncertain.4–10 We aimed to assess the effectiveness of tailored interventions to supportthe implementation of guidelines for the management of urinary tract infections in women and sore throat.
We developed evidence based guidelines for urinary tract infections in women and sore throat. 11 12 The main recommendation for sore throat is that most patients do not need antibiotics. Clinical examination and laboratory tests are therefore generally not necessary, and patients can be given advice by telephone. The main recommendation for urinary tract infections is that non-pregnant women aged 16-55 years with typical symptoms can be treated with antibiotics without any testing. Women who have had a previous urinary tract infection can be offered treatment by telephone.
Methods
Design
We tested the main hypotheses with a cluster randomised pretest-post-test controlled trial. General practices were randomised to receive tailored interventions to support implementing guidelines for either urinary tract infections in women or sore throat (fig 1). Practices receiving one set of guidelines served as controls for practices receiving the other. Data were collected 18 weeks before and after the intervention.
Participants
We randomly selected 292 of the 323 general practices in Norway with the WinMed electronic medical record system, and we sent a letter inviting them to participate. Overall, 142 practices were randomised by computer (fig 1).13
We included consultations for sore throat if they concerned patients over 3 years old with diagnosis codes relevant for sore throat from the international classification for primary care.14 We included consultations for urinary tract infection analysisif they concerned non-pregnant women aged 16 to 55 years with diagnosis codes relevant for urinarytract infections from the international classification for primary care.
Interventions
We tailored the interventions to overcome identified barriers. Both guidelines were published in the June 2000 issue of the Journal of the Norwegian Medical Association, which all Norwegian general practitioners receive.1112 Box 1 shows the main interventions for urinary tract infections and sore throat.
Main interventions for urinary tract infections and sore throat
Summary of the main recommendations in electronic and poster format
Patient educational material in electronic and paper format
Computer based decision support and reminders during consultations
An increase in the fee for telephone consultations for these two diagnoses from 22 to 50 Norwegian kroner(£1.92 to £4.35; €2.96 to €6.73) with no change in the fee for an office visit, which was 110 Kr (£9.58; €14.81) for non-specialists and 155 Kr (£13.50; €20.86) for general practitioner specialists
Printed material to facilitate discussions in the practice
Interactive courses for general practitioners and practice assistants
Points in the continuing medical education programme for project participators
The interventions were initiated in May 2000 and continued until January 2001. Because of the nature of the interventions, participating practices knew the group to which they were assigned. One member of each practice signed an informed consent form. The study protocol was approved by the Data Inspectorate, the Norwegian Board of Health, and the regional committee for medical research ethics.
Data
We extracted outcome data with two separate methods—Mediata and QTools, software that we developed in collaboration with Mediata.13 The software wassent to the practices on disk and installed by their staff. The staff ran the data extraction routines that transferred relevant data to disks, which were then posted back to us.
The Mediata software also included an interactive decision support application and a tool to collect additional data from pop-up screens that were triggered when a diagnosis code for sore throat or urinary tract infection was entered into a patient's record.
The Qtools software extracted relevant data, which is routinely collected, from the electronic medical record. We used these data for the main analysis because we only obtained data from Mediata for a subset of the consultations. We compared data from the baseline period, 1 January 2000 to 15 May 2000, with data from the follow up period, 15 September 2000 to 31 January 2001.
Outcomes
The primary outcomes for both conditions were changes in the rates of use of antibiotics, laboratory tests, and telephone consultations. We expected that implementation of the guidelines would reduce the proportion of patients who were prescribed antibiotics for sore throat but not change the proportion prescribed antibiotics for urinary tract infections, reduce the use of laboratory tests for both, and increase the proportion of telephone consultations for both.
We identified Anatomical Therapeutic Chemical Classification System codes15 for prescriptions and information from the prescription name field in WinMed.We classified the drugs as relevant or not relevant for urinary tract infections and sore throat. A standardised coding system for laboratory tests is not used in Norway. We classified all tests extracted from the field for laboratory tests in WinMed as relevant or not relevant for urinary tract infections and sore throat. We categorised a consultation as a telephone consultation if it wascoded as such in WinMed or if a fee code for a telephone consultation was registered.
Sample size
We calculated sample size with a method that takes into account the intracluster correlation coefficient, the number of events, the expected effect, and the power of the study.16 We assumed an intracluster correlation coefficient of ρ=0.2, a minimum of 25 patients for each practice, and a worst case control group rate of 50%. Under these assumptions we anticipated a power of 87% to detect a difference of 15% in rates between the two groups with α=0.05 with 60 practices for each intervention group. We anticipated a loss to follow up and therefore planned to randomise 70 practices in each group.
Statistical methods
We used cluster specific methods because practices rather than patients were randomised, and weexpected that variance in how patients were managed would be partly explained by the practice.17 Because some patients had more than one consultation, we added a third level to the analysis to account for the likelihood that variance in what was done at each consultation would be partly explained by the patient. For each of the outcome measures we tested the null hypothesis that the likelihood of the outcome was the same in both study groups after accounting for baseline rates, multiple patients for each practice, and repeated consultations for each patient.
We used hierarchical logistic regression to test for an effect of each intervention on the outcome measures (box 2).17–19 This technique allowed us to test for a positive interaction between time and intervention, which would indicate an effect of the intervention on outcomes. All hierarchical models were estimated with HLM5.20 All other analyses were done with Stata 7.
Three level logistical hierarchical model
To test effects of the two interventions on usage of antibiotics, laboratory tests, and telephone consultations, we estimated, for each intervention and each outcome measure, a three level logistical hierarchical model. For each dichotomous outcome this was specified as follows:
Level 1: Let pijk be probability that k-th consultation of j-th patient in i-th practice has given outcome. Then this probability was assumed to follow a logistic model
logit pijk=π0+π1T
where T is dichotomous variable for time, which equals zero for baseline time period and one for post intervention time period.
Level 2: π0 was estimated as random effect and π1 as fixed effect
π0=β00+r0
π1=β10
where βs represent individual level effects.
Level 3: β00 was estimated as random effect at practice level and β10 as fixed effect, with both allowed to vary by intervention group
β00=γ000+γ001(X)+0
β10=γ100+γ101(X)
where X is an indicator variable for intervention. By testing hypotheses of form H0:γ101=0 we were able to test the effect of the intervention on patient treatment; γ101 significantly different from zero indicates effect of intervention over time.
RETURN TO TEXTResults
Thirteen practices in the urinary tract infection arm and nine practices in the sore throat armdropped out after randomisation (fig 1). All the practices installed the Mediata and Qtools software. We did not get data from the QTools program for two urinary tract infection practices and five sore throat practices. This left 57 of 72 practices in theurinary tract infection arm and 56 of 70 practices in the sore throat arm for the main analyses.
The sore throat practices had more practitioners and registered a greater number of consultations than did the urinary tract infection practices (tables 1–3). The arms were similar for patient characteristics and baseline measurements (tables 2–4).
Characteristics of included practices
Characteristics of consultations for sore throat
Characteristics of consultations for urinary tract infection
Rate of use and percentage difference in change of use of antibiotics, laboratory tests, and telephone consultations
Outcome data were collected for 16 939 consultations for sore throat and 9887 consultations forurinary tract infection (tables 2 and 3).The intracluster correlation coefficients for the primary outcomes variedfrom 0.05 to 0.21 (table 4),indicating that cluster specific analytical methods were appropriate.
Use of antibiotics
After adjusting for baseline rates, patients with sore throat in the intervention group were less likely to receive antibiotics than patients withsore throat in the control group (table 4). The absolute reduction in the proportion of consultations where antibiotics were prescribed for sore throat was 3.0% greater in the intervention group. For patients with urinary tract infection there was little change in the proportion of consultations where antibiotics were prescribed in both the intervention group (−0.2%) and the control group (0.2%).
Use of laboratory tests
After adjusting for baseline rates, women with symptoms of urinary tract infection in the intervention group were less likely to have a laboratory test ordered than were women in the control group. The absolute reduction in the proportion of consultations for urinary tract infection where alaboratory test was ordered for urinary tract infections was 5.1% greater in the intervention group. No significant differences were found between the groups for use of laboratory tests for sore throat (table 4).
Use of telephone consultations
After adjusting for baseline rates, the absolute increase in the proportion of telephone consultations for sore throat was 1.2% greater in the control group than it was in the intervention group (table 4).The proportion of telephone consultations forurinary tract infections was decreased in both the intervention group and the control group. The reduction was 0.9% greater in the control group. These differences were not significant.
Other analyses
Large variations occurred across the included practices in the use of antibiotics, laboratory tests, and telephone consultations and in the extent of change for all three outcome measures (fig 2).
Rates of antibiotic use in consultations for sore throat before and after the tailored interventions from all practices with more than 10 consultations in each period
Discussion
Interventions that were tailored to address identified barriers to implementing guidelines for urinary tract infections in women and sore throat had little impact on changes in the use of antibiotics, laboratory tests, and telephone consultations.
Although only practices using WinMed—one of three electronic medical record systems used in Norway—were included in our study, there is no reason why these practices should differ from other practices.
We might have underestimated the rates for antibiotic prescriptions, ordering of laboratory tests, and telephone consultations because the practitioners may not have registered this informationcorrectly in some instances. This is unlikely to have differed between the groups and is thereforeunlikely to have affected the results.
It is possible that we underestimated the reduction in the prescription of antibiotics for sorethroat. Some of the general practitioners in the study reported that they now inform patients thatantibiotics normally are not necessary but prescribe them for throat infections with Streptococcus for use “if needed.” We do not know how often patients were told this or how often prescriptions that were given with this message were not used.
There are several possible explanations for the small effect that we found. We might have failed to identify important barriers to change, although neither the participants nor we identified additional barriers during the course of the study or in discussing the results. We did not tailor the interventions to the needs of individual practices or of individual general practitioners. Tailoring the intervention at this level might have had a greater effect, although we do not have any data to support this hypothesis.
Two more likely explanations are the passive character of the interventions and the lack of time. We did not have the resources to actively support the practices with outreach visits. Many practices did not have routines to support internal discussion and implementation of the guidelines. We identified some barriers—for example, problems with accessibility by telephone—that we were not able to adequately address with our interventions.
It is possible that our follow up period was not long enough to detect changes in practice. Women are used to taking a urine sample to their doctor when they have symptoms of a urinary tract infection, and patients may have expected to have laboratory tests taken and receive antibiotics forsore throats. In addition, the demands of daily practice prevented many participants from devotingmuch time to the project.
Few trials have been done on implementing guidelines for urinary tract infections or sore throat. A non-randomised controlled before and after study of the implementation of a telephone based guideline for managing presumed cystitis in women aged 18 to 55 years found significant decreases in the proportions of patients who received urinalysis, had a urine culture, or had an initial visit to their doctor.21 The design of this study was not robust, but the results are promising and the interventions used were intensive. A randomised controlled trial with groups of general practitioners using an educational intervention to influence prescribing for urinary tract infections found that prescription of first choice drugs increased from 52% to 70% in the intervention group and remained constant in the control group, but the intervention had no effect on the duration of treatment.22 We have not identified trials of the implementation of a guideline for sore throat similar to ours. Studies of scoring rules for sore throat have failed to show that they lower the rates for antibiotic prescription.23 It has been shown that it is possible to improve doctors' judgments of the probability of streptococcal pharyngitis, but this knowledge did not affect decisions to prescribe antibiotics.24
Figure 2 shows the importance of using adequately sized cluster randomised controlled trials to evaluate interventions to support the implementation of guidelines. Large variation exists in practice and in the extent of change among practices. Uncontrolled or inadequately controlled before and after evaluations in selected practices are likely to have spurious results that are, at best, difficult to interpret.
Conclusions
Despite the small effect that we found, it still seems logical to select strategies to support the implementation of guidelines that are tailored to address identified barriers. However, it is difficult to change practice, and large changes over short periods are not typical. Therefore it is important to use rigorous methods to measure the effects of the interventions that are used and to ensure that they are cost effective before widely employing them. It is also important to invest sufficient resources to ensure that active interventions can be used to engage clinicians and toensure that they find the time that is needed to change their routines.
Acknowledgments
We thank Arild Bj⊘rndal, Cheryl Carling, Jan Arve Dyrnes, Tor Arne Bertheussen, and Atle Fretheim for their help and support, the advisory committee for the project and the reference group for the development of the guidelines for support and advice, Mediata, which helped develop the software for data collection and decision support, and the participating practices.
Contributors: SF and ADO developed the study protocol and interventions. SF conducted the study. KH assisted in recruitment of practices and was responsible for the communication with and follow up of the participating practices. ST helped with development and implementation of the softwarein the practices. JH randomised the practices and was responsible for the statistical analyses. All contributors discussed core ideas and obstacles during the study. SF drafted the manuscript withsupport from the other authors. SF and ADO will act as guarantors for the paper.
Footnotes
-
Funding Quality Assurance Fund of the Norwegian Medical Association and the National Institute of Public Health
-
Competing interests None declared