Abstract
Background: The European Practice Assessment program provides feedback and outreach visits to primary care practices to facilitate quality improvement in five domains (infrastructure, people, information, finance, and quality and safety). We examined the effectiveness of this program in improving management in primary care practices in Germany, with a focus on the domain of quality and safety.
Methods: In a before–after study, 102 primary care practices completed a practice assessment using the European Practice Assessment instrument at baseline and three years later (intervention group). A comparative group of 102 practices was included that completed their first assessment using this instrument at the time of the intervention group’s second assessment. Mean scores were based on the proportion of indicators for which a positive response was achieved by all of the practices, on a scale of 0 to 100.
Results: We found significant improvements in all domains between the first and second assessments in the intervention group. In the domain of quality and safety, improvements in scores (mean scores were based on the proportion of indicators for which a positive response was achieved by all of the practices, on a scale of 0 to 100) were observed in the following dimensions: complaint management (from a mean score of 51.2 at first assessment to 80.7 at second assessment); analysis of critical incidents (from 79.1 to 89.6); and quality development, quality policy (from 40.7 to 55.6). Overall scores at the time of the second assessment were significantly higher in the intervention group than in the comparative group.
Interpretation: Primary care practices that completed the European Practice Assessment instrument twice over a three-year period showed improvements in practice management. Our findings show the value of the quality-improvement cycle in the context of practice assessment and the use of established organizational standards for practice management with the Europeaen Practice Assessment.
A variety of quality-improvement initiatives in health care management have been implemented in most health care systems.1,2 Countries such as Australia, New Zealand, the United Kingdom and the United States have a long tradition in, and established standards for, quality management in primary care. In Canada, such initiatives in primary care are in their infancy, despite support by the federal Primary Health Care Transition Fund since 2000.3 In Ontario, a comprehensive book was recently issued that provides recommendations on practice management and clinical indicators for improving quality in primary care settings in the province.4 The indicators were adopted and refined from the Royal New Zealand College of General Practitioners, the Royal Australian College of General Practitioners’ National Expert Committee on Standards for General Practices, the TOPAS–Europe Association European Practice Assessment, the United Kingdom’s Quality and Outcomes Framework and the Canadian Institute for Health Information.
In Germany, similar developments took place. In 2005, the German government stipulated that health care providers implement a system of annual assessment of quality management.5 One of the systems available to practices is the European Practice Assessment (www.epa-qm.de), a validated instrument based on quality indicators for assessing practice management.6 The five key domains of the European Practice Assessment instrument and their respective dimensions are described in Figure 1.
The domains and dimensions (and number of indicators) of the European Practice Assessment instrument used to measure the quality of practice management in primary care practices. For an example of how the pentagon shape is used to provide feedback to individual practices, see Appendix 1 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110412/-/DC1). GP = general practitioner.
The European Practice Assessment is used to help general practices both assess and improve their quality of practice management set against predefined criteria with embedded quality indicators. The improvement process is ongoing (e.g., through plan–do–study–act cycles7): each step is reviewed and redesigned with a view to improving the quality of the end product, thereby fostering continuous improvement.8 The multifaceted strategy has three essential components: assessment and feedback using validated instruments based on quality indicators; external support through an outreach educational visit by a trained visitor (physician or nurse) to support the practice in improving areas of management identified by the practice;9 and formal accreditation by an external organization.
Three requirements have to be fulfilled by practices to receive accreditation: achieve a positive response for more than 50% of the indicators; meet predefined safety indicators (e.g., the vaccination status of staff regarding hepatitis B vaccination is recorded and medical equipment is checked regularly according to national regulations); and highlight areas for continuous quality improvement.
Accreditation is one method for assessing and benchmarking the performance of general practice care across a broad range of clinical and organizational domains. It describes a formal process of self-assessment and external and independent peer review to encourage best management practice and can result in recommendations for continuous improvement of safety and quality in the practice.8 Practice accreditation can be used for different purposes: quality control, regulation, quality improvement, providing data on performance, and marketing.8 In Germany, it is used for quality improvement, leading to a certificate.
We conducted a study to determine whether improvements in practice management occurred in general practices that completed the European Practice Assessment twice over three years, compared with general practices that completed the European Practice Assessment once. We focused our analysis on the domain of quality and safety, expecting an association between practice organization and quality improvement.10,11 We hypothesized that the initial use of the European Practice Assessment and reassessment with it three years later would result in improved scores in the dimensions and indicators within the domain of quality and safety.
Methods
Study practices
We included general practices in Germany that had completed the European Practice Assessment as part of a quality-improvement program for ambulatory care between May 2004 and January 2009.
For the intervention group, we selected 110 practices that underwent their first assessment between May 2004 and November 2007 and whose staff completed the European Practice Assessment instruments. These practices underwent a second assessment between June 2007 and January 2009. The interval between assessments for each participating practice was 36 months. We excluded eight practices that had different intervals, which left 102 practices in the intervention group.
During the period when the intervention practices were undergoing their second assessment, 209 general practices underwent their first assessment with the European Practice Assessment program but had not yet had a second assessment. These practices were considered for inclusion in the comparative group.
Because both groups differed on several measures, we used a matched pair design based on propensity scores.12,13 The weights for the propensity score were calculated on the basis of binary logistic regression for the following covariates: practice type (solo v. group), location (urban v. rural), number of general practitioners per practice, number of nurses per practice and mean number of patients per practice seen in a quarter. We used the “Matching” package of the R Project (version 2.11.1) to perform the matching.14,15 We used the “MatchBalance” function of the package to evaluate the matching results, because this function not only examines differences in proportions or means, it also accounts for potential differences across the entire distribution.14 After matching, we had 102 comparable practices in the comparative group (Figure 2).
Selection of primary care practices for the intervention and comparative groups.
Practice assessment
The application of the European Practice Assessment is coordinated by the AQUA-Institute and is described in detail elsewhere.6 In brief, the self-administered questionnaire was sent to all practice staff (including the practice manager where employed) and individual general practitioners. Once completed questionnaires were returned, a practice visit was conducted by a specially trained visitor. The visitor first assessed the practice against a standardized checklist and then conducted a structured interview with the general practitioner who had management responsibilities or the practice manager. The visit finished with a feedback session on the preliminary outcomes of the assessment. The data file Visotool was used to show the results of the assessment and provide an anonymous comparison between the practice’s scores and all other practices that had undertaken the assessment, as a catalyst for quality improvement. (An example of a Visotool figure is presented in Appendix 1, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110412/-/DC1).
Statistical analysis
We used z scores to compare the mean scores for all domains of the European Practice Assessment instrument, and specifically for the dimensions and indicators in the “quality and safety” domain, in the intervention and comparative groups. (Mean scores were based on the proportion of indicators for which a positive response was achieved by all of the practices, on a scale of 0 to 100.) Establishing a z score is a procedure for normalizing aggregated data and is often used when comparing practices.16 We calculated 95% confidence intervals for the differences in scores in the intervention group (between the first and second assessments) and for the between-group differences in scores (between the second assessment in the intervention group and the first assessment in the comparative group). We used the χ2 test to compare characteristics of the practices in the intervention and comparative groups. An α level of p ≤ 0.05 was used for tests of statistical significance.
Ethics approval
Ethical approval was not required because we used data available from the routine implementation of a quality-improvement program in ambulatory care in Germany.5 All European Practice Assessment questionnaires and the information from the trained visitors were anonymized for data analysis in our study. No additional information or data from patients or staff were collected.
Results
The practices included in the intervention and comparative groups did not differ significantly in the relevant covariates of practice characteristics (Table 1).
Characteristics of primary care practices included in the study*
The descriptive results for European Practice Assessment are described elsewhere.17 We observed an improvement in quality management in the intervention group, with a change in score of 7.0 in the five domains overall. The change in score was 10.1 in the domain of quality and safety, which is analyzed in detail in this study.
Quality and safety
Changes in the scores for the dimensions in the domain of quality and safety are shown in Table 2. A significant improvement occurred in the intervention group in the dimensions “complaint management,” “analysis of critical incidents” and “quality development, quality policy.” Compared with the comparative group at first assessment, the intervention group at reassessment had better overall scores for the dimensions “detection of quality and safety problems,” “complaint management,” “analysis of critical incidents” and “quality development, quality policy.”
Mean scores* for the dimensions in the quality and safety domain of the European Practice Assessment instrument
Changes in scores for each of the indicators in the domain of quality and safety are presented in Table 3. Significant improvements were observed in the intervention group, especially in the dimensions “complaint management” (e.g., “there is a patient complaint procedure”), “analysis of critical incidents” (e.g., “there is a critical incident register”) and “quality development, quality policy” (e.g., “patient satisfaction is monitored regularly”).
Mean scores* for the indicators in the quality and safety domain of the European Practice Assessment instrument
Interpretation
In our study, primary care practices that had undergone the European Practice Assessment process showed overall improvements in practice management at the time of the second assessment. In the domain of quality and safety, we observed significant improvements in three of the five dimensions (“complaint management,” “analysis of critical incidents” and “quality development, quality policy”).
Two aspects are important when interpreting our results. An increase of 10.1 in the overall score for the domain of quality and safety could indicate well-organized and safe practices, which is important from the perspective of patients. From the perspective of practices, our study showed larger changes than studies did using clinical indicators.18
Similar to another evaluation of a quality-improvement program,19 we used repeated measurement to evaluate the effect of the assessment process. However, we included a comparative group. Moreover, the European Practice Assessment involves both quality assessment and quality improvement, for example in relation to the analysis of critical incidents, complaint management, and the detection of problems in quality and safety. This shows that patient safety is also an integral part of the European Practice Assessment, as one key component of quality of care.20
Quality improvement requires a combination of educational, organizational and financial approaches, using both intrinsic motivation and external incentives.21 For example, an Australia quality-improvement initiative “the Bettering the Evaluation and Care of Health program” showed that certification was positively associated with consultation behaviour and patient management.22
Quality of care is a broad concept, which requires a mix of objective and subjective measures.23 One method is formative accreditation, which combines a broad set of quality measures and various improvement activities such as audit and feedback,10 and educational outreach visits.11 These strategies have been shown to be effective, and the context of practice accreditation might further add to their impact. Many methods for quality improvement have been shown to have only short-term effects at best. For instance, pay-for-performance has been introduced in many countries and has shown short-term improvements, but the evidence has not been compelling and most incentives were attached to meeting a set target rather than quality improvement.24 Audit and feedback have been shown to have similar short-term effects.10 The same applies to feedback given to primary care providers on patients’ evaluations of care.25 This may suggest that such approaches need to be used as one part of a multiple-component strategy for quality improvement rather than used in isolation.26 Our findings support the use and role of formative accreditation as one part of such a strategy.
Limitations
Our study has limitations. First, the sample of participating practices was small and may not have been representative of general practices in Germany because we involved only practices that elected to use the European Practice Assessment to fulfil their obligation of following the national directive for quality management in office-based practices, which introduced a selection bias. Second, the improvement observed at the second assessment may have been attributed in part to the Hawthorne effect. Third, we used propensity scores to match comparable practices in the intervention and comparative groups, but not for statistical adjustment. Lastly, because this was an exploratory study, changes were only descriptive in nature and will need to be confirmed in further targeted studies. A well-organized, multifaceted quality management program motivates practices to change.27 However, until there is evidence of the impact on patient outcomes, it will be unclear how to weigh the benefits and risks of interventions such as the European Practice Assessment.
Conclusion
Primary care practices that completed the European Practice Assessment twice over a three-year period showed improvements in practice management. Our findings show the value of the quality-improvement cycle in the context of practice assessment. Unlike other assessments against predefined standards such as the Care Quality Commission or the Quality Outcome Framework in the United Kingdom18 or quality initiatives in Ontario,4 which all have target standards, quality improvement is an integral part of the European Practice Assessment. Our findings suggest that there is an intrinsic benefit to practices undertaking formative accreditation in terms of practice-specific quality improvement. The benchmark assessment motivates practices to improve their performance. Our study provides a better understanding of how accreditation can help to improve quality of care by enabling practices to both meet set standards and to identify and address internal priorities for quality improvement.
Acknowledgements
The authors thank the primary care practices and outreach visitors who participated in the study. The authors also thank Marc Brodowski and Mohamed Lambarki from the European Practice Assessment management team at the AQUA-Institute for providing management data, the Stiftung Praxissiegel e.V., which serves as an independent certification or accreditation body in Germany, and the European Association for Quality in General Practice / Family Medicine (EQuiP) for supporting the European Practice Assessment project and the development of the indicators.
Footnotes
-
Competing interests: Bjoern Broge and Sara Willms are employed by the AQUA-Institute, which disseminates the European Practice Assessment in Germany. Joachim Szecsenyi is its director and a stockholder. No competing interests declared by Stephen Campbell, Gunter Laux, Michel Wensing or Katja Goetz.
-
This article has been peer reviewed.
-
Contributors: All of the authors contributed to the study concept. Joachim Szecsenyi, Michel Wensing and Katja Goetz designed the study. Bjoern Broge and Sara Willms collected the data. Joachim Szecsenyi and Katja Goetz conducted the initial data analyses and wrote the first draft of the manuscript. Stephen Campbell, Gunter Laux and Michel Wensing contributed to the interpretation of the data. All of the authors reviewed and revised the manuscript and approved the final version submitted for publication.
-
Funding: This study was funded by the Department of General Practice and Health Services Research, University of Heidelberg.
References
In this issue
Article tools
Jump to section
Related Articles
Cited By...
- No citing articles found.