Effect of a community oriented problem based learning curriculum on quality of primary care delivered by graduates: historical cohort comparison study

Introduction

Periodic reforms are made in medical education on the assumption that basic training affects how doctors will practice.1 Although considerable investment is made in such reforms, systematic assessment of expected benefits in actual practice is lacking, particularly for quality and outcome of care.2 3^{w1 w2} The lack of progress in investigating the effects on practice outcomes of the hotly debated reform to community oriented problem based learning.4 5^–8^w2-w9 has been the subject of recent editorials and reviews.2 3^{w1 w2} In problem based learning, fundamental changes are made in the way in which medical knowledge and skills are acquired (through problem based education rather than by discipline specific education), the method of learning professional development skills (self directed, small group), and the environment within which experience of practice is acquired (greater emphasis on community based practice compared with hospital based practice). Community oriented problem based learning was expected to better prepare graduates to provide comprehensive preventive, acute, and chronic disease care through community oriented patient centred learning. Yet concerns have been expressed that such a curriculum might reduce core competencies in medical diagnosis and treatment through gaps in relevant medical knowledge and diluted experience of acute care.4 8 9^w5 These concerns have been buttressed by evidence of lower licensing examination scores and suboptimal diagnostic reasoning in some studies of graduates of a problem based learning curriculum, but the effects have not been assessed in practice.9^{w5 w6 w10 w11}

In 1987 the University of Sherbrooke in Quebec, Canada, was one of the first medical schools to undergo a reform from a traditional curriculum (system based or disciplinary based) to a problem based learning curriculum.10 We tested the hypothesis that graduates of a problem based learning curriculum would provide better continuity of care and preventive care in primary care than graduates of a traditional curriculum. We also assessed whether the Sherbrooke graduates exhibited a decline in performance in selected indicators of diagnosis and disease management.

Methods

To assess the effect of transition to a community oriented problem based learning curriculum on practice performance, we carried out a longitudinal follow-up study of four classes of Sherbrooke graduates; three before the transition (graduates of 1988-90) and one after the transition (graduates of 1991). We compared the differences in the performance of Sherbrooke graduates before and after the transition with a control group of graduate doctors trained in the same period in medical schools with a traditional curriculum. Eligible doctors were those who graduated from one of the four medical schools in Quebec (Sherbrooke, Laval, Montreal, and McGill), completed postgraduate training in family medicine, and were licensed to practise as a family doctor in Quebec between 1990 and 1993 (see bmj.com for details of examination). The follow-up period for each graduation cohort was between 1 July of the certification examination year and 31 December 1996, providing an assessment of practice for the first four to seven years. Eligible doctors were identified by the Quebec College of Physicians. We excluded salaried doctors because their patients could not be identified and doctors who subsequently trained in another specialty. The provincial insurance agency la Régie de l'assurance maladie du Québec provided the data on all patients seen by each doctor, for each year of practice.

Practice covariates

To adjust for differences in the case mix of the doctors' practice populations, we retrieved the characteristics for each eligible patient in the 12 months before the first contact with the study doctor in each year. We then aggregated patient specific values for a given year to characterise each doctor's practice population by age and sex distribution, mean family income, educational achievement, geographical access to health care,15 propensity for the use of healthcare services (ambulatory care groups) groupings, number of doctors seen),^w24 comorbidity (Charlson index),^w25 and rate of admissions to hospital. We modelled annual measures of practice population characteristics as time dependent covariates. For relatively contraindicated prescribing, we did not adjust for differences in practice population because the prescription of relatively contraindicated drugs is rarely justified by patient characteristics.^w16 To adjust for potential differences in practice experience, we used the cumulative number of preceding months of active experience in practice as a continuous time dependent covariate in the analysis.15

Analysis

To test hypothesised changes in performance related to the transition from a traditional to a problem based learning curriculum, we used repeated measures generalised estimated equations extension^w26 of multiple linear regression analysis. Annual values of each outcome represented the repeated measures for each doctor. We assumed an autoregressive order 1 structure of the covariance matrix of residuals to account for the correlation between annual measures for the same doctor. To minimise the effect of imprecise observations, we excluded from the analysis those doctors with fewer than five eligible patients, and we weighted the data for other doctors by the logarithm of their practice size.15^w16 We adjusted each model for doctors' sex, certification examination score, cumulative months of experience in practice, and annual measures of case mix.

For each outcome we estimated three multivariate generalised estimated equations models. In the first two models we estimated the adjusted differences between medical schools for graduates trained before the transition (graduates of 1988-90) and those trained after the transition (graduates of 1991). Sherbrooke was the reference category, and three dummy variables represented graduates from the other three medical schools. The third model tested separately for graduates from each of the four schools whether there was a significant within school difference in performance of graduates in the period before the transition compared with performance after the transition. To estimate the differences between before and after the transition, we analysed the data from all schools and cohorts in the same model. The model included a series of binary variables, each assigning the value of 1 to the 1991 graduation cohort (after transition) from a given school. In additional analyses we assessed if the change after transition in a given outcome at a particular school was different from the change at Sherbrooke after the transition by testing a series of transition by school interactions in a multivariable generalised estimated equations model that included the main effects of schools (before transition) and the effect of the transition among Sherbrooke graduates. In sensitivity analysis we assessed whether undergraduate and postgraduate training in the same university confounded associations. We used SAS procedure Genmod, version 8.0 in all generalised estimated equations analyses.

Results

Between 1988 and 1991, 2013 doctors completed undergraduate training in Quebec medical schools and 930 (46.2%) of these doctors entered postgraduate training in family medicine. Among these doctors, 813 (87.4%) were licensed to practise in Quebec (table 1). Doctors entering primary care practice were predominantly women, particularly in the period after the transition to a community oriented problem based learning curriculum, except for McGill graduates. Most family doctors completed undergraduate and postgraduate training in the same medical school. The mean age at the completion of postgraduate training was between 26 and 28 years. Family medicine certification examination scores for doctors from the four schools varied by slightly more than one quarter of a standard deviation. Although the mean score for Montreal graduates dropped by more than one quarter of a standard deviation in the period after the transition, no differences in scores between the periods before and after transition achieved statistical significance (table 1). Most doctors entered fee for service practice, the lowest proportion being from Laval.

Most of the doctors worked in two or more practice settings annually (table 2). McGill graduates had a distinctive profile, as they saw a higher proportion of patients in intensive care and inpatient settings, and most of their patients were from urban regions. In contrast, graduates from Laval and Sherbrooke saw the greatest proportion of patients from rural and remote areas. Montreal graduates saw the greatest number of patients per year and had the highest mean volume of patients per day.

Before the transition, Sherbrooke graduates had the highest mammography screening rate (mean 158 per 1000 v 94.6 to 114.0 per 1000 for the other schools), significantly higher than Laval graduates (table 3). After the transition, Sherbrooke graduates showed a significant and substantial increase in mammography screening rate (table 3). Increases in screening rates for the three other schools in the period after the transition were substantially smaller than for Sherbrooke and statistically non-significant.

Continuity of care was similar among graduates from the different medical schools in the period before the transition, with the exception of significantly lower continuity of care for McGill graduates (table 4). After the transition, Sherbrooke graduates showed a statistically significant improvement in continuity of care (table 4). Although McGill graduates also showed substantive improvement in the period after the transition, the change in continuity of care was statistically non-significant, possibly due to fewer graduates.

In the period before the transition, McGill graduates prescribed disease specific drugs to 10.3 more patients per 1000 than symptom relief drugs, whereas Sherbrooke graduates prescribed fewer disease specific drugs than symptom relief drugs (table 5). After the transition, however, Sherbrooke showed a fourfold greater improvement in the difference between the two rates than did McGill (32.2 v 8.0, table 5). This allowed Sherbrooke graduates to surpass the differences in rates for McGill graduates in the period after the transition (16.1 v11.8, table 5). The increase for Montreal graduates was also statistically significant although smaller than for Sherbrooke graduates, even if the difference in the magnitude of the changes before transition between the schools was statistically non-significant.

Sherbrooke graduates had significantly higher rates of contraindicated prescribing in the period after the transition than did graduates from some of the other schools (table 5). Contraindicated prescribing rates declined in the period after the transition, with the largest declines in Laval (11.8 per 1000). We found no evidence of any increase at Sherbrooke, where a non-significant decrease of 5.2 per 1000 was observed.

Results of additional analyses that tested interactions between the effect of the individual schools and cohorts after the transition were generally consistent with the results shown in tables 3, 4, and 5. The point estimates of the interaction effect suggested that changes in most outcomes in the other schools were two to three times smaller than in Sherbrooke. Although many interactions were statistically significant in conventional regression analysis, they became marginally statistically nonsignificant in the more conservative generalised estimated equations analysis that accounted for correlation of repeated outcomes of the same doctor. The magnitude of changes in practice outcomes in the period before and after the transition was not materially affected by the location of postgraduate training (same versus different medical school).

Discussion

Transition to a community oriented problem based learning curriculum at Sherbrooke University, Quebec, was associated with substantial and statistically significant improvements in preventive care and continuity of care, without the hypothesised decline in performance indicators associated with management and diagnosis. Indeed, the transition was associated with a significant improvement in the difference between disease specific and symptom relief prescribing rates which has been shown to be a marker of diagnostic ability.15^w16 Although significant improvements were seen in three of four indicators, we were not able to show that these changes were significantly larger than simultaneous changes in other schools, owing to a small number of graduates after the transition and low power. Future studies would need to provide several cohorts of graduates in the post-transitional period to provide conclusive evidence of such differences.

We were able to compare the quality of care delivered by primary care doctors who graduated from community oriented problem based and traditional curriculums. By assembling cohorts of graduates who entered primary care both before and after the transition to a problem based learning curriculum at Sherbrooke University, we were able to control for differences between graduates of different medical schools before the transition as well as for temporal trends in the quality of care. The use of administrative databases avoided participation and self report biases, as outcomes were measured among all graduates who entered primary care, and among all patients seen by these graduates in the first 4-7 years of practice.

Although requirements for admission were not altered as part of the reform,10 the particular attributes of the reform that may have generated the positive changes in practice performance can not be readily disentangled for Sherbrooke, or for most other schools that have implemented innovations to their curriculum.^w4 That significant differences existed in outcomes of graduates of different medical schools in the period before the transition, even after adjustment for certification examination scores, suggests that school level factors beyond the formal curriculum influence subsequent practice activity. The greater emphasis on community based practice experience in the problem based learning curriculum may have contributed to improvements in continuity and preventive care of Sherbrooke graduates. To inform future policy for medical education, the effect of the clinical training environment on subsequent medical practice needs to be systematically investigated.21

In our study the performance of new graduates fell short of desirable outcomes. In particular, mammography screening was below the 70% coverage expected to reduce mortality from breast cancer,22 a universal problem that has led to the introduction of screening centres for self referral for mammography.^{w27 w28} The prevalence of contraindicated prescribing for older patients is considered a systemic problem that contributes to avoidable morbidity.23 The variability in contraindicated rates between graduates from different medical schools suggests that basic training may be an additional factor that needs to be examined to improve future prescribing behaviour.

Our study has limitations. Using administrative databases we could assess only a subset of outcomes that would be of importance in judging the quality of primary care. We excluded salaried doctors from the evaluation. Although the proportion of salaried doctors varied by medical school, investigation has shown that salaried practice is predominantly influenced by the doctor's sex.^w29 Our evaluation of the effect of a community oriented problem based learning curriculum was limited to one school's experience and the first graduation class of that curriculum. To enhance the generalisability of our findings, similar evaluation should be carried out in other schools with a problem based learning curriculum. Future research should assess whether transition to a problem based learning curriculum is associated with other expected changes in the quality of care, particularly in populations of patients where important gaps exist between optimal and current therapy, such as those with asthma or diabetes and those needing management for cardiovascular risk.24^w30