- © 2008 Canadian Medical Association
Abstract
Background: Systematic reviews on the effects of problem-based learning have been limited to knowledge competency either during medical school or postgraduate training. We conducted a systematic review of evidence of the effects that problem-based learning during medical school had on physician competencies after graduation.
Methods: We searched MEDLINE, EMBASE, CINAHL, PsycINFO, Cochrane Databases, and the tables of contents of 5 major medical education journals from earliest available date through Oct. 31, 2006. We included studies in our review if they met the following criteria: problem-based learning was a teaching method in medical school, physician competencies were assessed after graduation and a control group of graduates of traditional curricula was used. We developed a scoring system to assess the quality of the studies, categorized competencies into 8 thematic dimensions and used a second system to determine the level of evidence for each competency assessed.
Results: Our search yielded 102 articles, of which 15 met inclusion criteria after full text review. Only 13 studies entered final systematic analysis because 2 studies reported their findings in 2 articles. According to self-assessments, 8 of 37 competencies had strong evidence in support of problem-based learning. Observed assessments had 7 competencies with strong evidence. In both groups, most of these competencies were in the social and cognitive dimensions. Only 4 competencies had moderate to strong levels of evidence in support of problem-based learning for both self-and observed assessments: coping with uncertainty (strong), appreciation of legal and ethical aspects of health care (strong), communication skills (moderate and strong respectively) and self-directed continuing learning (moderate).
Interpretation: Problem-based learning during medical school has positive effects on physician competency after graduation, mainly in social and cognitive dimensions.
Problem-based learning curricula have been introduced in many medical schools around the world.1,2 However, their adoption was met with some concern, primarily because of the substantial manpower needed. For example, student contact hours are 3–4 times greater for educators in a problem-based learning curriculum than for educators in a traditional curriculum. As a consequence, the economic viability of problem-based learning becomes a major concern when class sizes exceed 100 students.3 Given the limited resources available,4 evidence-based evaluation of the effects of problem-based learning during medical school on improving physician competency would certainly strengthen any justification for the adoption of such programs.5–9
Past reviews of problem-based learning focused only on its effects during medical school or postgraduate training.10–16 In addition, 4 of those reviews also studied student and educator preferences and indicated that medical students11,14,15 and educators16 generally prefer problem-based learning to traditional teaching methods. The reviews did not study the effects of problem-based learning during medical school on the competencies of practising physicians. Colliver emphasized this lack of evidence when he cautioned that student satisfaction cannot be extrapolated as a predictor of physician competency.17 To our knowledge, there has been only one systematic review, published in 1993, that indirectly reported the effects of problem-based learning during medical school on physician competency after graduation. The study was based on a small sample of doctors in their early postgraduate years.16 Since then, many rigorous studies have evaluated the effects of problem-based learning during medical school up to 20 years after graduation. We performed a systematic review of controlled studies to determine whether problem-based learning during medical school leads to greater physician competencies after graduation.
Methods
In our study, we used Maudsley's definition of problem-based learning, which she defined as both a method and philosophy involving problem-first learning via work in small groups and independent study.18
Search
To identify relevant studies, we searched each of the following databases, from the earliest available date through Oct. 31, 2006: MEDLINE (1966), EMBASE (1980), CINAHL (1982), PsycINFO (1967), and the Cochrane Database of Systematic Reviews, the Chochrane Database of Abstracts of Reviews of Effects and the Cochrane Central Register of Controlled Trials (1993). We also searched the tables of contents of 5 major medical education journals (Medical Education, Academic Medicine, Medical Teacher, BioMed Central Medical Education and Advances in Health Sciences Education) for papers published from date of first issue through Oct. 31, 2006. We used the search terms “problem-based learning,” “practice-based,” “self-directed,” “learner centred” and “active learning” to identify problem-based learning studies and combined them with other key terms such as “medical education,” “undergraduate,” “postgraduate,” “practising,” “physicians” and “doctors.” We did not search by subject heading because not all databases use “problem-based learning” as a subject heading. For example, MEDLINE introduced the term as a medical subject heading only in 1995. We also manually searched the references of retrieved articles to trace potentially relevant papers.
Selection
We selected studies if they met the following inclusion criteria: use of problem-based learning as a teaching method in medical school, assessment of study population after graduation from medical school (including first-year residents and pre-registration house-staff) and use of a control group19 from a traditional (non-problem-based learning) curriculum. We excluded studies that were qualitative, involved nonphysicians, assessed competency before graduation, used problem-based learning only during postgraduate medical education or reported combined outcomes for medical students and postgraduate trainees. Because of concerns related to contamination including curriculum changes and co-interventions such as increased community focus or emphasis on communication skills, we excluded studies in which all outcomes could be contaminated by concurrent curriculum changes. And although we included studies that contained some outcomes that were contaminated by co-interventions, we excluded these contaminated outcomes from the final analysis. We included nonrandomized controlled trials because there were very few randomized controlled trials in medical education; we felt that this approach was acceptable, given that most educational research has methodological limitations because purity of curricular change and random assignment of students are rarely possible.8,20–22 Medical schools in the United States are mainly entry-level graduate programs, whereas those in the United Kingdom traditionally accept medical students without a university degree. As such, age and a prior degree could be major confounding factors when comparing studies between these 2 medical education systems.23 Thus, we included only studies in which the control group came from the same medical education system. No restrictions were placed on study design, language or country. If there was overlap in study populations and reported outcomes, we considered combining the findings from those studies for our final analysis.
Data abstraction
One of us (G.C.H.K.) developed a standardized data form to extract information from the articles we included in our systematic analysis. We extracted the following information: sample size, source of control group (whether the control group came from the same or a different university), study design, whether random allocation was performed, years spent in problem-based learning curriculum, number and type of competencies assessed, type of assessment, response rate, whether adjustment for potential confounders was performed, and methodological inconsistencies. If such information was missing, we obtained it by looking up references in the paper that previously reported the information or, in the case of verifying the number of years spent in a problem-based learning curriculum, by consulting the medical schools' websites. One of us (G.C.H.K.) performed the data abstraction, and the rest of us (H.E.K., M.L.W. and D.K.) independently checked the data extracted. We did not perform a meta-analysis because of the wide variations in the constructs of outcome measures and study reporting, which made calculating the effect sizes impossible for more than half of the selected studies.
Validity assessment
Because most validated tools for assessing study quality were designed for clinical interventions, the only tool available to assess the quality of problem-based learning studies was the one that Smits and colleagues13 developed to study the effectiveness of problem-based learning in postgraduate medical education. However, of the 5 study-quality indicators they proposed, only 2 could be applied across the studies included in our review: response rate and statistical adjustment. As suggested by Reed and colleagues,24 we therefore adapted Smits and colleagues' instrument by adding 8 study-quality indicators of our own developed on the basis of our experience in epidemiology and medical education research.
The study-quality indicators we added were developed to adhere as closely as possible to validated quality-assessment instruments for clinical studies, such as the Scottish Intercollegiate Guidelines Network 50 instrument for cohort studies, and to apply to the full spectrum of heterogeneous studies. Like Smits and colleagues did with their instrument, we assigned some quality indicators (randomization, sample size, objective assessment and response rate) greater weight than others because we felt that they were of greater importance. We acknowledge that this approach is subjective, but our decisions were based largely on extensive discussion, reasonable assessment and consensus. We did not place restrictions on the timing of the post-graduation assessment because any differences due to the passage of time would be controlled by the comparison group.
All 4 of us independently assessed the quality of the studies using the adapted scoring model (Appendix 1, available at www.cmaj.ca/cgi/content/full/178/1/34/DC2). We subsequently classified the studies into 2 levels of quality. Similar to the cut-off point used by Smits and colleagues, we considered studies to be high in quality if they scored above the midpoint of the scoring range.
Data synthesis
One of us (G.C.H.K.) developed a thematic categorization of physician competencies based on the 7 dimensions of physician competence described by Epstein and Hundert.25 We combined 6 of their dimensions to form 3 for our study because of the lack of valid items within the 6 dimensions. The relationship and affective/moral dimensions were combined to form the social dimension, the context and habits-of-mind dimensions became the managerial dimension, and the integrative dimension was subsumed under the cognitive dimension. We also extracted research, teaching and knowledge competencies as independent dimensions to explore the effects of problem-based learning in these 3 important areas. We created a new “overall” dimension because some studies included in our review reported global competency. We categorized the outcomes from studies that we entered into the final stage of our systematic analysis according to whether the competencies of graduates of problem-based learning curricula were better than, the same as or worse than those of graduates in the control group. This categorization of competencies was stratified according to study quality. We present the results for self-and observed assessments separately because medical education research suggests that students and physicians have a limited ability to assess themselves accurately.26,27 To determine the levels of evidence supporting the effect of problem-based learning during medical school on various physician competencies, we used a system adapted from Smits and colleagues in which the level of evidence is determined based on the number of studies in support of the data, the quality of those studies and the number of conflicting studies13 (Appendix 2, available online at www.cmaj.ca/cgi/content/full/178/1/34/DC2).
Results
Our selection process is outlined in Figure 1. Our keyword searches yielded 2675 articles. After applying the inclusion criteria to the abstracts of these articles, we excluded 2573. We obtained the full-text versions of the remaining articles and excluded a further 87 articles, which left 15 eligible articles for our final systematic analysis.28–42
Figure 1: Search and selection of studies for systematic review. *Reasons for exclusion do not add up to total because some articles were excluded for multiple reasons.
There was overlap in study populations and reported outcomes among 4 of the 15 articles we retained for our final analysis. Although the studies by Mennin and colleagues35 and Santos-Gomez and colleagues36 were based on overlapping cohorts of graduates from the University of New Mexico, we treated these as 2 separate studies because both used different years of assessment and different types of assessment, and because only 4 of 17 self-assessed competencies overlapped. In 2 other cases, we opted to combine the findings from separate reports. In the first case Watmough and colleagues reported graduates' supervisor assessments and self-assessments in separate articles, which we treated as 1 study,32,33 and in the second case Woodard and colleagues' 1990 study included previous findings from a paper they published in 1981.41,42 As a result, our final analysis is based on 13 studies: 4 from the United States,28,29,35,36 3 from Canada,38,40–42 3 from Great Britain,30,32–34 2 from the Netherlands37,39 and 1 from Australia31 (Table 1).
Table 1.
Study characteristics
The study populations ranged from first-year residents to physicians who had been practising for up to 20 years. Graduates of problem-based learning curricula attended medical schools that either offered problem-based learning in the first 2 years or throughout the duration (up to 5 years) of medical school. Types of assessment fell into 2 categories: self-assessment or observed assessment. Observed assessment was usually in the form of a supervisors' survey or an objective measure, such as a performance indicator database34 or validated test.38 One study included a nurses' survey.36
Two studies had major methodological inconsistencies. Schmidt and colleagues39 surveyed participants in the problem-based learning group in 1999 but surveyed participants in the control group 5 years later. Woodard42 used a different methodology to select controls between 2 graduating years of doctors. There was excellent agreement among all 4 of us for scoring study indicators (average chance-corrected [kappa] = 0.89). Table 2 classifies the study outcomes according to whether the competencies of graduates of problem-based learning curricula were better than (positive), the same as (no different) or worse than (negative) those of graduates in the control group (a more detailed version of Table 2 is in Appendix 3, available online at www.cmaj.ca/cgi/content/full/178/1/34/DC2). The outcomes of 6 studies28,31–35,37could potentially be confounded by other reported changes in problem-based-learning curricula (see remarks column of Table 2 for details). These outcomes were excluded when we assessed the levels of evidence.
Table 2.
Outcomes
We categorized the competencies assessed in the 13 studies into 8 dimensions: overall, technical, social, cognitive, managerial, research, teaching and knowledge competencies (Table 3) (more detailed versions of Table 3 are available in Appendices 4 [self-assessment] and 5 [observed assessment], available at www.cmaj.ca/cgi/content/full/178/1/34/DC2). Of the 8 dimensions, the social dimension showed the strongest evidence in support of problem-based learning. Within this dimension, there was strong evidence that problem-based learning was superior for 4 of 8 competencies: teamwork skills, appreciation of social and emotional aspects of health care, appreciation of legal and ethical aspects of health care, and appropriate attitudes toward personal health and well-being. Moreover, within the same dimension, there was moderate evidence in support of problem-based learning for communication and interpersonal skills. In the technical dimension, continuity of care had a strong level of evidence in support of problem-based learning. Of the 12 competencies in the cognitive dimension, coping with uncertainty, use of computers or information resources and understanding of evidence-based medicine had strong levels of evidence for problem-based learning. In the knowledge dimension, there was strong evidence that problem-based learning graduates rated themselves as possessing less medical knowledge than graduates in the control group.
Table 3.
According to self-assessments, 8 competencies had a strong level of evidence in support of problem-based learning: continuity of care, teamwork, appreciation of social and emotional aspects of health care, appreciation of legal and ethical aspects of health care, attitudes toward personal health and well-being, coping with uncertainty, use of computers or information resources and understanding evidence-based medicine (Table 3). According to the observed assessments, 7 competencies had a strong level of evidence in support of problem-based learning. All but 1 (appreciation of legal and ethical aspects of health care) were different: diagnostic skills, communication skills, appreciation of cultural aspects of health care, responsibility or reliability, and self-or peer appraisal (Table 3). These different findings based on type of assessment were not unexpected because medical education literature suggests physicians have limited self-assessment abilities.26,27
Only 4 competencies had moderate to strong levels of evidence in support of problem-based learning according to both self-and observed assessments: coping with uncertainty (strong), appreciation of legal and ethical aspects of health care (strong), communication skills (moderate and strong respectively) and self-directed continuing learning (moderate).
According to self-assessments, the knowledge dimension showed a strong level of evidence against problem-based learning in terms of graduates' possession of medical knowledge (Table 2 and Table 3).
Interpretation
Our study offers objective evidence that problem-based learning during medical school has positive effects on physician competencies after graduation, especially in the social and cognitive dimensions. The absence of evidence of positive effects of problem-based learning on physician competencies in the technical and teaching dimensions is likely due to these dimensions requiring psychomotor skills, which are more effectively instilled using deliberate practice methods including formal procedural skills training and preceptorships. Our findings indicated a strong level of evidence against problem-based learning in terms of graduates' self-assessments of their possession of medical knowledge. The meta-analysis by Dochy and colleagues43 found nonrobust evidence that problem-based learning had negative effects on knowledge possession, but robust evidence that the method had positive effects on students' knowledge application. In our study, graduates of problem-based learning curricula assessed themselves as possessing less medical knowledge than graduates from the control group; however, supervisors generally found little difference between the 2 groups because, we suspect, they used knowledge application as their measure of physician knowledge. The fewer number of competencies with strong levels of evidence using observed assessments relative to self-assessment supports the findings by Davis and colleagues27 that physicians have limited self-assessment abilities. We did not find any differences in outcomes for problem-based learning graduates based on the country of study; however, the number of studies from each country was too few to form the basis for any meaningful comparison.
Most studies and systematic reviews of problem-based learning focused on the effect of problem-based learning on knowledge but neglected other aspects of physician competency.9,14,43 Controversy has often surrounded the issue of whether problem-based learning results in a weaker knowledge base in students, especially of basic medical sciences.11,16 In our study, it is reassuring that, although graduates of problem-based learning curricula assessed themselves as possessing less medical knowledge than graduates of traditional curricula, the evidence from the objective assessments did not support this finding.
Our study has a number of limitations. First, the heterogeneity of problem-based learning is a challenge inherent in all problem-based learning research. The definition and implementation of problem-based learning varies widely among medical schools and educators,18 and a number of factors that are difficult to measure and control affect the success of problem-based learning curricula.44,45 Second, most of the studies used surveys as the main measurement to assess graduates. Surveys are limited in reliability and validity, subjective and prone to rater biases.46 Third, the study-quality scoring system, the evaluation system of levels of evidence and the thematic categorization we used were not validated; however, they were adapted from similar assessment tools used in previous research on problem-based learning13,14,24 and physician competency.25 Furthermore, in our model we assigned greater weight to some quality indicators (randomization, sample size, objective assessment and response rate) than to others because we felt that they were of greater importance. We acknowledge that this approach is subjective, but our decisions were based largely on extensive discussion, reasonable assessment and consensus. Fourth, we did not perform a grey literature search, which may have contributed some information on publication bias. Fifth, with few exceptions, the introduction of problem-based learning into medical school curricula was accompanied by other changes such as increased community focus or emphasis on communication skills, which makes it impossible to fully dissociate the findings attributable to problem-based learning from those attributable to other curricula changes. Lastly, because we were unable to determine the adequacy of sample size, there may have been nonsignificant but true differences in outcomes because of small samples.
In summary, problem-based learning during medical school has positive effects on physician competencies, especially in the social and cognitive dimensions. Future research should look beyond knowledge competency and measure the effects of problem-based learning on other dimensions of physician competency.
@ See related article page 61
Footnotes
-
Une version française de ce résumé est disponible à l'adresse www.cmaj.ca/cgi/content/full/178/1/34/DC1
This article has been peer reviewed.
Contributors: All authors were involved in study conception, design and acquisition of data. All authors contributed to data analysis and interpretation, revised the article for important intellectual content and approved the final version for publication.
Acknowledgements: We acknowledge the support of the National University of Singapore Library in providing literature search support. We thank Goh Lee Gan MMed for his help in the early stages of research and Tai Bee Choo PhD for her statistical support. Both Goh Lee Gan and Tai Bee Choo are with the Community, Occupational and Family Medicine Department, Yong Loo Lin School of Medicine, National University of Singapore.
Competing interests: None declared.
REFERENCES
In this issue
Article tools
Jump to section
Related Articles
Cited By...
- Focused educational intervention improves but may not sustain knowledge regarding falls management
- Attitudes Towards Problem-Based Learning of Faculty Members at 12 U.S. Medical and Dental Schools: A Comparative Study
- Dental Faculty Members Pedagogic Beliefs and Curriculum Aims in Problem-Based Learning: An Exploratory Study
- Top-Cited Articles in Problem-Based Learning: A Bibliometric Analysis and Quality of Evidence Assessment
- The Didactic Makeover: Keep it Short, Active, Relevant
- Problem-Based Learning: Lessons From Medical Education and Challenges for Management Education
- Teaching Evidence-Based Management With A Focus on Producing Local Evidence
- Problem-based learning in continuing medical education: Review of randomized controlled trials
- Problem-Based Learning in Dental Education: A Systematic Review of the Literature
- Problem-Based Learning in the NICU
- An Overview of Case-Based and Problem-Based Learning Methodologies for Dental Education
- Rehabilitation impact indices and their independent predictors: a systematic review
- Case Histories in the Education of Advanced Practice Nurses
- Protected block time for teaching and learning in a postgraduate family practice residency program
- Should problem based learning be universally adopted? No
- Should problem based learning be universally adopted? Yes
- Association Between Students' Dental Admission Test Scores and Performance on Comprehensive Clinical Exams
- Changes in Educational Methodologies in Predoctoral Dental Education: Finding the Perfect Intersection
- Evaluation of a New Assessment Tool in Problem-Based Learning Tutorials in Dental Education
- Association Between National Board Dental Examination Part II Scores and Comprehensive Examinations at Harvard School of Dental Medicine
- Filling the gaps in undergraduate teaching of clinical biochemistry
- The Educational Challenge of Dental Geriatrics
- Psychiatry benefits from problem-based learning
- Problem based learning
- Le temps est-il venu d'une autre revolution des programmes d'etudes en medecine?
- Is it time for another medical curriculum revolution?
- Problem-based learning makes a difference. But why?