Alberta Physician Achievement Review

We thank Geoff Norman and John Cunnington for their interest in the Physician Achievement Review program of the College of Physicians and Surgeons of Alberta.1 At the request of CMAJ's editors, we condensed our 2 original submissions, one describing the purpose and operational aspects of the program and the other providing statistical results, into a composite report.2 Space limitations precluded inclusion of extensive technical results, but we would be pleased to correspond with interested readers directly and provide additional technical data.

Norman and Cunnington asked about concurrent validity and inter-rater reliability. Concurrent validity, which is the extent to which there are correlations between self, patient, peer, consultant and co-worker assessments, was investigated using confirmatory factor analysis. The factors identified for the patient surveys were positively and significantly correlated with the factors identified for the peer surveys (r = 0.25, p < 0.05), the patient factors were positively and significantly correlated with the co-worker factors (r = 0.20, p < 0.05) and the co-worker factors were positively and significantly correlated with the peer factors (r = 0.31, p < 0.05). In other words, different groups of raters tended to rate a physician in the same way.

Inter-rater reliability addresses the issue of whether different raters of the same physician tend to rate the physician the same way. Our results indicated that when a physician's performance was rated very high or very low, most of the raters assessed the physician the same way. For example, when a physician was rated low in the "clinical competency" category he or she was rated low by most peers. For this particular category there was up to 100% agreement among peers in placing physicians in the lowest group.

The Physician Achievement Review program has now been implemented as described[2–4] and the survey results provide a basis for further assessment by practice visits for some physicians. Our operational experience will be reported in due course.