Show us the evidence

We read the study of physician performance in Alberta1 with a growing sense of concern. The study appears to provide evidence of the reliability and validity of the approach using self-assessments and assessments by peers and patients. Unfortunately, appearances are deceptive; while the survey contained information that could have been used to assess reliability and validity, none of the analyses reported in the paper were actually relevant to these concerns.

The study obtained ratings from patients and peers. These multiple raters per physician could have been used to assess inter-rater reliability. Regrettably this was not done; instead the authors computed the Cronbach α statistic, which is simply a measure of the extent to which a rater who rates 1 item high will rate other items high.

Secondly, since all subgroups were presumably assessing aspects of competence, one would expect some correlation between assessments by the physician himself or herself, patients, peers, consultants and co-workers. This is an issue of concurrent validity and could have been addressed by computing the mean score for each physician by each rater type and then correlating these scores. While the correlations may be less than 1, to the extent that there is such a thing as competence, the correlations should be positive and in the mid-range. Again, this informative analysis was not conducted.

It is frustrating to see that critical analyses that could have been done, were not. It is also worrisome that the study, which provides essentially no evidence to substantiate the reliability and validity of the peer assessment method, might be misinterpreted by the casual reader as strong support for the approach.

The peer assessment approach may well be a useful strategy. Certainly the results of other studies provide support for the strategy. Regrettably, the present study provides no evidence to assess the utility of the method.