Validity of utilization review tools

We agree with Norman Kalant and colleagues that it is important to validate the use of utilization review tools in Canada,1 but we feel that the methodology they used for their study does not reflect the manner in which the tools are implemented and cannot adequately support their conclusions.

Whereas actual utilization review activity uses current criteria, the researchers chose criteria that are now 4 years old. Utilization review at the 2 largest Vancouver hospitals has shown that approximately 10% of inpatient days meet criteria for subacute care, yet the researchers failed to use the subacute care criteria.

In addition, the sample size was very limited, both in number and scope (i.e., 75 charts were reviewed for cardiology only). Generalization as to the validity of the entire tool is thus suspect.

Finally, implementation in our health region includes a secondary review process that improves upon tool validity as well as inter-rater reliability tests for the reviewers. Kalant and colleagues did not include a secondary review process in their study and they questioned its usefulness given “the frequent divergence of clinical opinion among individual physicians.”1 How valid is it to use 3 cardiologists as a “gold standard”?

Although utilization review is not a perfect science, it is one of many important strategies that we can employ to determine how best to improve our health system.