Further disquiet on the guidelines front

More discouraging news about clinical practice guidelines (CPGs). In this issue (page 157) Ian Graham and colleagues review the quality of drug therapy CPGs developed or endorsed by Canadian organizations from 1994 to 1998.1 Only 14.7% of the 217 guidelines reviewed met half or more of the 20 criteria defining the rigour of the development process, and independent reviewers recommended only 9.2% as being sound without modification. The drug therapy CPGs, it would seem, often retail opinion based on incomplete or haphazardly compiled scientific evidence. For much of the past decade, proponents of evidence-based medicine have decried the lack of uptake of CPGs. Given their quality, as reported by Graham and colleagues, perhaps we should instead applaud.

Or perhaps not. The authors assessed the guidelines using the Appraisal Instrument for Clinical Guidelines, in which 20 of the 37 appraisal criteria assess the rigour of methods used in guideline development. Almost all of the criteria are proxies for, rather than direct indicators of, the quality of the scientific evidence underlying the guidelines. Only 2 deal directly with quality: Are the sources of information used to select the evidence on which the recommendations are based adequate? and Are the methods for rating the evidence satisfactory? (The answer to both questions was No in over 80% of the CPGs that described this information.) However, even a guideline whose process and conclusions are based on good science would not pass muster if this connection were not transparent. Explicitness and transparency are the cornerstones of evidence-based medicine, and it is inexcusable to produce a CPG without coming clean about the methods and bases for recommendations. But it is possible for experts to produce a guideline compatible with the science without systematically reviewing and transparently reproducing the evidence, just as it is possible for a panel to have at hand a definitive, Cochrane-compatible literature review and arrive at unjustifiable conclusions.

Five of the criteria in the appraisal instrument deal with application or implentation of guidelines: for example, Does the guideline document suggest possible methods for dissemination and implementation? and Does the guideline document define measurable outcomes that can be monitored? Predictably, these criteria were universally unmet. CPG developers are no more responsible for the fate of their products than are authors of scientific articles for theirs. Social marketing is optional. At first blush it seems unreasonable to impose these requirements on CPG developers already charged with the difficult and at times fractious task of distilling science into practitioner-friendly recommendations.

But perhaps we should blush twice and upgrade our expectations in line with the appraisal instrument criteria. Our collective response to the evidence that practitioners only rarely follow guidelines, and sometimes think they do when they don't,2 is to produce more guidelines rather than rethink the whole enterprise. I have personally been involved in Western and national exercises (the latter funded by the Federal/Provincial/Territorial Advisory Committee on Health Services) that sought to address the cacophony of duplicated guidelines development by suggesting a common approach to reviewing the literature, populating CPG panels and other measures. Nothing came of either; here, as throughout the health care system, turf battles, parochialism and flagging will impede progress.

We might also remind ourselves to compare CPGs not with perfection but, rather, with current practice. Variations in practice are wide and persistent,3 almost certainly evidence of troublesome quality and accessibility problems. Appropriate use of just about any CPG developed by thoughtful people is likely to improve practice. Almost three-quarters of the guidelines assessed by the appraisers in the study by Graham and colleagues were recommended as is or (in most cases) with modifications despite general inattention to the appraisal instrument criteria. That both “pure” quality and transparency were generally deficient did not mean the guidelines were without clinical merit. Reflexive rejection of almost any guideline is as unwarranted as robotic adherence. Guidelines defensibly rated as unacceptable should be culled from the herd, but until we put in place high-quality, standardized development processes, all CPGs should be judged more on the basis of the wisdom of their recommendations than on the explicit processes of their creation.

Graham and colleagues conclude rather modestly with the suggestion that the quality of all CPGs in Canada be systematically assessed and that the results be disseminated widely and included in CMA Infobase (the CMA's online database of Canadian CPGs). Good ideas. But much more needs to be done. CPGs, even great CPGs, are ignored because there is neither reward for following them nor penalty for ignoring them. Information systems are too feeble to inform practitioners about their absolute and relative performance on most dimensions of practice, let alone adherence to CPGs. It is inevitable that the production of multiple CPGs on the same topic will be confusing, will lead to selective adoption of practices that reflect preferences and prejudices rather than evidence-based judgement and will create yet another excuse to do nothing about practice variations and perverse incentives.

If we are to persist in the CPG business, there should be one Canadian guideline for each area, nationally produced, federally funded and compatible with the criteria contained in the Appraisal Instrument for Clinical Guidelines. If practitioners are going to ignore CPGs, they might as well ignore the best.