Assessing and scaling the knowledge pyramid: the good-guideline guide

All guidelines are not created equal

Intuitively, the clinician understands that some guidelines will be better than others. Some factors in this judgment are objective, such as the quality of the evidence that supports the recommendations. Other factors are subjective, such as the perspectives that the authors of CPGs bring to bear. Although these subjective factors may seem to harm the validity of a guideline, they actually provide the experienced human judgment necessary to make decisions easier at the practice level. Understanding that subjectivity should not be eliminated from the guideline development process,9 organizations such as the Canadian Task Force on Preventive Health Care and the US Preventive Services Task Force have identified and tried to minimize factors that might lead to bias in the recommendation development process.

The Appraisal of Guidelines, Research and Evaluation (AGREE) Collaboration10 has created and validated tools for clinicians to rate guidelines according to their process of development by identifying the factors that are considered important in judging their quality. Within the AGREE instrument, the reader of a guideline is asked to consider, among other things, the scope and purpose of the guideline, including its objectives and the patient population to whom the guideline will be applied; the involvement of all relevant stakeholders necessary to the development of the guidelines; the format used to present the guideline, including clarity of the wording of its recommendations; and the applicability of the guideline in the setting in which its use is being reviewed, including organizational and cost barriers to implementation.

The threat of bias is addressed by the AGREE instrument in 2 additional domains: the rigour and the editorial independence of the development process. The measures of rigour address such questions as, Was the literature search on which the guideline is based thorough and systematic? Are the criteria for selection of articles clearly described? How have the recommendations been formatted? Is each recommendation clearly and explicitly linked to supporting sources of evidence, and is the grade of the evidence noted? Have health benefits, side effects and risks been considered in formulating the recommendations?

The question of editorial independence, also a prominent feature of the AGREE instrument, addresses the issue of whether and how the guideline development process, and the formulation of recommendations contained in the CPG, have identified and maintained independence from funding sources and other conflicts of interest. This is an issue on which editors, guideline dissemination groups and practitioners must maintain a constant vigilance.

Still, the clinician is left with a time-consuming task. When applying an instrument such as AGREE's, it helps to have some training in critical appraisal. Guidelines, like the primary research studies and systemic reviews that have spawned them, have proliferated exponentially. For example, a literature search of MEDLINE (performed Dec. 6, 2005) for guidelines for managing patients with cholesterol problems yielded more than 300 CPGs in English published in the past decade (1995–2005) alone!

Assisting physicians with the problem of sifting and distilling the quality of numerous available guidelines has been the work of the Guidelines Advisory Committee (GAC),11 a joint initiative of the Ontario Medical Association and the Ministry of Health and Long-Term Care. For selected clinical topics relevant to clinicians, patients and the health care system, the GAC identifies, rates and endorses the best available guideline. The GAC process includes a systematic search of published guidelines, from which evidence-based CPGs are selected. This smaller list of CPGs is then reviewed by practising physicians trained in the use of the AGREE guideline assessment instrument. The GAC has adopted the AGREE instrument as a foundation for its efforts to review, endorse and summarize the recommendations of guidelines. The GAC then considers the final AGREE quality rating as well as the relevance of the guideline to the provincial practice environment before it endorses and summarizes the key messages of the guideline for Ontario physicians. In this manner, practising clinicians need not be trained in critical appraisal nor invest a great deal of time to have access to concise, evidence-based guidance.

Not all guidelines that are published document all the process steps and measures identified by the AGREE instrument, as yet. Clearer and more regular declarations are necessary that pertain to the editorial independence of the guideline development process, and to commercial bias in producing the guidelines.12^,13^,14 Needed as well are further critiques and refinements of guideline assessment instruments.15^,16

The GAC and its partners do not suggest that its process and its use of the AGREE instrument are perfect answers to the challenges of information overload and the elimination of commercial bias in the development or endorsement of guidelines. Nevertheless, their results do provide a “leg up” in scaling the pyramid of knowledge.

@ See related articles pages 333 and 335