Partial updating of clinical practice guidelines often makes more sense than full updating: a systematic review on methods and the development of an updating procedure

doi:10.1016/j.jclinepi.2013.06.021

Journal of Clinical Epidemiology

Volume 67, Issue 1, January 2014, Pages 33-45

https://doi.org/10.1016/j.jclinepi.2013.06.021 Get rights and content

Abstract

Objectives

To conduct a systematic review of the methods used to determine when and how to update clinical practice guidelines (CPGs) and develop a procedure for updating CPGs.

Study Design and Setting

We searched MEDLINE, Embase, and the Cochrane Methodology Register for methodological publications on updating CPGs. Guideline development manuals were obtained from the Web sites of guideline-developing organizations. Using the information obtained from these records, a procedure for updating CPGs was developed.

Results

A total of 5,116 journal articles were screened, and seven articles met the criteria for inclusion. Forty-seven manuals were included; of these, eight included details about the methods used to update the guidelines. Most of the included publications focused on assessing whether the CPGs needed updating and not on how to update them. The developed procedure includes a systematic monitoring system and a scheduled process for updating the CPGs, which includes guidance on how to determine the type and scope of an update.

Conclusion

Partial updating often makes more sense than updating the whole CPG because topics and recommendations differ in terms of the need for updating. Guideline developers should implement a systematic updating procedure that includes an ongoing monitoring system that is appropriate for the nature of the guideline topics and the capabilities of the developers.

Introduction

Clinical practice guidelines (CPGs) are designed to reflect current research. To delineate evidence-based CPGs from other forms of clinical guidance, the Institute of Medicine (IOM) recently stated that guideline recommendations should be “informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” [1].

Based on a definition of updating systematic reviews [2], updating CPGs can be defined as a process that aims to identify new evidence to incorporate into a previously completed guideline. This process constitutes one essential step in ensuring the validity and quality of a CPG [3].

Out-of-date guidelines could be one determinant of inadequate patient care. Accordingly, it is reasonable to ensure the validity of CPGs by conducting regular evaluations and performing updates when appropriate. However, keeping recommendations up-to-date with rapidly changing evidence is difficult [4]. It requires ongoing monitoring and reviews of the relevant literature and developments in clinical practice. The most efficient way to update CPGs is still debated. Almost 10 years ago, an international survey of guideline programs showed that the updating process is often not standardized [5]. Since then, there have been noticeable efforts to improve the methods for keeping guidelines up-to-date [6].

Shekelle et al. [7] investigated the appropriate time for updating a CPG and mentioned several occasions that may demand an update of recommendations. However, there are no validated methods for systematically determining when new evidence is published in an ongoing manner [8], [9].

Particularly, when there is an acute necessity for a revision because major or invalidating evidence becomes available, the lack of standardized processes to bring the CPG up to date poses a problem because it takes a substantial amount of time to act. This problem became obvious, for example, during a scandal in which evidence from faked pain studies [10] was included in a German evidence-based CPG [11]. The guideline developers were ill prepared because there were no standardized procedures for reacting to such situations. However, National Institute for Clinical Excellence (NICE), for example, addresses this difficulty in their CPG manual and differentiates between scheduled and exceptional updates, which are partial updates of a CPG conducted before the expected expiry date when significant evidence emerges [12].

The objectives of this study were as follows:

1.
to conduct a systematic review of the methods used to determine when and/or how to update CPGs and
2.
to develop a procedure for updating CPGs based on the information generated from the systematic review.

Section snippets

Literature search

To identify journal articles describing methods and strategies for updating CPGs, systematic searches were performed through August 2012 in MEDLINE, Embase, and the Cochrane Methodology Register. For details about the search strategies, see Appendix A at www.jclinepi.com.

To identify guideline development manuals published by guideline organizations, we searched guideline databases [Guidelines International Network (G-I-N), AWMF (German Association of the Scientific Medical Societies), //www.leitlinien.de

Results of the literature search

The screening process is summarized in the flow chart (Fig. 1).

The search in the bibliographic databases resulted in 5,116 records. After the initial screening of the titles and abstracts, 59 full-text articles were reviewed and 52 journal articles were excluded (Appendix B at www.jclinepi.com). Seven articles met the criteria for inclusion.

In addition, a total of 398 Web sites of guideline development organizations were screened, and 57 manuals for developing clinical guidelines were

Agreements and disagreements between the manuals on updating key elements

The eight manuals with clear statements on updating CPGs agreed on most key elements in the CPG updating process.

Conclusion

Updating guidelines should be regarded as an essential part of the development process and should be considered when allocating resources. Conducting partial updates is preferable. The decision-making process for determining the type and scope of an update will increase costs initially, but overall savings can be expected in the future.

Guideline developers should implement a systematic updating procedure that includes an ongoing monitoring system, taking into account the nature of the guideline

Acknowledgments

The authors thank Mrs. Sunya-Lee Antoine for providing linguistic revision.

References (43)

D. Moher et al.
Systematic reviews: when is an update an update?
Lancet
(2006)
D. Moher et al.
A systematic review identified few methods and strategies describing when and how to update systematic reviews
J Clin Epidemiol
(2007)
K. Kaiser et al.
Versioning computer-interpretable guidelines: semi-automatic modeling of ‘Living Guidelines’ using an information extraction method
Artif Intell Med
(2009)
E. Parmelli et al.
Updating clinical recommendations for breast, colorectal and lung cancer treatments: an opportunity to improve methodology and clinical relevance
Ann Oncol
(2011)
I. Chalmers et al.
Avoidable waste in the production and reporting of research evidence
Lancet
(2009)
M. Chung et al.
Two methods provide similar signals for the need to update systematic reviews
J Clin Epidemiol
(2012)
A. Tsertsvadze et al.
Updating comparative effectiveness reviews: current efforts in AHRQ's Effective Health Care Program
J Clin Epidemiol
(2011)
Institute of Medicine (IOM). Clinical practice guidelines we can trust: National Academy of Sciences. Available at...
GuíaSalud-Aragon Institute of Health Sciences (IACS). Updating clinical practice guidelines in the Spanish national...
H. Bastian et al.
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
PLoS Med
(2010)

J.S. Burgers et al.

Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs

Int J Qual Health Care

(2003)

P. Shekelle et al.

When should clinical guidelines be updated?

BMJ

(2001)

M. Eccles et al.

Updating evidence-based clinical guidelines

J Health Serv Res Policy

(2002)

R.M. Rosenfeld et al.

Clinical practice guideline development manual: a quality-driven approach for translating evidence into action

Otolaryngol Head Neck Surg

(2009)

P.F. White et al.

Perioperative analgesia: what do we still know?

Anesth Analg

(2009)

E. Neugebauer et al.

Wissenschaftsbetrug/Gefälschte Studien: Auswirkungen auf die S3-Leitlinie?

Dtsch Arztebl

(2009)

National Institute for Health and Clinical Excellence (NICE). The guidelines manual 2009. Available at...

The AGREE Next Steps Consortium. Appraisal of guidelines for research and evaluation II: AGREE II instrument 2009....

G. Gartlehner et al.

Assessing the need to update prevention guidelines: a comparison of two methods

Int J Qual Health Care

(2004)

G. Lyratzopoulos et al.

Updating clinical practice recommendations: is it worthwhile and when?

Int J Technol Assess Health Care

(2012)

M.E. Johnston et al.

Keeping cancer guidelines current: results of a comprehensive prospective literature monitoring strategy for twenty clinical practice guidelines

Int J Technol Assess Health Care

(2003)

Cited by (46)

Development of criteria for the prospective assessment of the need for updating guideline recommendations: The AGIL criteria[Formula presented]
2024, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen
Evidenzbasierte Leitlinien- und Impfempfehlungen sollten fortlaufend auf dem aktuellsten Stand sein, um eine angemessene Gesundheitsversorgung zu unterstützen Gleichzeitig sind die Ressourcen für die Aktualisierung von Leitlinien begrenzt.
Ziel des Projekts war es, eine Kriterienliste für die prospektive Einschätzung des Aktualisierungsbedarfs einzelner Leitlinien- oder Impfempfehlungen zu entwickeln, die zum Zeitpunkt der Fertigstellung einer Leitlinie oder Leitlinienaktualisierung anwendbar sind.
In diesem Artikel wird die methodische Entwicklung der sogenannten AGIL-Kriterien beschrieben (AGIL: AktualisierunGsprIorität von Leitlinienempfehlungen; Innovationsfonds-Fördernr.: 01VSF20019). Die AGIL-Kriterien wurden in fünf Schritten entwickelt: 1) Erarbeitung einer initialen Kriterienliste innerhalb des Projektteams; 2) Durchführung eines Online-Surveys zur Befragung von Leitlinienexpert*innen zur initialen Kriterienliste; 3) Überarbeitung der Kriterienliste auf der Basis der Ergebnisse des Online-Surveys; 4) Durchführung eines Workshops zur Kriterienliste im Rahmen des EbM-Kongresses 2023; 5) Erstellung von Version 1.0 der AGIL-Kriterien, basierend auf den Ergebnissen des Workshops.
Die initiale Liste umfasste die folgenden drei Kriterien: 1) Relevanz der Fragestellung, 2) Verfügbarkeit neuer relevanter Evidenz und 3) Einfluss potenzieller neuer Evidenz.
Die Rücklaufquote des Online-Surveys für vollständig ausgefüllte Fragebögen betrug 31,0 % (N = 195; 630 Leitlinienexpert*innen wurden per E-Mail kontaktiert). Für 90,3 % (n = 176) der Befragten enthielt die Kriterienliste alle für die Einschätzung des Aktualisierungsbedarfs von Leitlinienempfehlungen wesentlichen Aspekte. Mehr als drei Viertel der Befragten bewerteten die Wichtigkeit der drei Kriterien als „sehr wichtig“ oder „wichtig“ (Kriterium 1–3: 75,3 %, 86,1 %, 85,2 %) und – mit Ausnahme von Kriterium 1 – die Verständlichkeit als „sehr verständlich“ oder „verständlich“ (Kriterium 1–3: 58,4 %, 75,9 %, 78,5 %).
Die Ergebnisse des Online-Surveys und des Workshops bestätigten insgesamt die drei Kriterien und ihre jeweils zwei Unterfragen. Die Einarbeitung sämtlichen Feedbacks resultierte in den AGIL-Kriterien (Version 1.0), zusammengefasst: 1) Relevanz der Fragestellung bezüglich a) PICO-Komponenten und b) weiterer Faktoren, z. B. epidemiologische Aspekte; 2) Verfügbarkeit neuer Evidenz a) zum gesundheitsbezogenen Nutzen und Schaden und b) zu anderen Entscheidungsfaktoren, z. B. Umsetzbarkeit, Akzeptanz; 3) Einfluss neuer Evidenz a) auf die Vertrauenswürdigkeit der Evidenz, auf der die Empfehlung basiert, und b) auf die vorliegende Empfehlung, z. B. Empfehlungsstärke.
Die moderate Rücklaufquote des Online-Surveys könnte die Repräsentativität eingeschränkt haben. Dennoch bewerten wir die Rücklaufquote in diesem Forschungskontext als zufriedenstellend. Der Einbezug vieler Expert*innen im Rahmen des Online-Surveys und des EbM-Kongress-Workshops stellt eine Stärke des Projekts dar und unterstützt die Qualität der Ergebnisse.
Die AGIL-Kriterien bieten eine strukturierte Hilfestellung für die prospektive Einschätzung des Aktualisierungsbedarfs einzelner Leitlinienempfehlungen und anderer evidenzbasierter Empfehlungen. Die Implementierung und Evaluierung der AGIL-Kriterien 1.0 im Rahmen eines Praxistests ist in Planung.
Evidence-based guideline and vaccination recommendations should continuously be updated to appropriately support health care decisions. However, resources for updating guidelines are often limited.
The aim of this project was to develop a list of criteria for the prospective assessment of the need for updating individual guideline or vaccination recommendations, which can be applied from the time a guideline or guideline update is finalised.
In this article we describe the development of the AGIL criteria (Assessment of Guidelines for Updating Recommendations). The AGIL criteria were developed by experienced scientists and experts in the field of guideline development in a multi-step process. The five steps included: 1) development of an initial list of criteria by the project team; 2) online survey of guideline experts on the initial version of the criteria list; 3) revision of the criteria list based on the results of the online survey; 4) workshop on the criteria list at the EbM Congress 2023; 5) creation of version 1.0 of the AGIL criteria based on the workshop results.
The initial list included the following three criteria: 1) relevance of the question 2) availability of new relevant evidence, and 3) impact of potentially new evidence.
The response rate of the online survey for fully completed questionnaires was 31.0 % (N = 195; 630 guideline experts were contacted by email). For 90.3 % (n = 176) of the respondents, the criteria list included all essential aspects for assessing the need for updating guideline recommendations. More than three quarters of respondents rated the importance of the three criteria as “very important” or “important” (criteria 1–3: 75.3 %, 86.1 %, 85.2 %) and – with the exception of criterion 1 – comprehensibility as “very comprehensible” or “comprehensible” (criteria 1–3: 58.4 %, 75.9 %, 78.5 %).
The results of the online survey and the workshop generally confirmed the three criteria with their two sub-questions. The incorporation of all feedback resulted in the AGIL criteria (version 1.0), recapping: 1) relevance of the question regarding a) PICO components and b) other factors, e. g. epidemiological aspects; 2) availability of new evidence a) on health-related benefits and harms and b) on other decision factors, e. g. feasibility, acceptability; 3) impact of new evidence a) on the certainty of evidence on which the recommendation is based and b) on the present recommendation, e. g. strength of recommendation.
The moderate response rate of the online survey may have limited its representativeness. Nevertheless, we consider the response rate to be satisfactory in this research context. The inclusion of many experts in the online survey and the EbM Congress workshop is a strength of the project and supports the quality of the results.
The AGIL criteria provide a structured guidance for the prospective assessment of the need for updating individual guideline recommendations and other evidence-based recommendations. The implementation and evaluation of the AGIL criteria 1.0 in a field test is planned.
An adapted ‘Ottawa’ method allowed assessing the need to update topic areas within clinical practice guidelines
2022, Journal of Clinical Epidemiology
Citation Excerpt :
In addition, time and financial resources were insufficient for a full update. Therefore, we sought for a systematic, transparent, and efficient partial updating method [14]. Partial updating can be defined as the selective updating only of those recommendations or sections within a guideline for which the evidence base has undergone a major change [15].
To adapt and evaluate a method for assessing the need to update guideline topic areas involving multiple recommendations.
The ‘Ottawa method’ uses literature signals to determine changes in evidence that trigger a need to update individual guideline questions. We adapted the Ottawa method to include a process for aggregating updating signals by topic area (e.g., resuscitation) and tested this method using the German guideline on the treatment of patients with severe/multiple injuries. This involved a focused systematic evaluation of current evidence to identify updating signals and classifying the need to update for each topic area. Then, we surveyed the guideline group online about the modified method.
We conducted focused literature searches for 37 topic areas and screened a mean of 97 abstracts per topic area in 2021. The need to update was high for eight (21.6%), intermediate for eight (21.6%), and low for 21 topic areas (56.8%) based on updating signals. The survey response rate was 56% (24/43). Most guideline group members (94%, 16/17 responders) would use the Ottawa method again but their comments identified some weaknesses.
The modified Ottawa method is a suitable, efficient tool to generate evidence-based updating signals for guideline topic areas involving multiple recommendations. Further fine-tuning is recommended.
Methodological approaches for developing, reporting, and assessing evidence-based clinical practice guidelines: a systematic survey
2022, Journal of Clinical Epidemiology
Citation Excerpt :
The search retrieved 10,563 citations with 18 additional records identified through cross-reference search and experts’ enquiry; of which 427 were considered potentially eligible and were reviewed in full text. From these, 46 papers were included (Supplementary Figure A1) [3–6,10–51]. These papers reported 11 CPG development approaches for standard original (n = 12 papers) [10–21], three rapid original (n = 3 papers) [3,22,23], 17 adapted/adopted (n = 17 papers) [4,24–39], and two updated CPGs (n = 2 papers) [5,40] (see Table 1, Supplementary Table A3); seven CPG reporting checklist instruments (n = 7 papers, see Table 2, Supplementary Table A4) [6,41–46]; and six CPG assessment tools (n = 6 papers, see Table 3, Supplementary Table A5) [44,47–51].
To produce a mapping and feature summary of approaches and tools available for the clinical practice guideline (CPG) community to develop, report, or assess four types of CPGs: (1) Standard original (or de novo) CPGs, (2) Rapid original CPGs, (3) Adapted/adopted CPGs, and (4) Updated CPGs.
The systematic literature search was conducted using Embase and PubMed, covering the period from January 2010 to October 13, 2020. Two websites that collect and recommend approaches/tools to develop, report, or assess CPGs were also searched: Guidelines International Network and Equator Network. We screened the search results to include methodological papers that aimed to develop specific approaches/tools to develop, report, or assess any of the aforementioned four CPG types.
Among 10,581 citations, 46 papers reporting 46 approaches/tools were included. Of these 46 approaches/tools, 33 were about CPG development, seven were for CPG reporting, and six for CPG assessment. Among the 33 development approaches/tools, 26 did not state usability or validity information; but nine from 13 reporting or assessment approaches/tools did.
This study provides an overall summary of the currently available approaches/tools, which serves to improve users’ understanding to pave the way for informed choice and application.
Feasibility of national living guideline methods: The Australian Stroke Guidelines
2022, Journal of Clinical Epidemiology
Citation Excerpt :
The process of checking with internal and external content experts during the development remains an important step in the process which should further reduce the small risk of missing crucial studies. Several approaches to evidence surveillance for updating guidelines have been documented, including incorporating machine learning. [28–31] Given the substantial resources the surveillance step can consume, our experience, in common with these examples, shows that a pragmatic approach to the evidence surveillance is likely needed for sustainability.
Maintaining clinical guideline currency has been one challenge to traditional guideline development. This paper describes the methods used to maintain a large national guideline for stroke management.
The Australian Stroke Clinical Guidelines are developed to meet Australian National Health and Medical Research Council (NHMRC) standards. Monthly surveillance is conducted for new systematic reviews and randomised controlled studies. Included studies undergo data extraction followed by preparation of updated evidence-to-decision frameworks which are used to inform updates, or development of new recommendations. Small writing groups made up of clinical experts and those with lived experience review and agree on changes, which are finally reviewed by a multidisciplinary Guidelines Steering Group. Draft changes are developed and published using the online MAGICapp platform, with dissemination and promotion via traditional methods as well as social media.
Each month approximately 350 abstracts are considered, covering 96 clinical topics and taking on average 16 h to review. There have been four major guideline updates covering 34 new and updated recommendations.
It is feasible to use ‘living’ methods to maintain the Australian Clinical Guidelines for Stroke Management. Further work is now needed to understand the impact of living guidelines.
Increasing the efficiency of guideline production: a narrative review
2019, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen
Die Erstellung von qualitativ hochwertigen klinischen Leitlinien ist aufwendig und zeitintensiv. Neue Methoden können den Prozess der Leitlinienerstellung effizienter gestalten. Deren Bekanntheitsgrad ist stark ausbaufähig.
Selektive Literaturrecherche in den Datenbanken PubMed/MEDLINE und Embase.
Für die Organisation, Planung und Kommunikation können bereits einfache Tools wie Surveys oder TED-Systeme genutzt werden. Inhaltlich sollten die einzelnen Fragestellungen vorab mit geeigneten Methoden priorisiert werden. Bestehende oder sich in Bearbeitung befindende internationale Leitlinien und systematische Übersichtsarbeiten können sinnvoll für die eigene Leitlinie genutzt werden. Bei Aktualisierungen sollte vorab überprüft werden, ob wirklich alle Teile der Leitlinie aktualisiert werden müssen, hierzu kann der Aktualisierungsbedarf festgestellt werden. Größtes Potential liegt im Konzept der „living guidelines“, bei denen eine kontinuierliche Aktualisierung stattfindet, die sich an der tatsächlichen Studienlage orientiert anstatt an der klassischen Periodizität.
Neue Methoden zur effizienteren Leitlinienerstellung sind entwickelt und teilweise auch etabliert worden. In der Praxis sollten bei der Planung einer Leitlinie (oder deren Aktualisierung) die jeweiligen Vor- und Nachteile/Risiken gegeneinander abgewogen werden.
The development of high-quality clinical practice guidelines is laborious and time-consuming. New methods have become available to streamline this process. However, the awareness of these methods should be improved.
Selective literature search in PubMed/MEDLINE und Embase.
Simple tools such as surveys or voting systems can facilitate the organization, planning and communication. Adequate methods should be used to prioritize all potential questions that should be addressed in the guideline. Published or ongoing international guidelines and systematic reviews can be used meaningfully for the planned guideline. In the case of guideline updates, it should be determined whether all parts of the guideline require an update of the evidence. The need for an update should be investigated. The concept of living guidelines has the biggest potential to provide gains in efficiency. Living guidelines are continuously updated based on new evidence instead of being regularly updated at a predefined time.
New methods allowing for more efficient guideline production have been developed and, in part, already been introduced. Before starting with the production of a guideline (or its update), the potential advantages and disadvantages/risks of the corresponding methods should be balanced.
A systematic decision-making process on the need for updating clinical practice guidelines proved to be feasible in a pilot study
2018, Journal of Clinical Epidemiology
Citation Excerpt :
Depending on the medical field, CPGs differ in terms of the need for updating, and also within an individual CPG, there will be recommendations that need modification while others remain still valid [2,10]. Thus, partial updating is often more reasonable than updating the whole CPG [11,12]. Moreover, full updates require resources that are comparable with those required for initial development [13,14] and can waste resources that are not necessary [2,15].
The objective of this study was to test and evaluate a new decision-making process on the need for updating within the update of a German clinical practice guideline (CPG).
The pilot study comprised (1) limited searches in Pubmed to identify new potentially relevant evidence, (2) an online survey among the members of the CPG group to assess the need for update, and (3) a consensus conference for determination and prioritization of guideline sections with a high need for update. Subsequently, we conducted a second online survey to evaluate the procedure.
The searches resulted in 902 abstracts that were graded as new potentially relevant evidence. Twenty five of 39 members of the CPG group (64%) participated in the online survey. Seventy six percent of those took part in the second online survey. The evaluation study found on average a grade of support of the procedure regarding the determination of the need for update of 3.65 (standard deviation: 0.76) on a likert scale with 1 = “no support” to 5 = “very strong support.”
The conducted procedure presents a systematic approach for assessing whether and to what extent a CPG requires updating and enables setting priorities for which particular guideline section to update within a CPG.

View all citing articles on Scopus

: Preliminary results of the research presented in this article have been presented at the 14th World Congress on Pain, August 27–31, 2012, Milan, Italy and G-I-N Conference, August 22–25, 2012, Berlin, Germany.

: Funding: None.

: Competing interests: None.

View full text

Review ArticlePartial updating of clinical practice guidelines often makes more sense than full updating: a systematic review on methods and the development of an updating procedure

Abstract

Objectives

Study Design and Setting

Results

Conclusion

Introduction

Section snippets

Literature search

Results of the literature search

Agreements and disagreements between the manuals on updating key elements

Conclusion

Acknowledgments

Lancet

J Clin Epidemiol

Artif Intell Med

Ann Oncol

Lancet

J Clin Epidemiol

J Clin Epidemiol

Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?

PLoS Med

Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs

Int J Qual Health Care

When should clinical guidelines be updated?

BMJ

Updating evidence-based clinical guidelines

J Health Serv Res Policy

Clinical practice guideline development manual: a quality-driven approach for translating evidence into action

Otolaryngol Head Neck Surg

Perioperative analgesia: what do we still know?

Anesth Analg

Wissenschaftsbetrug/Gefälschte Studien: Auswirkungen auf die S3-Leitlinie?

Dtsch Arztebl

Assessing the need to update prevention guidelines: a comparison of two methods

Int J Qual Health Care

Updating clinical practice recommendations: is it worthwhile and when?

Int J Technol Assess Health Care

Keeping cancer guidelines current: results of a comprehensive prospective literature monitoring strategy for twenty clinical practice guidelines

Int J Technol Assess Health Care

Review Article
Partial updating of clinical practice guidelines often makes more sense than full updating: a systematic review on methods and the development of an updating procedure