Skip to main content

Main menu

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
    • Trousse média 2021
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN

User menu

Search

  • Advanced search
CMAJ
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN
CMAJ

Advanced Search

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
    • Trousse média 2021
  • Visit CMAJ on Facebook
  • Follow CMAJ on Twitter
  • Follow CMAJ on Pinterest
  • Follow CMAJ on Youtube
  • Follow CMAJ on Instagram
Research

Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development

Wojtek Wiercioch, Elie A. Akl, Nancy Santesso, Yuan Zhang, Rebecca L. Morgan, Juan José Yepes-Nuñez, Sérgio Kowalski, Tejan Baldeh, Reem A. Mustafa, Kaja-Triin Laisaar, Ulla Raid, Itziar Etxeandia-Ikobaltzeta, Alonso Carrasco-Labra, Matthew Ventresca, Ignacio Neumann, Maicon Falavigna, Romina Brignardello-Petersen, Gian Paolo Morgano, Jan Brożek, Meghan McConnell and Holger J. Schünemann; for the PANELVIEW Working Group
CMAJ October 05, 2020 192 (40) E1138-E1145; DOI: https://doi.org/10.1503/cmaj.200193
Wojtek Wiercioch
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Elie A. Akl
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Nancy Santesso
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yuan Zhang
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Rebecca L. Morgan
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Juan José Yepes-Nuñez
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sérgio Kowalski
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Tejan Baldeh
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Reem A. Mustafa
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kaja-Triin Laisaar
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ulla Raid
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Itziar Etxeandia-Ikobaltzeta
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alonso Carrasco-Labra
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Matthew Ventresca
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ignacio Neumann
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Maicon Falavigna
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Romina Brignardello-Petersen
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Gian Paolo Morgano
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jan Brożek
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Meghan McConnell
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Holger J. Schünemann
Department of Health Research Methods, Evidence, and Impact (Wiercioch, Akl, Santesso, Zhang, Morgan, Baldeh, Mustafa, Etxeandia-Ikobaltzeta, Carrasco-Labra, Ventresca, Brignardello-Petersen, Morgano, Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Internal Medicine (Akl), American University of Beirut, Beirut, Lebanon; School of Medicine (Yepes-Nuñez), Universidad de los Andes, Bogotá, Colombia; Department of Internal Medicine (Kowalski), Federal University of Paraná, Curitiba, Brazil; Division of Nephrology and Hypertension (Mustafa), Department of Medicine, University of Kansas Medical Center, Kansas City, Kan.; Guideline Development Group (Laisaar), Faculty of Medicine, Institute of Family Medicine and Public Health, University of Tartu, Tartu, Estonia; Ministry of Social Affairs (Raid), Tallinn, Estonia; Department of Internal Medicine (Neumann), Pontificia Universidad Católica de Chile, Santiago, Chile; Federal University of Rio Grande do Sul (Falavigna), Institute for Health Technology Assessment, Porto Alegre, Brazil; Department of Medicine (Brożek, Schünemann), McMaster University, Hamilton, Ont.; Department of Innovation in Medical Education (McConnell), University of Ottawa, Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Tables
  • Related Content
  • Responses
  • Metrics
  • PDF
Loading

Abstract

BACKGROUND: Guideline recommendations may be affected by flaws in the process, inappropriate panel member selection or conduct, conflicts of interest and other factors. To our knowledge, no validated tool exists to evaluate guideline development from the perspective of those directly involved in the process. Our objective was to develop and validate a universal tool, the PANELVIEW instrument, to assess guideline processes, methods and outcomes from the perspective of the participating guideline panellists and group members.

METHODS: We performed a systematic literature search and surveys of guideline groups (identified through contacting international organizations and convenience sampling of working panels) to inform item generation. Subsequent groups of guideline methodologists and panellists reviewed items for face validity and missing items. We used surveys, interviews and expert review for item reduction and phrasing. For reliability assessment and feedback, we tested the PANELVIEW tool in 8 international guideline groups.

RESULTS: We surveyed 62 members from 13 guideline panels, contacted 19 organizations and reviewed 20 source documents to generate items. Fifty-three additional key informants provided feedback about phrasing of the items and response options. We reduced the number of items from 95 to 34 across domains that included administration, training, conflict of interest, group dynamics, chairing, evidence synthesis, formulating recommendations and publication. The tool takes about 10 minutes to complete and showed acceptable measurement properties.

INTERPRETATION: The PANELVIEW instrument fills a gap by enabling guideline organizations to involve clinicians, patients and other participants in evaluating their guideline processes. The tool can inform quality improvement of existing or new guideline programs, focusing on insight into and transparency of the guideline development process, methods and outcomes.

As a product of a group process that involves project planning, synthesizing evidence and deliberation by guideline group members to reach consensus and formulate recommendations, health guidelines are highly influential in determining practice.1,2 Guideline development also requires careful coordination of multiple teams with specialized knowledge.1,3,4 These teams typically include an oversight committee responsible for project planning, working groups responsible for preparation and technical aspects of evidence synthesis, and a guideline panel tasked with prioritizing questions and formulating recommendations.

Guideline group processes may be prone to influence by people with strong opinions, imbalanced group member characteristics or unqualified members, or may not use the best available evidence.2,5–8 Currently available instruments assessing the trustworthiness of practice guidelines rely on what the guideline authors report, typically in peer-reviewed publications, or reports of organizations or manuals.9,10 However, what authors report may be generic or incomplete, lack transparency or be inconsistent with the assessment of all group members, and what is reported may not always reflect what happened.11 For example, the Appraisal of Guidelines, Research and Evaluation (AGREE)9 and Reporting Items for Practice Guidelines in Healthcare (RIGHT)10 tools appropriately call for conduct of systematic reviews as part of guideline development and appropriate disclosure of potential influence of conflicts. Although systematic reviews should inform guidelines, their conduct guarantees neither their quality nor that they are used appropriately by the panel for making recommendations. Likewise, having a conflict-of-interest declaration and management policy for guidelines does not necessarily guarantee that conflicts are well managed when guideline groups make recommendations.

Existing tools do not evaluate essential steps and processes, such as giving appropriate consideration to the evidence and ensuring that all panel members have an equal voice, as they take place.9,10,12 An internal evaluation by participating guideline group members would provide this valuable insight. In addition, ensuring that panel members view the process as appropriate and one that results in a credible guideline will help ensure they see value in their contribution. By obtaining an assessment from the participants, guideline developers could identify areas of their processes participants view as needing improvement, as well as dissenting views among participants. They could then use this information to modify their methods and approaches, and to ensure the credibility of their guidelines and the trustworthiness of the recommendations.

The objective of this research was to develop and validate a tool for assessing guideline panel members’ perception of the appropriateness of, and satisfaction with, the process, methods and outcome of the development of a health guideline.

Methods

For the development of the instrument, which we named PANELVIEW, we followed methods for scale development, including item generation based on existing literature, item reduction through key informant and expert feedback and consensus and field testing with guideline panels (Table 1).13 At each step, participating panel members and guideline methodologists drawn from organizations from diverse geographic areas who produced guidelines about different clinical topics, assessed information about the items to evaluate the appropriateness of guideline development.

View this table:
  • View inline
  • View popup
  • Download powerpoint
Table 1:

Overview of steps and participants in the PANELVIEW tool development

Item generation

Item generation began with discussion by 2 investigators (W.W., H.J.S.) of key domains for capturing the evaluation of guideline-related processes based on domains in the GIN-McMaster Guideline Development Checklist.1 We hypothesized that all parts of the process might be relevant for assessing appropriateness and satisfaction of panel members.

We then conducted a systematic literature search to identify steps and themes in guideline development that relate to the appropriateness of the process. We searched MEDLINE and Embase from inception to November 2018 to identify studies that discussed or evaluated steps of guideline development.1 We used controlled vocabulary and keywords to capture evaluation of the guideline development process and panel member perceptions (see Appendix 1, Supplemental Figure S1 and Supplemental Box S1, available at www.cmaj.ca/lookup/doi/10.1503/cmaj.200193/tab-related-content, for additional details).

To supplement our literature search, we contacted a convenience sample of 19 key informants, identified in a previous project,1 who represented major guideline-development organizations globally. We asked whether the organizations currently conducted internal evaluation of their guideline-development processes or used specific tools (Table 1, step 1). For each step involving key informants, we used a new sample of participants as a method of confirming data and views obtained in the preceding step, and to ensure broad representation of views and perspectives.

Panel surveys

We surveyed members of 13 guideline panels to obtain primary data (Table 1, step 1). After their panel meetings had adjourned, 62 panellists completed hard-copy surveys consisting of 6 open-ended questions inquiring about the factors that affected their satisfaction and perception of the appropriateness of the process (Appendix 1, Supplemental Table S1, Supplemental Figure S2). We included the survey responses as a source document for data abstraction.

Data abstraction

We developed and pilot tested a structured data-abstraction form. Study team members (Y.Z., R.L.M., K.-T.L., U.R., M.V., J.J.Y.-N., R.A.M., N.S., S.K., T.B.) reviewed full texts of source documents and abstracted independently and in duplicate items that related to the appropriateness of the methods or processes of guideline development, panel members’ views about methods or processes, or panel members’ satisfaction. Supporting quotations from the source document were included for each item, along with proposed themes for grouping of items (e.g., conflict-of-interest management, training, group interaction).

A subgroup of the study team members (W.W., Y.Z., R.L.M., S.K., J.J.Y.-N.) independently identified and merged duplicate items (i.e., items that measured or asked about the same aspect of the guideline-development process). The decisions were assessed by a second reviewer and discussed in a team meeting during which we finalized the deduplication, initial item phrasing and allocation to specific themes of the guideline-development process.

Item reduction

Feedback from key informants

We surveyed and conducted interviews with a convenience sample of 22 key informants, including guideline developers, methodologists and panel members, to obtain feedback about the initial list of items (Table 1, step 2; Appendix 1, Supplemental Table S2). Respondents were asked to rate on a 7-point Likert-type scale ranging from 1 (not important) to 7 (very important) how important they considered each of the items to be for evaluating the guideline-development process, to suggest modifications and to identify any missing items. We also sought to obtain in-depth feedback about the initial list and asked participants to comment on the level of detail, clarity and redundancy in the items. We then sent the list of items to participants for review in advance and conducted interviews in person at guideline panel meetings in presence of a note taker. We pilot tested both the survey and the interview guide.

Study team review and consensus meeting

Concurrently with the key informant surveys and interviews, we provided study team members (N.S., I.E.I., Y.Z., K.-T.L., S.K., R.B.-P., M.V., M.F., G.P.M.) with a structured feedback form to review the initial list of items and provide suggestions for modifications and theme categorization, or to suggest potentially missing items. For each item, we summarized the study team’s suggestions, and the key informants’ feedback and rating of importance. Study team members reviewed the summary and individually suggested to keep, modify, merge or delete items in preparation for a consensus meeting, during which the study team finalized decisions about each item based on discussion and group consensus. We then refined item wording based on our experience with developing other measurement instruments.14,15

Phrasing of items and response options

We conducted surveys with 26 additional key informants to determine the phrasing of items and response options (Table 1, step 3; Appendix 1, Supplemental Table S2). We asked respondents to indicate their preference for one of three 7-point Likert-type options regarding phrasing of responses and items, presenting 8 example items from the tool. The first option asked about appropriateness, the second about satisfaction, and the third, representing the original Likert scale, about agreement with the topic presented by the item.

Testing with panels

After item reduction, we pilot tested the PANELVIEW tool with 1 guideline panel consisting of 12 members. Subsequently, we made minor revisions to clarify item wording and the order of items in the tool and then used the tool with an additional 8 guideline panels consisting of 94 panellists (Table 1, step 4; Appendix 1, Supplemental Table S3). Panel members completed the PANELVIEW survey individually, expressing their agreement with each survey item on a 7-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree) (e.g., “There was appropriate management of potential bias in panel members’ interpretation of evidence and alignment with prior beliefs”). Panel members also provided feedback on the clarity of the instructions, clarity of items and the survey length.

Analysis

We used generalizability theory (G theory), a method for evaluating the reliability of measurements, to assess the reliability of scores obtained across the different panel groups.13 We calculated the item mean scores, standard deviations (SDs) and ranges across individual panellists. For individual panellists, we calculated overall scores as the mean of their item ratings. We conducted the preliminary reliability analyses at the individual panellist level as well as at the panel level, whereby we obtained item means by collapsing across individual panellists. We estimated multiple sources of variance (G), including the respondent, panel, item and domain, using a nested G theory study.13 The guideline-development process of different panel groups served as the object of measurement, individual respondents were nested within panels, and individual items were nested within the PANELVIEW survey domains (Appendix 1, Supplemental Table S5, Supplemental Figure S5).

Ethics approval

The Hamilton Integrated Research Ethics Board approved this study before data collection.

Results

Our systematic literature search, contact with key informants and surveys of 13 guideline panels yielded 17 published articles,6,16–31 3 additional source documents (Appendix 1, Supplemental Figure S3) and 62 survey responses. We abstracted a list of 694 items, which, after evaluation and deduplication, resulted in 95 items grouped across 17 themes covering guideline development (Appendix 1, Supplemental Table S4).

Informed by the rating of importance of the 95 items and feedback from key informants, we removed 23 items that scored low on importance as part of our consensus process. We merged 38 items with other items considered redundant. We phrased each item to ensure it assessed only 1 component of the guideline-development process. The final list included 34 items. Of the 26 key informants surveyed about the phrasing of response options, 19 (73%) indicated preference for using the Likert scale.

Generalizability and reliability

The analysis of variance from the nested G study showed an overall test reliability coefficient of 0.35 (Appendix 1, Supplemental Table S5). This result is likely an effect of enrolling homogeneous panels with regard to processes and methods. The tool’s domains and individual items within the domains each accounted for 4% of the variance, which also suggests that the processes for the guideline efforts we evaluated were similar across the domains and items. The guideline panels accounted for 28% of the variance, and participants within panels accounted for 55% of the variance in scores. This indicates that variation was captured in panellists’ assessments between the guideline panels and in panellists’ ratings within the panels. Despite the homogeneity of the groups, the tool was able to identify varying views of guideline panel members indicating higher and lower satisfaction or perception of appropriateness.

Response variation, item–item correlation and internal consistency

Within the panels, mean scores for items ranged from 4.0 to 7.0, item–item correlation values ranged from –0.76 to 0.96, and item–total correlation values ranged from −0.17 to 0.89. Across the 8 panels, the mean scores for items ranged from 5.5 to 6.8 (Table 2). There was high internal consistency in rating of satisfaction and appropriateness of the process within the 8 panels, with Cronbach’s α ranging from 0.85 to 0.98 (Table 3). For individual panellists, item responses ranged from 1 to 7, and item–item correlation values ranged from 0.003 to 0.719. This suggests, on an individual respondent level, that the tool distinguished between responses and that there was no end-of-scale aversion. Item–total correlation values by individual raters ranged from 0.40 to 0.80, which suggests that the items were measuring different aspects of the guideline process.

View this table:
  • View inline
  • View popup
Table 2:

Mean scores across panels for the PANELVIEW tool

View this table:
  • View inline
  • View popup
  • Download powerpoint
Table 3:

PANELVIEW tool mean scores and internal consistency across guideline panels

Feedback from guideline panel group members

Respondents reported that they did not have difficulty completing the questionnaire (mean rating on Likert scale 6.4 [SD 0.6]). Respondents, on average, felt that the questionnaire was neither too long nor too short (mean rating 3.5 [SD 1.7], with a rating < 4 suggesting that the questionnaire was not too long). For 68 respondents who completed the survey online, the average completion time was 12 (SD 7) minutes, and the median time was 10 minutes (we excluded 12 respondents with a recorded completion time of ≥ 30 min, who presumably took a break while completing the questionnaire). In response to a suggestion from 8 respondents, we added an option to respond to relevant items as “not applicable” (e.g., to allow panel chairs to skip items that request evaluation of their chairing of the panel). The final PANELVIEW tool is available at https://heigrade.mcmaster.ca/guideline-development/panelview.

Interpretation

We developed a tool, the PANELVIEW instrument, that allows guideline developers to assess their processes, methods and outcomes by directly involving clinicians, patients and any other guideline group member in the evaluation.

Existing instruments for assessing guideline credibility rely on the guideline authors’ report, which may describe the process as planned but not as implemented or as viewed by all group members, and may not reflect all relevant nuances of the process that affect the trustworthiness of recommendations. The PANELVIEW tool focuses on these important nuances and on the transparency of the guideline-development process, allowing organizations responsible for guideline development to inform their quality-improvement efforts.

We followed best practice for instrument development, including reviewing the literature, contacting key informants at guideline organizations and surveying panellists about key factors affecting guideline development. We tested the tool successfully with panels from international guideline organizations.

The PANELVIEW instrument is designed to identify strengths and weaknesses of a guideline-development group’s process and methods in a structured manner, and highlight specific areas for improvement as identified by the participants by assessing ratings within individual domains. The tool enables evaluation of guideline development by participating group members in its entirety or in phases. How the guideline process is organized may differ between organizations, for example, between those that convene 1 final panel meeting and those that maintain a standing panel with repeated meetings. This will determine whether developers administer the PANELVIEW tool once at the conclusion of a guideline project or throughout the process as the steps take place.

The PANELVIEW instrument is not intended to replace existing tools that offer guidance on the appropriate steps for guideline development or assess the credibility of published guideline reports. It offers an approach for identification of issues in the guideline-development process and methods by those who participate in it or directly observe it, such as technical experts and methodologists. The tool can serve to inform evaluation or quality improvement of new or existing guideline programs, respectively.

The rigour of development with the end-user in mind is the main strength of our work. First, we applied item-generation methods drawing on multiple sources: literature, contacting organizations, panel surveys and a team with extensive experience in the guideline field. Second, we involved other key informants from multiple organizations and participation on panels for input on items and face validity, allowing data saturation. Third, we field tested the tool with groups that focused on a variety of guideline topics.

We plan to administer the PANELVIEW tool with additional, diverse panels from various guideline organizations. Guideline organizations can access the tool at https://heigrade.mcmaster.ca/guideline-development/panelview to participate. We will seek further feedback on use of the tool, for example, about the potential for public reporting of PANELVIEW assessments to increase transparency. The high Cronbach α coefficients may indicate the presence of redundant items. Sampling of more panels will allow us to assess whether any refinement of tool items is necessary and conduct factor analysis for further evaluation of tool domains. Additional opportunities include comparative studies, for example, comparing PANELVIEW assessments of panellists to those of other group members (e.g., nonvoting observers), as well as evaluating global ratings and judgments of panel success and guideline credibility against ratings of the tool.

Limitations

A potential limitation of our research is that we did not conduct systematic searches of the nonmedical literature in the areas of business, education and policy-making for relevant items. At each step involving key informants, we used convenience sampling, which may introduce sampling bias. To address this, we drew on a broad representation of working guideline panellists, with varying levels of experience, as well as guideline-development experts from organizations representing a wide range of processes and methods.

The 8 guideline groups involved in field testing the PANELVIEW tool were recruited through key informants, and, for some aspects of development, the groups used similar methods (e.g., using the GRADE approach for assessing quality of evidence and strength of recommendations) and involved experienced group chairs. The high scores on many items and the lower overall reliability coefficient of 0.35 to discriminate between groups indicated that the groups were likely all high performing. Despite this, we observed variability in scores within the groups, which would allow guideline developers to identify whether individual panellists viewed the process and specific aspects of the process as more or less appropriate.

Conclusion

The PANELVIEW instrument allows capturing of panellists’ perspectives when they participate in guideline development, which addresses a gap in the field. Given the importance of guidelines and their impact on recipients and providers of care, optimizing the quality of their development is a logical step.

Acknowledgements

The authors acknowledge Dr. John J. Riva for helping to pilot test the item-reduction survey and Dr. Gordon Guyatt for helping to field test the PANELVIEW tool. They also acknowledge all the participating guideline panel members who provided feedback for item generation and reduction, and helped with field testing of the tool.

Footnotes

  • ↵* PANELVIEW Working Group members: Yngve Falck-Ytter MD, Division of Gastroenterology, Case Western Reserve University Cleveland, Ohio.; Iván D. Florez MD PhD, Department of Pediatrics, Universidad de Antioquia, Medellin, Colombia; Amir Qaseem MD PhD, American College of Physicians, Philadelphia, Penn.; Richard M. Rosenfeld MD MPH, Department of Otolaryngology, SUNY Downstate Medical Center, Brooklyn, NY; Craig W. Robbins MD MPH, Center for Clinical Information Services, Kaiser Permanente Care Management Institute, Oakland, Calif.; Judith Thornton PhD, Centre for Clinical Practice, National Institute for Health and Clinical Excellence, Manchester, UK

  • Competing interests: Authors of this manuscript have been involved in the development of various guideline manuals that are referenced in this article. Wojtek Wiercioch, Elie Akl and Holger Schünemann will retain copyright of the PANELVIEW tool through McMaster University (Technology ID 20-031), and the tool may be licensed in the future. Maicon Falavigna received consulting fees from Novartis, AbbVie, Roche, Boehringer Ingelheim, PTC Therapeutics and Sanofi Genzyme, unrelated to the submitted work. No other competing interests were declared.

  • This article has been peer reviewed.

  • Contributors: Holger Schünemann was the principal investigator. Holger Schünemann, Elie Akl and Wojtek Wiercioch conceptualized and designed the study with input from Nancy Santesso and Meghan McConnell. Meghan McConnell and Wojtek Wiercioch analyzed the data. Wojtek Wiercioch and Holger Schünemann drafted the manuscript. All of the authors contributed to the acquisition and interpretation of data, revised the manuscript critically for important intellectual content, approved the final version submitted for publication and agreed to be accountable for all aspects of the work.

  • Funding: This study was supported by internal funds from the McMaster GRADE centre, McMaster University. Kaja-Triin Laisaar receives institutional research support through grant IUT34-17 from the Ministry of Education and Research through the Estonian Research Council.

  • Data sharing: Data not shared.

  • Accepted June 10, 2020.

References

  1. ↵
    1. Schünemann HJ,
    2. Wiercioch W,
    3. Etxeandia I,
    4. et al
    . Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ 2014;186:E123–42.
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. Fretheim A,
    2. Schunemann H,
    3. Oxman A
    . Improving the use of research evidence in guideline development: 5. Group processes. Health Res Policy Syst 2006;4:17.
    OpenUrlCrossRefPubMed
  3. ↵
    Institute of Medicine Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. In: Graham R, Mancher M, Miller Wolman D, et al., editors. Clinical practice guidelines we can trust. Washington: National Academies Press; 2011.
  4. ↵
    1. Qaseem A,
    2. Forland F,
    3. Macbeth F,
    4. et al
    . Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med 2012;156:525–31.
    OpenUrlCrossRefPubMed
  5. ↵
    1. Schünemann HJ,
    2. Woodhead M,
    3. Anzueto A,
    4. et al
    . A guide to guidelines for professional societies and other developers of recommendations: introduction to integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc 2012;9:215–8.
    OpenUrlCrossRefPubMed
  6. ↵
    1. Neumann I,
    2. Karl R,
    3. Rajpal A,
    4. et al
    . Experiences with a novel policy for managing conflicts of interest of guideline developers: a descriptive qualitative study. Chest 2013;144:398–404.
    OpenUrl
    1. Boyd EA,
    2. Bero L
    . Improving the use of research evidence in guideline development: 4. Managing conflicts of interests. Health Res Policy Syst 2006;4:16.
    OpenUrlCrossRefPubMed
  7. ↵
    1. Fretheim A,
    2. Schunemann H,
    3. Oxman A
    . Improving the use of research evidence in guideline development: 3. Group composition and consultation process. Health Res Policy Syst 2006;4:15.
    OpenUrlCrossRefPubMed
  8. ↵
    1. Brouwers MC,
    2. Kho ME,
    3. Browman GP,
    4. et al
    . AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ 2010;182:E839–42.
    OpenUrlFREE Full Text
  9. ↵
    1. Chen Y,
    2. Yang K,
    3. Marušić A,
    4. et al
    . A reporting tool for practice guidelines in health care: the RIGHT statement. Ann Intern Med 2017;166:128–32.
    OpenUrlCrossRefPubMed
  10. ↵
    1. Alexander PE,
    2. Li SA,
    3. Gionfriddo MR,
    4. et al
    . Senior GRADE methodologists encounter challenges as part of WHO guideline development panels: an inductive content analysis. J Clin Epidemiol 2016;70:123–8.
    OpenUrl
  11. ↵
    1. Shiffman RN,
    2. Shekelle P,
    3. Overhage JM,
    4. et al
    . Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003;139:493–8.
    OpenUrlCrossRefPubMed
  12. ↵
    1. Streiner DL,
    2. Norman GR,
    3. Cairney J
    . Health measurement scales: a practical guide to their development and use. 5th ed. Oxford (UK): Oxford University Press; 2015.
  13. ↵
    1. Puhan MA,
    2. Behnke M,
    3. Laschke M,
    4. et al
    . Self-administration and standardisation of the chronic respiratory questionnaire: a randomised trial in three German-speaking countries. Respir Med 2004;98:342–50.
    OpenUrlCrossRefPubMed
  14. ↵
    1. Schünemann HJ,
    2. Goldstein R,
    3. Mador MJ,
    4. et al
    . A randomised trial to evaluate the self-administered standardised chronic respiratory questionnaire. Eur Respir J 2005;25:31–40.
    OpenUrlAbstract/FREE Full Text
  15. ↵
    1. Atkins L,
    2. Smith JA,
    3. Kelly MP,
    4. et al
    . The process of developing evidence-based guidance in medicine and public health: a qualitative study of views from the inside. Implement Sci 2013;8:101.
    OpenUrlCrossRefPubMed
    1. Calderón C,
    2. Rotaeche R,
    3. Etxebarria A,
    4. et al
    . Gaining insight into the clinical practice guideline development processes: qualitative study in a workshop to implement the GRADE proposal in Spain. BMC Health Serv Res 2006;6:138.
    OpenUrlCrossRefPubMed
    1. Chang LW,
    2. Kennedy CE,
    3. Kennedy GE,
    4. et al
    . Developing WHO guidelines with pragmatic, structured, evidence-based processes: a case study. Glob Public Health 2010;5:395–412.
    OpenUrlPubMed
    1. Gardner B,
    2. Davidson R,
    3. McAteer J,
    4. et al
    . A method for studying decision-making by guideline development groups. Implement Sci 2009;4:48.
    OpenUrlCrossRefPubMed
    1. Harding E,
    2. Brown D,
    3. Hayward M,
    4. et al
    . Service user perceptions of involvement in developing NICE mental health guidelines: a grounded theory study. J Ment Health 2010;19:249–57.
    OpenUrlPubMed
    1. Hopthrow T,
    2. Feder G,
    3. Michie S
    . The role of group decision making processes in the creation of clinical guidelines. Int Rev Psychiatry 2011;23:358–64.
    OpenUrlPubMed
    1. Hutchings A,
    2. Raine R
    . A systematic review of factors affecting the judgments produced by formal consensus development methods in health care. J Health Serv Res Policy 2006;11:172–9.
    OpenUrlCrossRefPubMed
    1. Hutchings A,
    2. Raine R,
    3. Sanderson C,
    4. et al
    . An experimental study of determinants of the extent of disagreement within clinical guideline development groups. Qual Saf Health Care 2005;14:240–5.
    OpenUrlAbstract/FREE Full Text
    1. Khodyakov D,
    2. Hempel S,
    3. Rubenstein L,
    4. et al
    . Conducting online expert panels: a feasibility and experimental replicability study. BMC Med Res Methodol 2011; 11:174.
    OpenUrlCrossRefPubMed
    1. Moreira T,
    2. May C,
    3. Mason J,
    4. et al
    . A new method of analysis enabled a better understanding of clinical practice guideline development processes. J Clin Epidemiol 2006;59:1199–206.
    OpenUrlCrossRefPubMed
    1. O’Brien KK,
    2. Wilkins A,
    3. Zack E,
    4. et al
    . Developing clinical practice guidelines in HIV rehabilitation: process recommendations and guiding principles. AIDS Educ Prev 2011;23:457–68.
    OpenUrlPubMed
    1. Pagliari C,
    2. Grimshaw J
    . Impact of group structure and process on multidisciplinary evidence-based guideline development: an observational study. J Eval Clin Pract 2002;8:145–53.
    OpenUrlPubMed
    1. Pagliari C,
    2. Grimshaw J,
    3. Eccles M
    . The potential influence of small group processes on guideline development. J Eval Clin Pract 2001;7:165–73.
    OpenUrlCrossRefPubMed
    1. Raine R,
    2. Sanderson C,
    3. Black N
    . Developing clinical guidelines: a challenge to current methods. BMJ 2005;331:631–3.
    OpenUrlFREE Full Text
    1. Raine R,
    2. Sanderson C,
    3. Hutchings A,
    4. et al
    . An experimental study of determinants of group judgments in clinical guideline development. Lancet 2004;364: 429–37.
    OpenUrlCrossRefPubMed
  16. ↵
    1. Williams W,
    2. Lowery NE,
    3. Lyalin D,
    4. et al
    . Development and utilization of best practice operational guidelines for immunization information systems. J Public Health Manag Pract 2011;17:449–56.
    OpenUrlCrossRefPubMed
PreviousNext
Back to top

In this issue

Canadian Medical Association Journal: 192 (40)
CMAJ
Vol. 192, Issue 40
5 Oct 2020
  • Table of Contents
  • Index by author

Article tools

Print
Download PDF
Article Alerts
To sign up for email alerts or to access your current email alerts, enter your email address below:
Email Article

Thank you for your interest in spreading the word on CMAJ.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development
(Your Name) has sent you a message from CMAJ
(Your Name) thought you would like to see the CMAJ web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development
Wojtek Wiercioch, Elie A. Akl, Nancy Santesso, Yuan Zhang, Rebecca L. Morgan, Juan José Yepes-Nuñez, Sérgio Kowalski, Tejan Baldeh, Reem A. Mustafa, Kaja-Triin Laisaar, Ulla Raid, Itziar Etxeandia-Ikobaltzeta, Alonso Carrasco-Labra, Matthew Ventresca, Ignacio Neumann, Maicon Falavigna, Romina Brignardello-Petersen, Gian Paolo Morgano, Jan Brożek, Meghan McConnell, Holger J. Schünemann
CMAJ Oct 2020, 192 (40) E1138-E1145; DOI: 10.1503/cmaj.200193

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
‍ Request Permissions
Share
Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development
Wojtek Wiercioch, Elie A. Akl, Nancy Santesso, Yuan Zhang, Rebecca L. Morgan, Juan José Yepes-Nuñez, Sérgio Kowalski, Tejan Baldeh, Reem A. Mustafa, Kaja-Triin Laisaar, Ulla Raid, Itziar Etxeandia-Ikobaltzeta, Alonso Carrasco-Labra, Matthew Ventresca, Ignacio Neumann, Maicon Falavigna, Romina Brignardello-Petersen, Gian Paolo Morgano, Jan Brożek, Meghan McConnell, Holger J. Schünemann
CMAJ Oct 2020, 192 (40) E1138-E1145; DOI: 10.1503/cmaj.200193
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • Methods
    • Results
    • Interpretation
    • Acknowledgements
    • Footnotes
    • References
  • Figures & Tables
  • Related Content
  • Responses
  • Metrics
  • PDF

Related Articles

  • PubMed
  • Google Scholar

Cited By...

  • Managing conflicts of interest in the development of health guidelines
  • Google Scholar

More in this TOC Section

  • Rates of rebleeding, thrombosis and mortality associated with resumption of anticoagulant therapy after anticoagulant-related bleeding
  • Trends in prevalence of chronic disease and multimorbidity in Ontario, Canada
  • COVID-19 in patients undergoing long-term dialysis in Ontario
Show more Research

Similar Articles

Content

  • Current issue
  • Past issues
  • Collections
  • Sections
  • Blog
  • Podcasts
  • Alerts
  • RSS
  • Early releases

Information for

  • Advertisers
  • Authors
  • Reviewers
  • CMA Members
  • Media
  • Reprint requests
  • Subscribers

About

  • General Information
  • Journal staff
  • Editorial Board
  • Governance Council
  • Journal Oversight
  • Careers
  • Contact
  • Copyright and Permissions

Copyright 2021, Joule Inc. or its licensors. All rights reserved. ISSN 1488-2329 (e) 0820-3946 (p)

All editorial matter in CMAJ represents the opinions of the authors and not necessarily those of the Canadian Medical Association or its subsidiaries.

To receive any of the resources on this site in an accessible format, please contact us at cmajgroup@cmaj.ca.

Powered by HighWire