On bias and transparency in the development of influential recommendations

Clinical practice guidelines and reimbursement recommendations such as those of the Canadian Diabetes Association (CDA) and the Canadian Expert Drug Advisory Committee (CEDAC) about insulin glargine1^,2 have a potentially great effect on clinical practice. Minimizing bias during their development is therefore at least as important as it is during clinical trials. A conflict-of-interest guideline is only one method of minimizing bias. In this article I will review the potential for bias that exists in the Common Drug Review (CDR) process (Box 1) and contrast this with my understanding of the development of the CDA guidelines.

• Download figure
• Open in new tab
• Download powerpoint

Box 1.

The CDR includes conflict-of-interest guidelines.3 Annually, all CEDAC members complete a detailed form describing all interactions with any party that might be considered conflicts of interest during the past 5 years. These are shared with the CEDAC chair; at each meeting, it is decided whether a potential conflict exists that should exclude a member from participating in the review of a drug. No CEDAC member owns personal stock in any drug company, and none has received pharmaceutical company support of any kind related to a drug being considered by CEDAC.

The many potential sources of bias in the clinical literature almost always result in overestimation of benefit and underestimation of harm. These include preferentially reporting positive studies,4 more frequently reporting the outcomes measured in a trial that improve with therapy than those that do not,5 changing the prespecified primary outcome after a trial is analyzed5 and paying less attention to harm than to efficacy.6 A voluntary registry of clinical trials is a first step toward decreasing publication bias,7 but only mandatory registration of all trials before approval by any research ethics board will eliminate this bias. Other sources of bias will be avoided only when the protocols of clinical trials are registered at the time the studies begin, when all the results of all clinical trials are made public at the time a drug is licensed,8 and when the side effects of a drug are more rigorously assessed in clinical trials. Progress in these areas has been painfully slow, but I am delighted that the Pharmaceutical Research and Manufacturers of America has endorsed the routine submission of trial protocols.9 I will be even more delighted when it happens, and all protocols become public.

The CDR uses standard methodology to perform a detailed independent systematic review of the available literature about the benefits, harms and cost-effectiveness of each drug under consideration for reimbursement by publicly funded drug-benefit plans.10 These reviews, which attempt to summarize the literature in as unbiased a manner as possible, are shared with the manufacturer for comment before the CEDAC meeting. They are not made public, however, in part because they often contain summaries of information from manufacturers that is confidential. CDA guidelines describe a method of grading their recommendations, but they provide little detail about the methodology used to review the literature and do not provide a critical appraisal of the various randomized trials of insulin glargine.1

Interpreting the literature is not the same as summarizing it: interpretation inevitably incorporates an individual's values and perspectives. CEDAC's mandate is to make reimbursement recommendations from the perspective of the health care system, based not only upon a drug's effectiveness but also its cost-effectiveness.10 Its 11 members are physicians, pharmacists and nurses, chosen to represent a variety of clinical backgrounds, all with the ability to critically and sensibly interpret clinical findings and cost-effectiveness data. Moreover, each CDR review involves at least one external reviewer who is a content expert. In the case of insulin glargine, the clinical expert was an endocrinologist who had declared receiving honoraria from the manufacturer of insulin glargine that totaled less than $4000.

The CDA has complained that because no member of CEDAC is an endocrinologist, the committee underestimated the impact of insulin glargine upon patient care.11^,12 I would argue that the CEDAC reviewers, both internal and external, met the most important criterion needed to fulfill their mandate: to be able to review the available literature competently and objectively. In addition, the committee's clinicians had sufficient clinical experience with diabetes to assess the benefits of insulin glargine. Indeed, CEDAC's documented reasons for recommendation acknowledged that insulin glargine decreased the frequency of hypoglycemia in several clinical trials. The reason CEDAC recommended against reimbursement was that the relatively small improvement in hypoglycemia was not felt to justify the drug's more than 3-fold price relative to NPH (neutral protamine Hagedorn) insulin.2

Finally, it is important to consider whether the structure within which experts work could lead to bias or the perception of bias. CEDAC members are appointed by federal, provincial or territorial deputy ministers of health and are paid an honorarium by the CDR; public-drug-plan managers are allowed to observe CEDAC meetings; and CEDAC has no formal interaction with members of the public. In addition, CEDAC reports to the Board of the Canadian Coordinating Office of Health Technology Assessment (CCOHTA), which is made up entirely of representatives of the federal/provincial/ territorial Ministries of Health. No wonder a recent external assessment of the CDR found that members of advocacy groups representing people with various diseases called for greater public involvement in the CDR process.13 On the other hand, the CDA guidelines were sponsored by pharmaceutical and diagnostic companies, the methods of reviewing and summarizing the literature were not fully described, and the potential conflicts of interest of authors are unknown. Neither the guidelines nor the CDA letter of protest to the Canadian Ministers of Health about CEDAC's recommendation1^,12 acknowledged the 3-fold price differential associated with insulin glargine compared with NPH insulin. No wonder those who pay for drugs are concerned that groups such as the CDA sometimes preferentially emphasize the evidence that supports their position and minimize the evidence that does not.

Although I believe that the CDR process of reviewing and interpreting the available literature is as unbiased as possible, transparency would be enhanced if the CDR's reviews were made public. Most of the concerns about reporting biases that have been highlighted previously are beyond CEDAC's ability to correct. In terms of transparency, CEDAC's conflict-of-interest guidelines are publicly available.3 The CDR Web site will soon be modified to indicate that CEDAC members are paid for their work, and it will include a summary of each CEDAC member's declared potential conflicts of interest. Finally, CCOHTA is considering methods of greater public involvement in the CDR process; this cannot occur soon enough.

@ See related articles pages 333 and 337