Comparing estimates of cost effectiveness submitted to the National Institute for Clinical Excellence (NICE) by different organisations: retrospective study

Introduction

Evidence suggests that profit making organisations are more likely to report favourable results and conclusions from clinical studies than non-profit organisations. The most recent of these studies concluded that for profit organisations were 5.3 (95% confidence interval 2.0 to 14.4) times more likely to recommend the use of an experimental drug compared with non-profit groups.1 Only three studies, however, have formally assessed this association with regard to the results from economic evaluations. One showed that evaluations of oncology products sponsored by drug companies were less likely to report unfavourable qualitative conclusions (5% v 38%; P = 0.04) compared with studies sponsored by non-profit organisations.2 The two other studies also reported similar findings.3^–4

The technology appraisals programme of the National Institute for Clinical Excellence (NICE) provides guidance to the NHS in England and Wales on the use of new and existing health technologies. Manufacturers (or relevant UK agent) of the relevant technology, professional groups, and national groups representing patients submit evidence. Collectively, these three groups are known as consultees. An academic centre commissioned through the NHS's health technology assessment programme, called an assessment group, also assesses the evidence. Guidance to the NHS is formulated with regard to this evidence, which includes information on cost effectiveness.

The criterion traditionally used to assess cost effectiveness is the magnitude of a statistic known as the “incremental cost effectiveness ratio,” defined as the difference in costs between two technologies divided by the difference in their benefits.5 The lower the incremental cost effectiveness ratio, the more cost effective a technology.

We evaluated the association between source of funding and the results from economic evaluations submitted to NICE's technology appraisals programme.

Methods

Detailed information regarding the technology appraisals programme is available elsewhere (http://www.nice.org.uk/). Briefly, after receiving the final scope of the appraisal (the scope document outlines the patients, interventions, comparator technology, and outcomes of interest) from NICE, consultees have about 15 weeks to submit relevant clinical and economic evidence. During, and for a short period after, this time, the assessment group also undertakes its independent assessment of the clinical and economic evidence.

The appraisals committee meets twice to discuss the evidence. After the first meeting, the committee's preliminary recommendations are outlined in a document called the appraisal consultation document. At the second meeting, consultees' comments on this document are discussed and a final appraisal determination is produced. If consultees do not appeal against it, this document becomes the basis of NICE guidance.

Each manufacturer and the assessment group typically submit an economic evaluation so that the resulting incremental cost effectiveness ratio estimates can be compared. The institute has published its preferred rules regarding economic methods,6 but they are not absolute.

When possible, we extracted estimated incremental cost effectiveness ratios for the first 62 technology appraisals, issued between March 2000 and May 2003. To do this, we needed to address several issues. Firstly, although 62 appraisal reports had been issued, some contained guidance on more than one aspect of a technology. Each of these recommendations was therefore treated as a separate observation for which cost effectiveness estimates could exist. Secondly, we included pairings of estimates in the final dataset only if the manufacturer and the assessment group had each submitted at least one incremental cost effectiveness ratio for a technology using the same health outcome measure—for example, a manufacturer's estimate of the incremental cost per quality adjusted life year (QALY) for drug A was matched to the incremental cost per QALY for the same drug as estimated by the assessment group. However, if the assessment group had expressed its incremental cost effectiveness ratio as an incremental cost per life year saved, this pairwise comparison was excluded from the analysis. We assumed that results from cost minimisation analyses indicated clinical equivalence, together with an increase or decrease in cost and included them in the analysis accordingly.

Over the course of an appraisal, but before the final appraisal determination, manufacturers and assessment groups may revise their original incremental cost effectiveness ratio estimates. However, we restricted our analysis to estimates contained in first submissions.

We abstracted estimates from published guidance, assessment reports, and manufacturers' submissions, and these were double checked by a second reviewer. In the small number of instances where the two reviewers' estimates did not match, we sought opinion from the relevant technology analyst at NICE.

Results

Of the 62 appraisals, we excluded 35 because the assessment group alone did not produce an incremental cost effectiveness ratio (n = 20), there was no manufacturer or the manufacturer did not produce an estimate (n = 3), the assessment group and the manufacturer did not produce an estimate (n = 8), or different measures of health benefit were used (n = 4). The 27 remaining appraisals contained 54 pairwise comparisons.

Over 80% of the manufacturers' incremental cost effectiveness ratios were between dominant (meaning that the appraised technology was considered to be less costly and more effective than the comparator technology) and £15 000 per unit of outcome (categories 0 and 1 combined) whereas the assessment groups' estimates were more uniformly distributed over the five categories (fig 1). Only four (7%) estimates from manufacturers were above £30 000 per unit of outcome compared with 19 (35%) estimates from the assessment groups.

Fig 1

Distribution of estimates of incremental cost effectiveness ratios from assessment groups and manufacturers according to category of cost effectiveness (see table)

• Download figure
• Open in new tab
• Download powerpoint

There were 25 cases of negative rank (where the manufacturers' estimates were lower than those of the corresponding assessment group), 29 ties (where the estimates were in the same category), and no positive ranks (P < 0.01; see box for example). Sensitivity analysis around the category definitions did not alter the significance of this result. Figure 2 plots the log of these pairwise comparisons; the 45° line indicates identical incremental cost effectiveness ratios. Points above this line indicate that a manufacturer's estimate was higher (less favourable) than the estimate produced by the assessment group, whereas points below this line indicate the manufacturer submitted a lower (more favourable) estimate than that of the assessment group. The analysis conducted with the second method reduced the 54 pairs of incremental cost effectiveness ratios to 27 pairs, of which two were a tie. The analysis showed that in 21 instances the manufacturers' estimates were lower than the assessment groups' estimates and in the four remaining they were higher (P < 0.001).

Fig 2

Logged pairwise comparison of incremental cost effectiveness ratios. Only 45/54 pairs of recommendations have been plotted because remaining nine pairs contained at least one negative estimate and could not be logged. However, in six of nine pairs estimates from the two sources were broadly similar and in three pairs estimates reported by manufacturers were much more favourable towards the technology than those reported by the assessment groups

• Download figure
• Open in new tab
• Download powerpoint

Discussion

We assessed whether results from economic evaluations by contracted academic centres (assessment groups) and by manufacturers of technologies were similar. Our analysis showed that the estimates of incremental cost effectiveness ratio submitted by manufacturers were on average significantly lower than those provided by the assessment groups. This finding is similar to those reported by others.2^–4 Unlike previous studies, however, ours is based on specific pairwise comparisons of estimates that have been put forward for the same purposes and constructed with the same terms of reference (the scope document).

Whether differences between competing cost effectiveness estimates ultimately matter in terms of decision making depends on several factors, including the level of uncertainty surrounding the incremental cost effectiveness ratios11 and their absolute value. One method of assessing this would have been to divide the dataset into two, with one section including all pairings that contained at least one estimate above an implied maximum threshold of willingness to pay and another containing the remaining pairs of observations. However, the dataset was considered to be too small to do this. Nevertheless, when differences in estimated incremental cost effectiveness ratios do lead to different conclusions, the role of the appraisal committee is to judge the appropriateness of each evaluation and to determine which estimate is the most reasonable.12

NICE will soon be adopting a “reference case” approach designed to be the most appropriate for the NHS to improve consistency across all submitted economic evaluations.13 It is therefore feasible that differences between estimated incremental cost effectiveness ratios submitted will be smaller in the future.

We used two different analyses because, though the first method was more informative, we were concerned about the possibility of overstating the accuracy of the difference being investigated. Both approaches, however, produced similar results and led to the same conclusion.