Abstract
Background: Clinical trials and meta-analyses have produced conflicting results of the efficacy of unconjugated pneumococcal polysaccharide vaccine in adults. We sought to evaluate the vaccine's efficacy on clinical outcomes as well as the methodologic quality of the trials.
Methods: We searched several databases and all bibliographies of reviews and meta-analyses for clinical trials that compared pneumococcal polysaccharide vaccine with a control. We examined rates of pneumonia and death, taking the methodologic quality of the trials into consideration.
Results: We included 22 trials involving 101 507 participants: 11 trials reported on presumptive pneumococcal pneumonia, 19 on all-cause pneumonia and 12 on all-cause mortality. The current 23-valent vaccine was used in 8 trials. The relative risk (RR) was 0.64 (95% confidence interval [CI] 0.43–0.96) for presumptive pneumococcal pneumonia and 0.73 (95% CI 0.56–0.94) for all-cause pneumonia. There was significant heterogeneity between the trials reporting on presumptive pneumonia (I2 = 74%, p < 0.001) and between those reporting on all-cause pneumonia (I2 = 90%, p < 0.001). The RR for all-cause mortality was 0.97 (95% CI 0.87–1.09), with moderate heterogeneity between trials (I2 = 44%, p = 0.053). Trial quality, especially regarding double blinding, explained a substantial proportion of the heterogeneity in the trials reporting on presumptive pneumonia and all-cause pneumonia. There was little evidence of vaccine protection in trials of higher methodologic quality (RR 1.20, 95% CI 0.75–1.92, for presumptive pneumonia; and 1.19, 95% CI 0.95–1.49, for all-cause pneumonia in double-blind trials; p for heterogeneity > 0.05). The results for all-cause mortality in double-blind trials were similar to those in all trials combined. There was little evidence of vaccine protection among elderly patients or adults with chronic illness in analyses of all trials (RR 1.04, 95% CI 0.78–1.38, for presumptive pneumococcal pneumonia; 0.89, 95% CI 0.69–1.14, for all-cause pneumonia; and 1.00, 95% CI 0.87–1.14, for all-cause mortality).
Interpretation: Pneumococcal vaccination does not appear to be effective in preventing pneumonia, even in populations for whom the vaccine is currently recommended.
The burden of disease due to Streptococcus pneumoniae falls mainly on children, elderly people and people with underlying conditions such as HIV infection.1 Pneumococcal polysaccharide vaccines were developed more than 50 years ago and have progressed from 2-valent vaccines to the current 23-valent vaccine, which has been available since the early 1980s. The 23-valent vaccine includes serotypes accounting for 72%2 to 95%3 of invasive pneumococcal disease, depending on the geographic area. In many industrialized countries, pneumococcal vaccination is currently recommended for people aged 65 years and older and for individuals aged 2–64 who are at increased risk of pneumococcal disease.4–6
Meta-analyses of controlled clinical trials have produced conflicting results of the efficacy of unconjugated pneumococcal polysaccharide vaccine.7–22 The lack of consistency between results reported from observational studies and controlled trials is another reason why the efficacy of the vaccine remains controversial. Empirical studies have shown that inadequate quality of clinical trials can lead to biases that distort their results.23 For example, inadequate allocation concealment or failure to blind patients, caregivers or those assessing outcomes may exaggerate treatment effects.23 Despite this, none of the previous reviews formally compared effect sizes in trials of high methodologic quality with effect sizes in trials of lower quality. We conducted a systematic review and meta-analysis of clinical trials examining the efficacy of pneumococcal polysaccharide vaccination on clinical outcomes, taking the quality of trials into account.
Methods
Data sources
We searched MEDLINE (1966 to May 10, 2007) and EMBASE (1974 to May 10, 2007) as well as the Cochrane Central Register of Controlled Trials (CENTRAL) using the key words “pneumococcal vaccine” or “pneumococcal immunization” in combination with “polysaccharide.” For the searches of MEDLINE and EMBASE, these key words were combined with terms describing the study design: “randomized controlled trial,” “clinical trial” and “controlled clinical trial”; details of the searches appear in Appendix 1, available at www.cmaj.ca/cgi/content/full/180/1/48/DC2.
In addition, we searched the LILACS (Latin American and Caribbean Health Sciences Literature), IndMed (Indian Medlars Centre) and AIM (African Index Medicus) databases using “pneumococc*”, “vac*” and “polysac*.” We also searched the Cochrane Library for meta-analyses and systematic reviews, using the search terms “pneumococc*” and “polysac*.”
We screened bibliographies of retrieved articles for relevant studies, review articles and meta-analyses.
Study selection
We included clinical trials that compared pneumococcal polysaccharide vaccine with placebo, other vaccines or no intervention; reported on selected clinical outcomes or death; and allocated participants prospectively and concurrently to comparison groups using random allocation or some quasi-random method of allocation (e.g., alternation, date of birth or case record number). No language restrictions were applied. We excluded uncontrolled studies, observational intervention studies, and animal and laboratory studies; trials that examined antibody responses only, since immunologic correlates of protection are poorly defined;21 trials involving children; and studies in which pneumococcal polysaccharide vaccine was used as a booster after vaccination with conjugate pneumococcal vaccine.
Data extraction
Two reviewers (A.H. and P.S.) independently evaluated all retrieved articles for eligibility for inclusion in the meta-analysis. The same 2 reviewers independently extracted data on the study population, intervention types (valency of vaccine and control interventions), sample size, sample size calculations, loss to follow-up, outcomes and source of funding. We also extracted information on the methodologic quality of the studies, including whether allocation of study participants to trial arms was concealed, whether the trials were described as double-blind and who was blinded. The observed inter-rater agreement was measured using the kappa statistic. Discrepancies were resolved by consensus, with the senior epidemiologist (M.E.) acting as the arbiter.
Outcomes of the meta-analysis
We considered the following 8 outcomes: (a) definitive pneumococcal pneumonia, defined as typical clinical or radiologic findings and S. pneumoniae isolated from normally sterile body fluid such as blood; (b) presumptive pneumococcal pneumonia, defined as typical clinical or radiologic findings, and either S. pneumoniae isolated from respiratory tract samples or seroconversion against S. pneumoniae; (c) pneumonia from all causes; (d) bronchitis from all causes; (e) death from all causes; (f) death from pneumonia; (g) death from pneumococcal infection; and (h) bacteremia or invasive pneumococcal disease, defined as S. pneumoniae isolated from a usually sterile body fluid such as blood.
Statistical analysis
We conducted DerSimonian and Laird random-effects meta-analyses24 and calculated tests of heterogeneity. We quantified between-trial heterogeneity using the I2 statistic, which can be interpreted as the proportion of the total variation in estimated risk ratios that is due to between-trial heterogeneity rather than to chance.25 Low, moderate and high levels of heterogeneity correspond to I2 values of 25%, 50% and 75% respectively. Analyses were based on episodes when available; otherwise we used the number of individuals who experienced the event. For trials with more than one control group, we analyzed data for the saline placebo group where reported; otherwise we used data for combined control groups.
For outcomes reported by 10 or more trials, we stratified analyses by trial quality (blinding and concealment of allocation), study setting and study population. For all-cause pneumonia, the extent to which one or more of these variables explained heterogeneity was further explored using random-effects meta-regression models, adjusted and not adjusted for trial quality. We examined to what extent between-study variance (tau squared) was reduced in models including different sets of variables. Blinding was analyzed in 3 groups: double-blind (double-blind, with placebo or other vaccine used in control group); controlled (not double-blind, but placebo or other vaccine used in control group); and open (no intervention used in control group). Concealment of allocation was classified as adequate or unclear. We classified the study population as elderly people and adults at increased risk because of chronic illness or a history of pneumonia; miners or soldiers; children; and “other.” HIV-infected patients were included in the “other” group rather than the chronically ill group. The trial settings were classified in 3 groups: Western Europe and North America; Africa, Papua New Guinea or the Caribbean; and Russia. Differences in the results between small and large trials were assessed by visual inspection of funnel plots and, for outcomes reported by 10 or more trials, by means of a statistical test for asymmetry of funnel plots.26
We calculated relative risks (RRs) from meta-analyses and ratios of RRs from meta-regression, with 95% confidence intervals (CIs). Unless otherwise specified, p values relate to tests of heterogeneity.
Results
Selected trials
Figure 1 depicts the process of identifying trials for the meta-analysis. We considered 91 of 516 references to be potentially eligible and examined the full text of these publications. We included 22 trials, described in 18 reports, that met our inclusion criteria (Table 1).27–44 An overview of the study characteristics is given in Table 1, with further details in Appendix 2 (available at www.cmaj.ca/cgi/content/full/180/1/48/DC2).
Fourteen trials (64%) were performed in Western Europe or the United States, 5 (23%) in lower income countries (South Africa, Uganda, Papua New Guinea and Jamaica) and 3 (13%) in Russia. There were 13 studies (59%) involving elderly people or adults with chronic illness or a history of pneumonia, 7 (32%) involving miners or soldiers, and 2 (9%) involving other populations. The 23-valent vaccine was used in 8 trials (36%).
Seven of the trials (32%) were described as double-blind; 5 of these reported who was blinded. Of the 15 trials that were not described as double-blind, 7 described blinding of either participants or trial staff. Seven (32%) were open trials, with no intervention in the control groups. Concealment of allocation was described in 10 (45%) of the 22 trials; it was done appropriately in 6 of these. Agreement on study quality was near perfect for double-blinding (kappa = 0.9) and good for concealment of allocation (kappa = 0.55). Eleven trials (50%) reported maximum follow-up times (mean 2.6 years), and 7 (32%) reported mean follow-up times (mean 2.1 years); 6 trials (27%) did not provide information on duration of follow-up. Seven trials (32%) reported on loss to follow-up; 4 (18%) reported loss to follow-up by intervention group.
Diagnostic criteria for pneumonia
Of the 21 trials that reported on all-cause pneumonia or presumptive or definitive pneumococcal pneumonia, 14 (67%) reported on cases that were radiographically confirmed (Appendix 2, available online at www.cmaj.ca/cgi/content/full/180/1/48/DC2). Of the 11 trials reporting on presumptive pneumococcal pneumonia, 7 based this on culture of S. pneumoniae in addition to clinical and radiologic findings. The criteria for death from pneumonia were not well described: 2 of the 8 trials reporting on this outcome used data from death certificates; 1 used data provided by relatives of the deceased.
Meta-analyses
Depending on the outcome, from 2 to 19 trials were included in our meta-analyses (Table 2). There were between 794 and 82 665 study participants in each analysis. The number of patients with the outcome of interest ranged from 7 to 2722. The forest plots for the 8 outcomes are shown in Figure 2.
Combined RRs from random-effects meta-analyses, without trial quality being taken into account, indicated beneficial effects of the vaccine on presumptive pneumococcal pneumonia (RR 0.64, 95% CI 0.43–0.96) and all-cause pneumonia (0.73, 95% CI 0.56–0.94); there was pronounced heterogeneity between trials (p < 0.001). The results for definitive pneumococcal pneumonia were compatible with a protective effect; however, the analysis was based on 2 trials and 7 events only, and confidence intervals were wide (RR 0.62, 95% CI 0.05–8.61). Confidence intervals for death from pneumonia were also wide and included 1 (RR 0.88, 95% CI 0.62–1.25). The RRs for other outcomes, including bacteremia, were close to 1, indicating no benefit of the vaccine (Table 2).
Analyses stratified by methodologic quality, study population, study setting and vaccine valency are shown in Table 3 for outcomes reported by 10 or more trials: presumptive pneumococcal pneumonia, all-cause pneumonia and all-cause mortality. Trials of higher quality (double-blind or adequate concealment of allocation) showed no benefit of the vaccine, in contrast to trials of lower quality. The influence of trial quality on effect estimates is illustrated in Figure 3. When trial quality was ignored, there was little evidence of benefit among elderly patients or patients with chronic illness. The risk of pneumonia was reduced among miners and soldiers; however, studies involving miners and soldiers tended to be of lower quality than those involving elderly patients or chronically ill patients.
Meta-regression analyses
The results of the meta-regression analyses of the 19 trials that examined all-cause pneumonia are shown in Table 4. Results are presented as ratios of RRs (relative risk with characteristic divided by relative risk without characteristic). Ratios above 1.0 correspond to a larger relative risk for trials with the characteristic and hence a smaller apparent benefit of the vaccine. Overall, the between-study variance, tau2, was 0.32. In the univariable analysis, the degree of blinding reduced tau2 to 0.16. Less reduction was seen for concealment of allocation. Tau2 was also reduced in models that included the type of study population (to 0.19) and setting (to 0.20). The year of publication, the year the study started and the vaccine valency had little influence on the between-trial variance.
In the multivariable models, after adjustment for blinding and allocation concealment, we found little evidence for a difference in RRs between groups, except for the analysis of study populations: the ratio of RRs was 0.50 (95% CI 0.31–0.80) for miners and soldiers. This value indicated an apparent beneficial effect of the vaccine in this group compared with other study populations, including elderly and chronically ill people (Table 4). However, none of the trials that involved miners and soldiers were described as double-blind, which makes it statistically difficult to control for blinding. Finally, there was little evidence of funnel plot asymmetry (p > 0.10 for all outcomes).
Interpretation
In our systematic review and meta-analysis, we found a high degree of heterogeneity between trials in the efficacy of unconjugated pneumococcal polysaccharide vaccine in the prevention of a range of clinical outcomes. Much of this heterogeneity could be explained by differences in the methodologic quality of the trials. We found little evidence of protection among elderly people or adults with chronic respiratory illness, for whom the pneumococcal vaccine is recommended in many industrialized countries. Trials of higher quality (i.e., those with a double-blind design and adequate concealment of allocation) generally showed little evidence of a protective vaccine effect, regardless of the study population and setting.
The first meta-analysis of trials of the pneumococcal polysaccharide vaccine, published in 1994, reported protective effects on definitive and presumptive pneumococcal pneumonia.22 Since then, numerous meta-analyses have been published, some of which focused on specific populations or outcomes (these meta-analyses are summarized in Appendix 3, available at www.cmaj.ca/cgi/content/full/180/1/48/DC2). Several meta-analyses reported protective effects of the vaccine on all-cause pneumonia in subgroups, for example low-risk groups or populations from lower income countries.11,16,19 Some previous meta-analyses agreed with our finding of no compelling evidence for the vaccine being associated with a reduced risk of all-cause pneumonia, particularly among elderly or chronically ill people.7,16,17,19,22 Conclusions for other outcomes were often conflicting, especially for invasive pneumococcal disease.7,11,12,16,20
A recent Cochrane review7 found strong evidence supporting the vaccine's efficacy against invasive pneumococcal disease, reporting a combined odds ratio of 0.26 (95% CI 0.15–0.46). This finding contrasts with our result (combined RR 0.90, 95% CI 0.46–1.77) and is largely explained by the inclusion, in the Cochrane review, of the 1977 study involving Papua New Guinea highlanders40 and the 1947 study involving elderly people in New York.34 We excluded the Papua New Guinea study40 because the material examined to diagnose invasive disease included lung aspirates and because access to care and diagnostic procedures were limited. We excluded the New York study34 from our analyses of invasive pneumococcal disease because a substantial number of the participants had volunteered to be vaccinated and were not randomly allocated to the vaccine group.45
Of the 17 meta-analyses of the pneumococcal polysaccharide vaccine that we identified, 13 assessed trial quality (generally by composite scores), and 10 studies used the quality scores for sensitivity analyses and excluded trials with the lowest quality scores. However, none of the previous meta-analyses formally considered study quality in their analyses as a means to explore causes of heterogeneity of results as we did. This is not surprising: a recent bibliographic study of 272 meta-analyses of clinical trials46 found that 140 (52%) had assessed the quality of component studies, but only 66 (24%) stratified analyses by trial quality.
One might expect that observational studies, because of their nature, would be more likely than randomized controlled trials to show a protective effect of the vaccine. The situation has been observed in the literature on influenza vaccination, where cohort studies have shown large reductions in all-cause mortality, in contrast to the results from randomized controlled trials.47
These discrepant results could be explained by selection bias, if vaccinated individuals were at lower risk of the outcomes than unvaccinated individuals.47 Studies from the United States have documented substantial racial and ethnic differences in the uptake of vaccination and have shown that individual attitudes and preferences are important determinants for the uptake of preventive interventions in general.48,49 Such attitudes are also important predictors of outcome: participants in clinical trials who adhere to prescribed medicines, whether active treatment or placebo, have been found to have reduced mortality compared with those who do not adhere to the prescribed medicines.50,51 Conversely, individuals with skeptical attitudes toward health care have increased mortality, perhaps because of higher rates of unhealthy behaviours.52
The majority of observational studies examining the pneumococcal polysaccharide vaccine have reported large protective effects.53–55 For example, a recent case–control study in a long-term care setting in Austria found strong evidence (p < 0.0001) for reductions in the risk of pneumonia (odds ratio 0.28) and death (odds ratio 0.27).55 Even within observational studies, some have shown little effect. A large cohort study in the United States showed little evidence of a protective effect of the vaccine against pneumonia among elderly people, but it did show a protective effect against pneumococcal bacteremia.56
The conflicting results between observational studies and clinical trials of lower and higher methodologic quality illustrate the difficulty in the interpretation of results from studies at risk of bias.
Empirical evidence and theoretical considerations support the notion that trials of higher methodologic quality will provide results closer to the truth than lower quality trials will.23,57,58 Meta-analysis cannot prove that this is necessarily the case here, since other factors associated with both the quality of trials and their results could have confounded the analyses. For example, the immunogenicity of the different pneumococcal vaccine antigens varies, and differences in the distribution of serotypes across studies would be expected to introduce some heterogeneity.59 Notably, when we restricted analyses to trials of higher quality, we found that between-trial heterogeneity was reduced, which speaks against the presence of important confounding. Interestingly, blinding was more important than concealment of allocation. Neither blinding nor concealment of allocation was important for all-cause mortality. This is in accordance with recent research that examined 146 meta-analyses of clinical trials with a range of outcomes.60 In trials with subjective outcomes, effect estimates were exaggerated when there was no blinding or the concealment of allocation was inadequate; however, there was little evidence of bias in trials with objective outcomes, including all-cause mortality.60
Fedson,61 in 2003, argued that clinical trials of pneumococcal polysaccharide vaccine were “destined from the outset to be inconclusive” because they suffered from methodologic problems and were too small to reliably show effects on all-cause pneumonia. We included several trials published since then and found that, when we restricted our analysis to trials of higher quality, substantial beneficial effects on all-cause pneumonia were excluded with confidence. This was not the case for some of the other outcomes, in particular invasive pneumococcal disease, because the analyses were based on few trials and few events. Mangtani and colleagues59 argued that the pneumococcal polysaccharide vaccine might prevent pneumococcal bacteremia but not pneumonia because of poor production of opsonizing antibodies.62 Our results are not incompatible with this possibility, but also provide little evidence for it.
Of concern was that our meta-analysis of high-quality trials did not exclude a detrimental effect of the pneumococcal polysaccharide vaccine on presumptive pneumonia or all-cause pneumonia. The large cohort study from the United States also showed an increase in risk among elderly people.56 Our results for all-cause pneumonia were strongly influenced by a double-blind trial involving HIV-positive adults in Uganda, which showed a substantial, detrimental effect of the vaccine.31 The authors suggested a mechanism related to HIV infection that may explain the harmful effect. However, they also speculated that there may be a more general, as yet unknown, explanation.
Strengths and weaknesses
The strengths of our review include a thorough literature search and the novel comprehensive analysis of potential sources of heterogeneity between trials. Poor reporting of the methodology of the trials, many of which were published before the Consolidated Standards of the Reporting of Trials (CONSORT)57 became available, meant that trial quality could probably not be precisely assessed. The level of allocation concealment remained unclear in many trials. Also, authors often did not state who exactly was blinded and whether this varied across outcomes.
The results of individual trials and therefore our meta-analysis may have been affected by inaccurate diagnoses of outcomes. The diagnosis of presumptive pneumococcal pneumonia was based on the symptoms and signs of pneumonia combined with culture of S. pneumoniae in sputum. Culture in sputum does not, however, distinguish between infection and colonization, which could lead to an incorrect diagnosis of pneumococcal pneumonia in carriers of S. pneumoniae. This may have led to an underestimate of the vaccine's efficacy. Further bias may have been introduced if diagnostic errors affected groups differentially.63
Our protocol stipulated that adverse events would be examined. However, this was not systematically assessed because of poor reporting: adverse events were often listed only for a subgroup of participants or not reported separately for vaccine and control groups. Nevertheless, no serious short-term side effects were reported in any of the trials.
Policy implications and conclusions
The pneumococcal polysaccharide vaccine has been widely introduced into vaccination programs for elderly people and high-risk populations in industrialized countries. In many cases, decisions to implement vaccination programs were based on a possible protective effect of the pneumococcal polysaccharide vaccine on invasive pneumococcal disease, which has been seen in observational studies but not clearly documented in high-quality clinical trials.
Meta-analysis can be a useful tool for assessing vaccine efficacy and informing health authorities. However, the conclusions that can be drawn from them depend on the methodologic quality of trials analyzed. We applied a robust method for assessing trial quality, which we found was important in the context of pneumococcal polysaccharide vaccination. Our study supports the notion that further high-quality trials of the vaccine would fail to show any protective effect against pneumonia. Future trials should therefore focus on protein conjugate vaccines, which have been shown to be efficacious in children.64 They should also maximize the accuracy of the diagnosis of pneumococcal pneumonia using newer tests65 and ensure appropriate blinding of outcome assessors.
The prevention of the large burden of disease associated with pneumococcal pneumonia66,67 should be a major objective from a public health perspective. This will not be achieved with the use of the currently available pneumococcal polysaccharide vaccine, even allowing for a modest protective effect against invasive pneumococcal disease. Furthermore, countries that have introduced pneumococcal conjugate vaccines into childhood immunization programs can reasonably expect to experience reductions in the incidence of invasive pneumococcal disease, even among unvaccinated adults, because of reduced transmission of vaccine-type pneumococci in the population.64,68 This reduction in the burden of invasive pneumococcal disease may make continued use of the polysaccharide vaccine in elderly populations even less attractive. It may also suggest that alternative approaches to the control of pneumococcal disease in adults may be more effective. Policy-makers may therefore wish to reconsider their current recommendations for the use of pneumococcal polysaccharide vaccine, especially where routine use of the conjugate vaccine has been introduced.
@@ See related commentary by Andrews and Moberley, page 18
Footnotes
-
Une version française de ce résumé est disponible à l'adresse www.cmaj.ca/cgi/content/full/180/1/48/DC1
Funding: This study was funded by the World Health Organization (contract no. OD/AP-07-02439).
This article has been peer reviewed.
Contributors: Anke Huss, Pippa Scott and Matthias Egger designed the study and drafted the manuscript. Anke Huss and Pippa Scott completed the literature search, extracted the data and performed the data analysis. Andreas Stuck and Caroline Trotter helped with the interpretation of the results. All of the authors revised the manuscript critically for important intellectual content and approved the final version submitted for publication.
Acknowledgements: We thank Natalia Smirnov for translating the article written in Russian and Arthur Reingold for valuable comments on a previous draft of this manuscript.
The views expressed in this article are entirely those of the authors and not necessarily those of the funding agency.
Competing interests: None declared.
REFERENCES
- 1.↵
- 2.↵
- 3.↵
- 4.↵
- 5.
- 6.↵
- 7.↵
- 8.
- 9.
- 10.
- 11.↵
- 12.↵
- 13.
- 14.
- 15.
- 16.↵
- 17.↵
- 18.
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.↵
- 24.↵
- 25.↵
- 26.↵
- 27.↵
- 28.
- 29.
- 30.
- 31.↵
- 32.
- 33.
- 34.↵
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.↵
- 41.
- 42.
- 43.
- 44.↵
- 45.↵
- 46.↵
- 47.↵
- 48.↵
- 49.↵
- 50.↵
- 51.↵
- 52.↵
- 53.↵
- 54.
- 55.↵
- 56.↵
- 57.↵
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.↵
- 66.↵
- 67.↵
- 68.↵