Skip to main content

Main menu

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
    • Classified ads
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN

User menu

Search

  • Advanced search
CMAJ
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN
CMAJ

Advanced Search

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
    • Classified ads
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
  • Visit CMAJ on Facebook
  • Follow CMAJ on Twitter
  • Follow CMAJ on Pinterest
  • Follow CMAJ on Youtube
  • Follow CMAJ on Instagram
Analysis

Challenges in evaluating SARS-CoV-2 vaccines during the COVID-19 pandemic

Bahaa Abu-Raya, Soren Gantt and Manish Sadarangani
CMAJ August 24, 2020 192 (34) E982-E985; DOI: https://doi.org/10.1503/cmaj.201237
Bahaa Abu-Raya
Vaccine Evaluation Center, BC Children’s Hospital Research Institute, and Division of Infectious Diseases, Department of Pediatrics, University of British Columbia, Vancouver, BC
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Soren Gantt
Vaccine Evaluation Center, BC Children’s Hospital Research Institute, and Division of Infectious Diseases, Department of Pediatrics, University of British Columbia, Vancouver, BC
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Manish Sadarangani
Vaccine Evaluation Center, BC Children’s Hospital Research Institute, and Division of Infectious Diseases, Department of Pediatrics, University of British Columbia, Vancouver, BC
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Tables
  • Related Content
  • Responses
  • Metrics
  • PDF
Loading
KEY POINTS
  • Challenges to the evaluation of candidate vaccines for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) before approval or licensure during the ongoing pandemic include rapidly changing levels of exposure to the virus and population immunity and social distancing practices.

  • To measure vaccine efficacy accurately, researchers should account for these factors in sample size calculations and carefully consider selection of trial end points.

  • Vaccines for SARS-CoV-2 must also be evaluated in populations known to be at increased risk for severe coronavirus disease 2019, such as older adults, Black people and people with multiple comorbidities.

  • Given the speed of vaccine development for SARS-CoV-2, careful attention must be paid to postlicensure assessment of vaccines, including the risk of antibody-dependent enhancement of disease, which must be actively monitored closely over multiple years after vaccination.

Coronavirus disease 2019 (COVID-19) is a rapidly evolving pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). In addition to minimizing risk of transmission by nonpharmacologic measures, 1 or more effective vaccines would be invaluable to reduce the burden of COVID-19. More than 140 candidate SARS-CoV-2 vaccines are in development and being assessed in preclinical studies and clinical trials.1 In general, for a vaccine to be approved or licensed by regulatory authorities, it must demonstrate both safety and high efficacy in the prevention of a specific disease in the relevant populations. However, the COVID-19 pandemic poses specific logistic and scientific challenges with respect to the assessment of SARS-CoV-2 vaccine candidates. Evaluation of the efficacy of SARS-CoV-2 vaccines must consider population risk of exposure, susceptibility to the virus, current social distancing practices and geography. Moreover, SARS-CoV-2 vaccines need to be evaluated in populations at greatest risk for severe COVID-19.2–4 We discuss challenges to the clinical evaluation of SARS-CoV-2 vaccines, as well as some potential solutions.

How are vaccines usually evaluated before licensure?

Vaccines considered for licensure are assessed initially for their safety profile and their ability to induce immune responses to vaccine antigens in small phase 1 studies. Candidate vaccines that are safe and immunogenic may then advance to phase 2 studies, in which safety, immunogenicity and sometimes preliminary efficacy are assessed in larger cohorts. Vaccine efficacy is primarily evaluated in phase 3 studies to determine the proportionate reduction in predefined infection rate or disease events among vaccinated participants. Vaccine efficacy represents the best-case scenario of protection derived from vaccination and needs to be demonstrated for a vaccine against a novel pathogen to be licensed by regulatory authorities.2,3 Ideally, evaluation of efficacy is undertaken in well-controlled studies (e.g., double-blind, randomized, controlled clinical trials). Successful vaccines that have been shown to have high efficacy in clinical trials and are licensed for use in a population are continually assessed for their safety and effectiveness postlicensure and postimplementation (in phase 4 studies that evaluate how a vaccine reduces disease in a population in “real world” conditions).

What factors may affect the evaluation of candidate vaccines for SARS-CoV-2?

In general, host-related (e.g., age, genetic and environmental exposures) and vaccine-related (e.g., antigen selection, adjuvants, formulation, delivery mode and waning of immunity over time) factors influence individuals’ immune response to vaccines and thus determine the vaccines’ efficacy. However, several other factors, some of which may be difficult to quantify, such as the level of transmission of the target pathogen (i.e., exposure) and the level of pre-existing immunity (i.e., susceptibility) in the population are also important to consider. Such factors are related and are different when a pathogen is endemic (i.e., constantly and usually present within a given geographic area or population) versus pandemic (i.e., emerging and spreading worldwide), as with SARS-CoV-2. The baseline level of exposure of a population to a specific pathogen, the pathogen’s seasonality and the level of population immunity are usually relatively predictable in endemic compared with pandemic states of infection. Table 1 summarizes the potential confounders unique to the COVID-19 pandemic that might affect the demonstration of efficacy of candidate vaccines for SARS-CoV-2.

View this table:
  • View inline
  • View popup
  • Download powerpoint
Table 1:

Population-related factors affecting evaluation of vaccine efficacy in endemic versus pandemic states of infection and those unique to coronavirus disease 2019

Level of exposure and immunity to SARS-CoV-2

The dynamic and rapidly changing pattern of virus exposure and level of population immunity during the evolving pandemic are potentially important confounders in the assessment of efficacy of SARS-CoV-2 vaccines; this should be considered in sample size calculations for efficacy trials. For example, if a vaccine is trialled in a low-incidence population, or if immunity wanes substantially over time postvaccination, a highly efficacious vaccine might not show significant protection if the trial sample is too small to demonstrate a significant reduction in occurrence of disease in vaccinated participants in such conditions. Thus, careful attention must be paid to trial sample size calculations, and there may be a need to recruit more participants in areas where the disease prevalence is very low. Moreover, levels of transmission of SARS-CoV-2 might vary between or within countries and will change over time as the pandemic progresses, which must be considered when trialling the same vaccine in different geographic areas.

Population seropositivity, or baseline level of immunity, might affect vaccine immunogenicity or population susceptibility to the target pathogen, which would influence the outcome of a vaccine trial. For example, a highly efficacious vaccine may not show benefit in settings with high seroprevalence, as vaccination may not add substantially to the protection afforded by natural infection. Although this may not be a problem very early in the pandemic while population seropositivity is still low,4 high baseline immunity could confound the results of vaccine trials as the pandemic progresses and seroprevalence increases, depending on the population. For example, the prevalence of anti-SARS-CoV-2 antibodies increased (from 3.1% to 6.1% to 9.7%) during 3 subsequent weeks in April 2020 in a population-based sample in Geneva, Switzerland.5 Measurement of pre-existing immunity in efficacy trials is important, as is the inclusion, vaccination and evaluation of seropositive participants.

Social distancing and other public health interventions

Social distancing has been shown to be effective in mitigating the transmission of SARS-CoV-2.6,7 In settings where transmission is low because of social distancing measures, the benefit of a highly efficacious vaccine might not be readily demonstrable. Study sample size calculations should account for this. Furthermore, public health interventions will likely vary as the pandemic progresses, which will challenge the ongoing assessment of vaccines. Flexibility in trial design may be necessary to achieve a sufficient number of end points to determine efficacy, depending on fluctuating transmission rates in different locations.

What potential vaccine-related harms may be anticipated?

Although the goal of a vaccine is to reduce the burden of COVID-19, SARS-CoV-2 vaccines may theoretically lead to antibody-dependent enhancement.8 This phenomenon, which has been described with SARS-CoV and other coronavirus vaccines in animal models,9 results from low-titre or poorly neutralizing antibody after vaccination that facilitates viral entry or replication in target cells, causing more severe disease from infection in vaccinated individuals. As shown with the CYD-TDV vaccine for dengue, vaccine-induced antibody-dependent enhancement can have enormous negative effects, not only directly among vaccinated individuals, but also on public trust and uptake of other vaccines.10,11 Antibody-dependent enhancement may be observed only after vaccination with specific SARS-CoV-2 vaccine adjuvants, platforms or products — not all. Importantly, vaccine-related antibody-dependent enhancement may become evident only when enough people have been vaccinated and there is high circulation of the virus to show a large burden of COVID-19 among the vaccine population, or it may accompany waning of a vaccine immune response in subsequent years or be related to genotypic changes in the virus over time.12 We suggest that the risk of antibody-dependent enhancement be actively monitored closely over multiple years to account for waning antibody titres or variation in circulating viral strains (Table 1).

Why is careful consideration of study population and trial end points important?

Vaccines for SARS-CoV-2, like other vaccines, are being first studied in populations of healthy adult volunteers. However, COVID-19 has been shown to affect older adults, Black people and people with multiple comorbidities most severely.13,14 Thus, results from low-risk populations may not reflect benefits or risks in higher-risk populations. It is also possible that a vaccine may not provide sterilizing immunity (defined as immune status after vaccination that prevents virus infection of the host). This means that vaccinated individuals may still get the mild form of the disease. Thus, it is possible that vaccination will have a substantially greater impact on preventing severe disease than on preventing milder symptoms or acquisition of infection. These issues should be considered when choosing the clinical end points for trials assessing SARS-CoV-2 vaccine efficacy. For example, if the aim is to show that a vaccine is effective in reducing the severe forms and outcomes of COVID-19, end points such as hospital admission, intensive care unit admission, the need for respiratory support and death could be considered as primary outcomes, although lower rates of more severe events, depending on overall disease incidence rates, may require more trial participants.

It is also important that vaccines be tested in the populations at greatest risk for severe COVID-19, such as older adults, health care workers, Black people and those with predisposing health problems.15 This is critical as it will inform policy-makers about the populations that will benefit the most from vaccination and thus need to be prioritized for vaccination when a vaccine is available but the demand outweighs the supply.

What are some other challenges to the development of SARS-CoV-2 vaccines?

Regulatory challenges

Development of vaccines for human use usually takes at least 10–15 years, from preclinical development to licensure. However, owing to the urgent need for SARS-CoV-2 vaccines, the timeline for their development and approval will need to be shortened substantially, ideally to months rather than years. This may impose a substantial burden on regulatory agencies to process all the usual work in a greatly reduced time frame.

Safety challenges

New technologies for vaccine delivery have been developed during the past decade (e.g., DNA or RNA vaccines); these technologies feature among the SARS-CoV-2 vaccines that have most rapidly progressed to clinical trials, despite their not having been used for any licensed vaccine to date.15 The safety of the SARS-CoV-2 candidate vaccines based on these new technologies must be assessed especially carefully. This will require continuous assessment for both common and rare potential adverse effects in a large number of vaccine recipients, during clinical trials as well as during active postmarketing surveillance.

What are some particular organizational challenges and opportunities of SARS-CoV-2 vaccine trials?

Logistics of multiple vaccines in trials

Studying multiple vaccines at the same time in the same population may be a challenge if there is limited trial capacity. This might be overcome by using an adaptive study design allowing addition or removal of vaccines as data on safety, immunogenicity and efficacy accumulate. This approach is currently being employed in clinical trials assessing different pharmacologic treatments for COVID-19 (e.g., the Solidarity Trial, launched by the World Health Association and partners16). Additionally, infrastructure is being developed to enable sites with experience of nonvaccine drug trials to be used, and for new trial sites to be rapidly set up, including appropriate focused training for research staff.

National and international collaborations

Establishing national and international consortia for SARS-CoV-2 vaccine trials will increase cooperation within and between countries’ vaccine groups. For example, the Canadian Immunization Research Network is a national collaborative network of vaccine researchers who conduct collaborative research related to different aspects of vaccinology. To ensure equity in evaluating vaccines, individual study sites should be representative of the entire population in a specific country, and evaluation of vaccines should be performed in high- as well as low- and middle-income countries. As demand for efficacious vaccines will be high, setting parameters for equitable postlicensure distribution of vaccines across countries is also critical.

How should postlicensure assessment of vaccines be conducted?

Postlicensure assessment of the effectiveness of a vaccine that is being administered in the population is mainly done by retrospective case–control analysis, or ecologic or observational studies. Analyses of these studies should include populations with varying risk of exposure, pre-existing immunity (if known), geography and baseline characteristics (e.g., age and comorbidities). Time since vaccination can also be captured in these studies to assess the effectiveness of vaccination with SARS-CoV-2 vaccines over time after vaccination. This will help to determine whether there is waning immunity after vaccination.

Conclusion

We have discussed the unique challenges in evaluating SARS-CoV-2 vaccines during the ongoing pandemic. These arise from rapidly changing levels of virus exposure, the development of population immunity and the effects of local social distancing practices, as well as the speed with which vaccine candidates are being developed and tested, safety considerations of untested technologies, and the need to consider which end points matter, who is most affected and issues of equity in access to approved vaccines. In addition, as efficacy might be challenging to demonstrate in some populations, researchers should aim to fully dissect the protective immune response (humoral and cellular immunity) to natural infection and after vaccination, as establishing correlates of protection would also inform vaccine development and evaluation.

Acknowledgements

Bahaa Abu-Raya is supported by the Canadian Health and Research Institute Vanier Canada Graduate scholarship. Manish Sadarangani is supported via salary awards from the BC Children’s Hospital Foundation, the Canadian Child Health Clinician Scientist Program and the Michael Smith Foundation for Health Research.

Footnotes

  • Competing interests: Soren Gantt receives consulting fees related to vaccines from Moderna, Merck, GlaxoSmithKline and Curevo; research funding for vaccine research from Merck, GlaxoSmithKline and VBI; and research funding for a COVID-19 vaccine trial from Symvivo. Soren Gantt is also a voting member of the Canadian National Advisory Committee on Immunization. Manish Sadarangani has been an investigator on projects funded by Pfizer, Merck, Seqirus, Sanofi Pasteur, VBI Vaccines and GlaxoSmithKline, and has received research funding for a COVID-19 vaccine trial from Symvivo. All funds have been paid to his institution, and he has not received any personal payments. No other competing interests were declared.

  • This article has been peer reviewed.

  • Contributors: All authors contributed to the conception and design of the work. Bahaa Abu-Raya drafted the manuscript. All authors critically reviewed and revised the article, gave final approval of the final version to be published and agreed to be accountable for all aspects of the work.

References

  1. ↵
    Draft landscape of COVID-19 candidate vaccines. Geneva: World Health Organization; 2020. Available: www.who.int/publications/m/item/draft-landscape-of-covid-19-candidate-vaccines (accessed 2020 May 26).
  2. ↵
    1. Clemens J,
    2. Brenner R,
    3. Rao M,
    4. et al
    . Evaluating new vaccines for developing countries. Efficacy or effectiveness? JAMA 1996;275:390–7.
    OpenUrlCrossRefPubMed
  3. ↵
    1. Weinberg GA,
    2. Szilagyi PG
    . Vaccine epidemiology: efficacy, effectiveness, and the translational research roadmap. J Infect Dis 2010;201:1607–10.
    OpenUrlCrossRefPubMed
  4. ↵
    1. Sood N,
    2. Simon P,
    3. Ebner P,
    4. et al
    . Seroprevalence of SARS-CoV-2-Specific Antibodies Among Adults in Los Angeles County, California, on April 10–11, 2020. JAMA 2020;323:2425–7.
    OpenUrl
  5. ↵
    1. Stringhini S,
    2. Wisniak A,
    3. Piumatti G,
    4. et al
    . Repeated seroprevalence of anti-SARS-CoV-2 IgG antibodies in a population-based sample from Geneva, Switzerland. medRxiv 2020 May 6. doi: 10.1101/2020.05.02.20088898.
    OpenUrlAbstract/FREE Full Text
  6. ↵
    1. Shoukat A,
    2. Wells CR,
    3. Langley JM,
    4. et al
    . Projecting demand for critical care beds during COVID-19 outbreaks in Canada. CMAJ 2020;192:E489–96.
    OpenUrlAbstract/FREE Full Text
  7. ↵
    1. Tuite AR,
    2. Fisman DN,
    3. Greer AL
    . Mathematical modelling of COVID-19 transmission and mitigation strategies in the population of Ontario, Canada. CMAJ 2020; 192:E497–505.
    OpenUrlAbstract/FREE Full Text
  8. ↵
    1. Hotez PJ,
    2. Corry DB,
    3. Bottazzi ME
    . COVID-19 vaccine design: the Janus face of immune enhancement. Nat Rev Immunol 2020;20:347–8.
    OpenUrlCrossRef
  9. ↵
    1. Tseng CT,
    2. Sbrana E,
    3. Iwata-Yoshikawa N,
    4. et al
    . Immunization with SARS coronavirus vaccines leads to pulmonary immunopathology on challenge with the SARS virus. PLoS One 2012;7:e35421.
    OpenUrlCrossRefPubMed
  10. ↵
    1. Halstead SB
    . Critique of World Health Organization recommendation of a dengue vaccine. J Infect Dis 2016;214:1793–5.
    OpenUrlCrossRefPubMed
  11. ↵
    Philippines gripped by dengue vaccine fears. British Broadcasting Corporation 2018 Feb. 3. Available: www.bbc.com/news/world-asia-42929255 (accessed 2020 May 26).
  12. ↵
    1. Korber B,
    2. Fischer WM,
    3. Gnanakaran S,
    4. et al
    . Spike mutation pipeline reveals the emergence of a more transmissible form of SARS-CoV-2. bioRxiv 2020 Apr. 30. doi: 10.1101/2020.04.29.069054.
    OpenUrlAbstract/FREE Full Text
  13. ↵
    1. Yancy CW
    . COVID-19 and African Americans. JAMA 2020;323:1891–1892.
    OpenUrlCrossRefPubMed
  14. ↵
    1. Onder G,
    2. Rezza G,
    3. Brusaferro S
    . Case-fatality rate and characteristics of patients dying in relation to COVID-19 in Italy. JAMA 2020 Mar. 23 [Epub ahead of print]. doi: 10.1001/jama.2020.4683.
    OpenUrlCrossRefPubMed
  15. ↵
    1. Amanat F,
    2. Krammer F
    . SARS-CoV-2 vaccines: status report. Immunity 2020; 52: 583–9.
    OpenUrlCrossRefPubMed
  16. ↵
    1. Cheng MP,
    2. Lee TC,
    3. Tan DHS,
    4. et al
    . Generating randomized trial evidence to optimize treatment in the COVID-19 pandemic. CMAJ 2020;192:E405–7.
    OpenUrlFREE Full Text
PreviousNext
Back to top

In this issue

Canadian Medical Association Journal: 192 (34)
CMAJ
Vol. 192, Issue 34
24 Aug 2020
  • Table of Contents
  • Index by author

Article tools

Print
Download PDF
Article Alerts
To sign up for email alerts or to access your current email alerts, enter your email address below:
Email Article

Thank you for your interest in spreading the word on CMAJ.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Challenges in evaluating SARS-CoV-2 vaccines during the COVID-19 pandemic
(Your Name) has sent you a message from CMAJ
(Your Name) thought you would like to see the CMAJ web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Challenges in evaluating SARS-CoV-2 vaccines during the COVID-19 pandemic
Bahaa Abu-Raya, Soren Gantt, Manish Sadarangani
CMAJ Aug 2020, 192 (34) E982-E985; DOI: 10.1503/cmaj.201237

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
‍ Request Permissions
Share
Challenges in evaluating SARS-CoV-2 vaccines during the COVID-19 pandemic
Bahaa Abu-Raya, Soren Gantt, Manish Sadarangani
CMAJ Aug 2020, 192 (34) E982-E985; DOI: 10.1503/cmaj.201237
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • How are vaccines usually evaluated before licensure?
    • What factors may affect the evaluation of candidate vaccines for SARS-CoV-2?
    • What potential vaccine-related harms may be anticipated?
    • Why is careful consideration of study population and trial end points important?
    • What are some other challenges to the development of SARS-CoV-2 vaccines?
    • What are some particular organizational challenges and opportunities of SARS-CoV-2 vaccine trials?
    • How should postlicensure assessment of vaccines be conducted?
    • Conclusion
    • Acknowledgements
    • Footnotes
    • References
  • Figures & Tables
  • Related Content
  • Responses
  • Metrics
  • PDF

Related Articles

  • Difficultés associées à l’évaluation des vaccins contre le SRAS-CoV-2 pendant la pandémie de COVID-19
  • PubMed
  • Google Scholar

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • Integrating research into clinical practice: challenges and solutions for Canada
  • Managing conflicts of interest in the development of health guidelines
  • COVID-19 in long-term care homes in Ontario and British Columbia
Show more Analysis

Similar Articles

Collections

  • Topics
    • Vaccination
    • Infectious diseases: COVID-19

Content

  • Current issue
  • Past issues
  • Collections
  • Sections
  • Blog
  • Podcasts
  • Alerts
  • RSS
  • Early releases

Information for

  • Advertisers
  • Authors
  • Reviewers
  • CMA Members
  • Media
  • Reprint requests
  • Subscribers

About

  • General Information
  • Journal staff
  • Editorial Board
  • Governance Council
  • Journal Oversight
  • Careers
  • Contact
  • Copyright and Permissions

Copyright 2021, Joule Inc. or its licensors. All rights reserved. ISSN 1488-2329 (e) 0820-3946 (p)

All editorial matter in CMAJ represents the opinions of the authors and not necessarily those of the Canadian Medical Association or its subsidiaries.

Powered by HighWire