The performance of COVID-19 vaccines in clinical trials and in real world conditions

There are currently three COVID-19 vaccines that have been approved for use in the UK and mass immunisation against COVID-19 has started around the world in the past couple of months.

This article will focus on how vaccine performance is measured, what an ideal vaccine would look like and the latest data available from clinical trials on the COVID-19 vaccines approved in the UK. Current immunisation strategies in the UK and the latest real-world data on vaccine performance will also be described.

How do we measure how well vaccines work?

There are already robust methods to work out how well a vaccine works as part of randomised controlled clinical trials. Testing begins in a small group of people to check safety (Phase 1 trials), before proceeding to several hundreds to confirm safety and understand whether the vaccine stimulates an immune response (Phase 2 trials). Finally, several thousands of people receive the vaccine to check for rare side effects and how well a vaccine protects against the infection or disease (Phase 3 trials).

Clinical trials are conducted by a research team, including doctors, nurses, statisticians, data managers, clinical trial coordinators, among others. They work within organisations such as hospitals, universities or pharmaceutical companies that act as ‘sponsors’ and have legal responsibility for the trial. Before starting clinical trials in the UK, regulators such as the Medicines & Healthcare products Regulatory Agency (MHRA) and the Human Research Authority (HRA) need to grant permission. All trials approved by HRA must be registered on a publicly accessible database. For COVID-19 studies, the HRA has developed a fast-track approval process, which allows rapid review from its Research Ethics Committee.

Usually, when a research team tests a new medicine, they collect evidence until the end of the trial and then submit it to the MHRA for medicines approval. This process is different for COVID-19 vaccines, as the MHRA has developed a rolling-review process. Once 12 months have passed since the end of any trial, a clinical trial summary report must be published in the clinical trials database.

Together with the Commission on Human Medicines (CHM, a UK Government independent advisory body), the MHRA uses this evidence to advise on the safety, quality and efficacy of vaccines before granting approval. A previous POST rapid response describes in more details the Regulatory approval of COVID-19 vaccines in the UK.

Evidence collected at this stage is also used by the Joint Committee on Vaccination and Immunisation (JCVI), a committee that advises on vaccination priorities according to the data available. Following approval, vaccines are constantly monitored to evaluate any adverse effects and to determine how well they work long-term. A previous POST rapid response describes this process in more detail (sometimes referred to as ‘Phase 4’) – see Monitoring COVID-19 vaccine safety in national immunisation programmes.

On 5 February 2021, the MHRA published its safety surveillance strategy for monitoring all UK-approved COVID-19 vaccines.

What is the difference between efficacy and effectiveness?

Vaccine efficacy is the performance of the vaccine candidate during randomised controlled clinical trials, while effectiveness is the performance of a vaccine in real-life conditions, evaluated during post-license observational studies. Both assess how well a vaccine works, often measured as the difference in rates of disease between vaccinated and unvaccinated people. For example, a vaccine with 90% efficacy means that during clinical trials, the vaccine reduced the risk of infection by 90% when compared with people receiving the placebo.

Efficacy and effectiveness are often quite different. For example, clinical trials often recruit healthy younger volunteers, who are monitored for a relatively short period of time, while in real-life, age, other pathologies (such as immunodeficiency) and uptake of the vaccine by the population can have an impact on vaccine performance.

What can affect vaccine performance?

There are two different categories of vaccine. Vaccines preventing infections are called ‘sterilising vaccines’, while those that reduce the severity of the disease are called ‘disease-modifying vaccines’. One of the challenges during large scale Phase 3 clinical trials is to determine if the vaccine being tested belongs to one of these categories.

Sterilising vaccines reduce transmission (the spread of the virus from one infected person to another) and prevent carriage (when a virus is present in the body without causing any harm to the person infected, who remains asymptomatic or pre-symptomatic).

Disease-modifying vaccines limit the severity of infection (including reducing symptoms). For example, they could lead to a reduction in the number of severe cases (and therefore hospitalisation and mortality), which instead would be mild or asymptomatic.

Different vaccines can trigger an immune response mediated by different parts of the immune system (for example T cells and antibodies, see Immunity to COVID-19). The duration of the immune response is another important factor. Vaccine-induced antibody levels and T cell reactivity can be monitored over time to evaluate the duration of vaccine-triggered immunity. Alternatively, surveillance programmes can monitor infections in the vaccinated population. A previous POST rapid response describes in more detail how surveillance programmes work and which tests they use.

The British Society of Immunology recently published a report on Immunity and COVID-19 with more details on immunity to SARS-CoV-2 following vaccination.

What would an ideal vaccine look like?

An ideal COVID-19 vaccine would prevent SARS-CoV-2 infection, confer long-lasting immunity and abolish the transmission of SARS-CoV-2 in the population. Even if a vaccine is only disease-modifying, it would still lead to substantial public health benefits. For example, a vaccine that makes the incidence of more serious disease less likely will mean that fewer people become seriously ill or die.

The World Health Organization (WHO) has guidelines for how to assess COVID-19 vaccine efficacy. This can be done by working out how well a vaccine candidate acts to prevent symptomatic disease, severe disease, viral transmission or viral shedding (the release of virus from an infected individual with the risk it is transmitted to others).

Evaluating how well COVID-19 vaccines work is challenging. This is because the severity of disease and mortality rates vary according to age, sex and ethnicity. Trials can also be impacted by other interventions to tackle a disease. For example, when rates of COVID-19 are lower, due to local lockdowns, then it is harder to evaluate infection rates between vaccinated and non-vaccinated participants.

What standards are expected to be met by COVID-19 vaccines?

The WHO recommends that successful COVID-19 vaccines should demonstrate at least 50% efficacy at a population level. Similarly, the US Food and Drug Administration requires 50% efficacy for granting an ‘Emergency Use Authorization’ to COVID-19 vaccines. In a recent open letter, the European Medicines Agency (EMA) stated that “any authorised COVID-19 vaccines must have the highest possible degree of efficacy and safety”. DHSC’s ‘Consultation document for changes to Human Medicine Regulations to support the rollout of COVID-19 vaccines’ stated that “a COVID-19 vaccine would only be authorised (…) [ in the UK] if the licensing authority was satisfied that there was sufficient evidence to demonstrate the safety, quality and efficacy of the vaccine”.

Is it possible to compare the efficacy of different vaccines?

It is difficult to compare efficacy among different vaccine candidates, even after completion of all Phase 3 trials and data publication. The protocols of the Phase 3 trials conducted to date have varied; with differing groups of volunteers (age, ethnic background, etc.), levels of circulating SARS-CoV-2 in the population, and methods for evaluating efficacy (such as the number of just symptomatic COVID-19 cases or the number of all cases, including asymptomatic ones).

Many experts have called for transparency of COVID-19 clinical trials, including protocol disclosure and results sharing. This would enable scrutiny and earn public trust. To allow direct comparisons of different vaccine candidates, the WHO drafted the COVID-19 vaccines Solidarity Trial in April. Some experts argue that human challenge models, where volunteers are infected with SARS-CoV-2 in a controlled manner, could be used to perform head-to-head comparison among candidates, speeding up vaccine development. On 20 October, the UK Government announced £33.6 million investment towards these models.

Efficacy of the COVID-19 vaccines approved in the UK

So far, three vaccines have been given regulatory approval in the UK by the Medicines & Healthcare products Regulatory Agency (MHRA). This process is described in detailed in Regulatory approval of COVID-19 vaccines in the UK.

The next section summarises the efficacy and safety data available for each vaccine from initial press releases and peer-reviewed analyses of Phase 3 clinical trials. Data are also synthesised in the table below.

Pfizer/BioNTech vaccine

Initial results for the Pfizer/BioNTech vaccine were announced in press releases and on 10 December 2020, the results from Phase III clinical trials were published.

Participants in the study: 43,448 participants were recruited across the United States, Argentina, Brazil, South Africa, Germany and Turkey. 21,720 of them received two doses of the vaccine and 21,728 a placebo. Safety analysis included 37,706 participants (50.6% were male and 49.4% female; 18,860 part of the placebo group and 18,846 of the vaccine group), who were followed up for at least two months. Vaccine efficacy was evaluated in 36,523 participants.

Most participants (82.9%) were from a White ethnic background, while 9.3% were Black or African-American and 4.3% Asian. A majority of participants (57.8%) were aged between 16–64 years. 21% suffered from other diseases, including chronic lung disease (7.8% of the total), diabetes (8.4%), liver disease (0.6%) and HIV (0.3%). Pregnant women and children younger than 16 years were not included in the study.

Dosing regimen: 2 doses of the vaccine (or placebo) 21 days apart.

Efficacy against symptomatic confirmed COVID-19: Efficacy was evaluated at least 7 days after the second dose of vaccine. Only eight symptomatic COVID-19 cases were seen in the vaccine group, compared with 162 in the placebo group. Therefore, the efficacy was evaluated as 95%. Vaccine efficacy was also evaluated for individual groups, including:

• 3% for participants at risk of severe COVID-19, regardless of age
• 2% for participants aged between 16–64 years, not in a risk group
• 4% for obese participants, regardless of age
• 7–100% for individuals aged over 65 years, regardless of their risk for severe COVID-19. This confirms Phase 1 published data suggesting that the vaccine was able to trigger an immune response in older adults.

Efficacy against asymptomatic cases: Not assessed in this study, but preliminary evidence is emerging from the vaccine roll-out (see below).

Adverse events: Adverse events related to the injections were reported more often in the vaccine group than the placebo group (21% and 5%, respectively). These included 64 vaccine recipients (0.3%) and six placebo recipients (<0.1%) who had swollen lymph nodes. There were four serious adverse events in the vaccine group: one case of shoulder injury, one of abnormal heart rhythm, one case where armpit lymph nodes changed size and one case of abnormal sensation in the skin of one leg.

Further research

On 7 January 2021 a preprint study (not peer reviewed) suggested that the Pfizer/BioNTech vaccine could produce an immune response against the N501Y mutation, although some experts warn that the results do not include all the mutations present in the SARS-CoV-2 UK variant.

Approval and pharmacovigilance

On 2 December 2020, the MHRA authorised the Pfizer/BioNTech vaccine for use in the UK. The vaccine was approved by the EMA for use in the EU on 21 December.

Pharmacovigilance: Cases of severe allergic reaction (including two in the UK) were reported during the initial roll-out of the Pfizer/BioNtech vaccine. These are probably due to nanoparticles, a component of the vaccine acting as a ‘packaging system’ for the RNA.

On 8 January 2021, the WHO published its recommendations for use of the Pfizer/BioNTech vaccine, advising against vaccination of people with severe allergic reaction to any component. This is in line with the recommendations from Public Health England COVID-19: Green Book for vaccinations.

On 29 January 2021 the EMA published its first vaccine safety update of the Pfizer/BioNtech vaccine. Based on pharmacovigilance data collected from the CDC during the initial phases of the USA roll-out, the frequency of anaphylaxis was estimated to be 11 cases per million doses. The EMA also investigated some safety concerns that arose from the deaths of older people in Norway following vaccination and concluded that these events were not linked to the vaccine.

Recent data from the COVID Symptom Study collected from 40,000 people in the UK who received the Pfizer/BioNTech vaccine show that more than a third reported pain and swelling in the arm 1–2 days following the first injection. Among the 12,444 people who already had received the second injection, one in 10 experienced fatigue or headache, and around one in 20 experienced chills or shivers 1–2 days following the second dose.

On 5 February 2021, the MHRA published its first report on the safety of COVID-19 vaccines. This included the first data collected through the Yellow Card Scheme, an MHRA programme to report suspected side effects and adverse events related to COVID-19 medicines and vaccines. 16,756 Yellow Cards were reported for the Pfizer/BioNTEch vaccine; equivalent to a side effect rate of 3 in 1000 people. Most of the reported events included sore arms, headaches, chills and fatigue.

Moderna vaccine

Initial results for the Moderna vaccine were announced in press releases and on 30 December 2020 the company published its efficacy and safety data. A one-page research summary is available here.

Participants in the study: The study included 30,420 participants (52.7% males and 47.3% females). Most participants (79.2%) were from a White ethnic background, while 10.2% were Black or African-American and 4.6% Asian. Most participants (75.3%) were aged between 18–64 years. 5,065 (16.7% of the total) were individuals at risk of severe COVID-19.

Individuals at risk of severe COVID-19 included: chronic lung disease (4.8% of the total), significant cardiac disease (4.9%), severe obesity (6.7%), diabetes (9.5%), liver disease (0.6%) and HIV (0.6%). Pregnant women and children were not included in the study.

Dosing regimen: 2 doses of the vaccine (or placebo) 28 days apart.

Efficacy against symptomatic confirmed COVID-19: Participants were monitored for up to 4 months following the first injection. Starting from 14 days after the second injection, there were 185 symptomatic confirmed COVID-19 cases in the placebo group and 11 in the vaccine. Vaccine efficacy was therefore evaluated as 94.1%. Severe COVID-19 disease occurred in 30 people, all in the placebo group. Vaccine efficacy was also evaluated for different groups:

• 4% for participants aged 65 years and over
• 4% for participants aged between 18–64 years at risk of severe COVID-19
• 5% for participants from ethnic minority backgrounds

Symptomatic confirmed COVID-19 cases were also evaluated in the time window from 14 days after the first injection to the second injection (the same way that single dose efficacy has been evaluated for other vaccines): 35 cases were found in the placebo group, compared with two in the vaccinated group, indicating a comparable vaccine efficacy for that 2-week period.

Efficacy against asymptomatic cases: Protection against asymptomatic infection and transmission was not investigated in the study. However, initial observations while screening with PCR tests of participants coming for the second dose identified 39 asymptomatic cases amongst the placebo group and 15 cases in the vaccinated group.

Adverse events: Adverse events related to the injections were reported more often in the vaccine group than the placebo group (8.2% and 4.5%, respectively).Severe adverse events were more frequent in the vaccine group (71 participants, equivalent to 0.5% of the group) compared with the placebo group (28 participants, equivalent to 0.2% of this group).

Further research

On 25 January 2021, a pre-print study (not peer reviewed) showed that the antibodies produced by the Moderna vaccine were able to neutralise new SARS-CoV-2 variants (including the UK-detected B.1.1.7 and South Africa-detected B.1.351, see SARS-CoV-2 virus variants: a year into the COVID-19 pandemic ), although neutralisation levels were reduced with B.1.351.

Approval and pharmacovigilance

On 6 January 2021, EMA recommended the Moderna vaccine for authorisation in the EU. On 8 January 2021, the vaccine was given regulatory approval in the UK by the MHRA. On 26 January 2021, the WHO published its interim recommendations for use of the Moderna vaccine in people aged 18 years and older.

As of 5 February 2021, pharmacovigilance reports for the Moderna vaccine are not yet available.

University of Oxford/AstraZeneca vaccine

On 8 December 2020, data on safety and efficacy of the Oxford/AstraZeneca vaccine were published.

Participants in the study: 23,848 participants were enrolled in trials in the UK, Brazil and South Africa. Data from 11,636 participants (7548 in the UK, 4088 in Brazil) were presented in this interim analysis. 5807 people received two doses of the vaccine and 5829 were included in the control group. The majority of participants were aged 18–55 years (86.7% in the UK and 89.9% in Brazil), female (60.5%) and White (91.4% in the UK and 66.6% in Brazil). Some participants suffered from chronic diseases such as cardiovascular disease (7–13% across groups), respiratory disease (10–13%) and diabetes (1–3%).

Dosing regimen: Two dosing regimens were tested:

1. two ‘standard doses’ of vaccine in 4440 participants
2. an initial ‘low dose’ followed by a ‘standard dose’. This was tested in 1367 participants

The control groups either received the meningococcal vaccine or a placebo. Timings of the second dose varied between 4 and 12 weeks.

Efficacy against symptomatic confirmed COVID-19: Efficacy was evaluated by identifying the number of confirmed symptomatic COVID-19 cases more than 14 days after the second dose. The study analysed data available up to 4 November 2020.

• For the standard dosing regimen, efficacy was evaluated as 62.1% (27 cases among the 4440 participants who received the vaccine compared with 71 out of 4455 in the control group).
• Efficacy was instead 90.0% in the alternative dosing regimen, with 3 cases out of 1367 in the vaccinated group vs. 30 out of 1374 in the control.

Combining results from these two regimens (30 cases out of 5807 in the vaccinated groups vs. 101 out of 5829 in the control groups) produced an overall estimated vaccine efficacy of 70.4%. When assessing symptomatic confirmed cases more than 21 days after their first dose, 10 participants were hospitalised due to COVID-19, all in the control group.

Although vaccine efficacy in older age groups could not be assessed with the data available, results from phase 2/3 trials show similar immune responses in young and older adults vaccinated with two doses of the vaccine.

Efficacy against asymptomatic cases: Participants in the UK cohort were also tested regularly to monitor asymptomatic infections:

• for the standard dosing regimen, asymptomatic cases were 22/2168 in the vaccinated group vs. 23/2223 in the control group.
• for the alternative dosing regimen, 7/1120 cases were found in the vaccinated group vs. 17/1127 in the control group. These initial results may suggest efficacy against asymptomatic infection of 58·9%

Adverse events: Adverse events were evaluated in 12,174 participants in the vaccine group and 11,879 in the control group. Serious adverse events were seen in 168 participants (79 from the vaccine group and 89 from the control). These included:

• a case of haemolytic anaemia in the control group 10 days after receiving the meningococcal vaccine (control group)
• a case of transverse myelitis reported 14 days after the second dose of the COVID-19 vaccine
• a case of fever higher than 40^oC reported 2 days after injection in a participant whose group allocation remained unknown at the time of the analysis (see double-blind study).

Further research

On 1 February 2021 a follow-up pre-print (not peer reviewed) reported further analysis to clarify the interim results.

Participants in the study: The pre-print included data from 17,177 participants from three clinical trials across the UK (8948 participants), Brazil (6753) and South Africa (1476). 8597 participants received a vaccine and 8580 a placebo.

Dosing regimens: The pre-print again presents data from the two different dosing regimens described above.

Efficacy against symptomatic confirmed COVID-19: The pre-print analysed symptomatic confirmed COVID-19 cases at different time points. Confirmed COVID-19 cases are defined as cases with a positive PCR test combined with at least one COVID-19 symptom (fever ≥37.8^oC, cough, shortness of breath, or loss of smell or taste). Efficacy was evaluated at the following time points:

• more than 14 days after the second dose, evaluated in the different dosing regimes
• more than 21 days after the first dose, to evaluate the effect of a single dose
• across groups receiving the booster dose at different time points, to evaluate the effect of delays in efficacy

1. For the standard dose: 74 cases were found among the vaccinated group, while 197 in the control group. Efficacy was estimated as 63.1%.
2. In the alternative dosing regimen: 10 cases were found among the vaccinated group, while 51 in the control group. Efficacy was estimated to be 80.7%.

The two regimens combined suggest an efficacy of 66.7%

Efficacy against asymptomatic cases: the pre-print increased the data available on participants in the UK cohort who were also tested regularly to monitor asymptomatic infections.

1. For the standard dose: 41 asymptomatic cases were found among the vaccinated group, while there were 42 in the control group. There was no evidence of protection.
2. In the alternative dosing regimen: 16 asymptomatic cases were found among the vaccinated group, while 31 were found in the control group. Efficacy was estimated to be 49.3%.

The overall reduction in the infection rate in the two groups (asymptomatic and symptomatic, evaluated as PCR positive results) suggested 54.1% efficacy in reducing transmission.

Efficacy of a single dose: for a single dose, efficacy to prevent symptomatic COVID-19 cases in the time window between 22 to 90 days after the injection was estimated to be 76%. An overall reduction in the infection rate (both asymptomatic and symptomatic PCR positive cases) suggested a 67% efficacy of a single dose in reducing transmission.

Efficacy at different intervals: Vaccine efficacy increased with a longer delay for the booster dose (54.9% at 6 weeks compared to 82.4% at more than 12 weeks). This was correlated with higher antibody levels.

On 4 February, another pre-print (not peer reviewed) confirmed reduction of viral load in vaccinated participants, corroborating the evidence in favour of an effect of the Oxford/AstraZeneca vaccine on transmission. In addition, the study evaluated vaccine efficacy against symptomatic infections caused by the B.1.1.7 variant (74.6%), suggesting that it is comparable with the earlier variants of the virus (84%)

Approval and pharmacovigilance

On 30 December 2020, the University of Oxford/AstraZeneca vaccine was authorised by the MHRA. The first vaccinations started on 4 January 2021.On 29 January 2021 the EMA recommended the vaccine for authorisation in the EU.

On 5 February 2021, the MHRA published its safety data on the Oxford/AstraZeneca vaccine. 6,014 Yellow Cards were reported for this vaccine, equivalent to a side effect rate of 3 in 1000 people. Similarly to the Pfizer/BioNTech vaccine, most of the reported events included sore arms, headaches, chills and fatigue.

Table summarising results currently available. From The Vaccine Knowledge Project  & Baden et al. (2021). *Range in brackets show difference between the first dose and the second dose.

What are the current immunisation strategies in the UK?

On 30 December 2020, JCVI published its updated priority groups These are the following:

1. residents in a care home for older adults and their carers
2. all those 80 years of age and over and frontline health and social care workers
3. all those 75 years of age and over
4. all those 70 years of age and over and clinically extremely vulnerable individuals
5. all those 65 years of age and over
6. all individuals aged 16 years to 64 years with underlying health conditions which put them at higher risk of serious disease and mortality
7. all those 60 years of age and over
8. all those 55 years of age and over
9. all those 50 years of age and over

The current immunisation strategy aims to prevent COVID-19 mortality and protect health and social care staff and systems. This is achieved by administering the vaccine first to those who have the highest risk of dying of severe COVID-19 disease. In its ‘COVID-19 vaccines delivery plan’, the UK Government prioritised offering the first vaccination dose to all the first four priority groups (~15m people across the UK) by 15 February 2021.

The vaccines currently available in the UK to be administered in people are the Pfizer/BioNTech and the University of Oxford/AstraZeneca vaccines. Supply from Moderna will begin in March 2021.

JCVI advises a two-dose schedule for the vaccines currently available in the UK, with a 3 to 12 week delay for the Pfizer/BioNTech vaccine and a 4 to 12 week delay for the University of Oxford/AstraZeneca one. JCVI does not advise combining different vaccines. In the case of incomplete vaccination of at-risk individuals (or individuals unlikely to come back to the vaccine clinic), the Public Health England COVID-19: Green Book for vaccinations recommends to give any vaccine as a booster. While this has not yet been tested in clinical trials, it is based on the fact that both vaccines target the same part of the virus to trigger an immune response.

Preliminary results from animal models show that combining different vaccines (the candidate developed by Imperial College London with the University of Oxford/AstraZeneca one) triggers a stronger immune response than a single dose. This supports the PHE assumptions in the Green book. On 4 February, the UK government launched the first clinical trial worldwide to test the use of different vaccines for the first and second dose. The National Immunisation Schedule Evaluation Consortium (NISEC) will run the trial, also evaluating the effectiveness of different dose intervals.

How soon will we be able to see the vaccine’s benefits?

On 7 January 2021, SAGE concluded that the effect of the vaccination programme on mortality rates will be likely observed later in February. Modelling data from the COVID-19 Actuaries Response Group analysed the effect of meeting the Government’s targets (i.e. vaccinating all the first four priority groups by 15 February) and concluded that a decrease in hospitalisation will be observed starting from mid-February and reported deaths will decrease in early March.

When will any gaps in understanding be filled? How might immunisation strategies change in response?

A COVID-19 vaccine surveillance strategy has been set out by the Government to monitor vaccine safety, effectiveness and infection levels in the UK population. Part of this strategy includes enhanced surveillance of COVID-19 cases in vaccinated individuals. This will allow better understanding of the vaccines’ role in the prevention of severe COVID-19 disease, as well as SARS-CoV-2 transmission.

International data can also help fill any gaps in our current understanding. A recent pre-print analysed the effectiveness of the Pfizer/BioNTech vaccine in Israel on 503,875 individuals and suggested around 50% effectiveness in preventing transmission (evaluated as the number of positive cases between 13 to 24 days after immunisation compared with those during the first 12 days). Preliminary data found a 33% reduction in transmission in people over 60 years of age who received a single dose of the vaccine. Other preliminary analyses support the effectiveness of a single dose of vaccine in preventing hospitalisation of older adults.

More peer-reviewed publications and pharmacovigilance databases will contribute to this shared information between different countries and, if needed, refine vaccination strategies by revisiting dosing regimens or priority groups.