Abstract

The evaluation of vaccine effectiveness is conducted with real-world data. They are essential to monitor the performance of vaccination programmes over time, and in the context of the emergence of new variants. Until now, the effectiveness of COVID-19 vaccines has been assessed based on classic methods, such as cohort and test-negative case-control studies, which may often not allow for adequate control of inherent biases in the assignment of vaccination campaigns. The aim of this review was to discuss the study designs available to evaluate vaccine effectiveness, highlighting quasi-experimental studies, which seek to mimic randomized trials, by introducing an exogenous component to allocate to treatment, in addition to the advantages, limitations, and applicability in the context of Brazilian data. The use of quasi-experimental approaches, such as interrupted time series, difference-in-differences, propensity scores, instrumental variables, and regression discontinuity design, are relevant due to the possibility of providing more accurate estimates of COVID-19 vaccine effectiveness. This is especially important in scenarios such as the Brazilian, which characterized by the use of various vaccines, with the respective numbers and intervals between doses, applied to different age groups, and introduced at different times during the pandemic.

Key words:
Coronavirus; Causal inference; Immunization

Introduction

During development and licensing, vaccines go through a series of stages to evaluate their safety and efficacy. The use of a new vaccine becomes possible following its approval by regulatory bodies. The regulatory decisions required to approve their use in the population focus on the balance between the risk and benefit, expressed in safety and efficacy measures. However, they do not capture all of the information required to guarantee their continued use in public health¹1 Clemens J, Brenner R, Rao M, Tafari N, Lowe C. Evaluating new vaccines for developing countries: Efficacy or effectiveness? J Am Med Assoc 1996; 275:390-397..

Efficacy data derives from phase III pre-licensing studies, usually randomized clinical trials, and provide the measure of a proportional reduction of the risk of infection, or disease, in the vaccinated group, compared to one that has usually received a placebo. Although randomized clinical trials have a strong internal validity, and provide robust evidence of the direct biological effects of the vaccine on an individual level, they cannot be generalized to a population-level vaccination programme²2 Saadatian-Elahi M, Horstick O, Breiman RF, Gessner BD, Gubler DJ, Louis J, Parashar UD, Tapia R, Picot V, Zinsou JA, Nelson CB. Beyond efficacy: The full public health impact of vaccines. Vaccine 2016; 34:1139-1147..

From the time when the vaccine starts to be used in the population, its effectiveness needs to be evaluated from two main objectives: verifying if the levels of efficacy registered in the phase III studies were maintained in the real world, and continuing to monitor its safety, which involves several thousand people and, therefore, may enable rare, but severe, events to be confirmed. Thus, “efficacy” is usually defined as the performance of an intervention in ideal, controlled circumstances, while “effectiveness” refers to its performance under conditions of use in the real world³3 Halloran ME, Haber M, Longini Jr IM, Struchiner C. Direct and Indirect Effects in Vaccine Efficacy and Effectiveness. Am J Epidemiol 1991; 133:323-331..

In the case of COVID-19, we have a scenario in which various highly effective vaccines were developed in a short space of time, and have been applied to millions of people. On the other hand, the virus has been demonstrating the capacity to mutate quickly, which has affected some of its characteristics, such as transmissibility. The possibility of mutations that affect the vaccine`s capacity for protection is possible, although this has not been significantly documented. A further important issue is the duration of the effect of the vaccine, i.e., the effectiveness may reduce in just a few months⁴4 Hitchings MDT, Ranzani OT, Torres MSS, Oliveira SB, Almiron M, Said R, Borg R, Schulz WL, Oliveira RD, Silva PV, Castro DB, Sampaio VS, Albuquerque BC, Ramos TCA, Fraxe SHH, Costa CF, Naveca FG, Siqueira AM, Araújo WN, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of CoronaVac among healthcare workers in the setting of high SARS-CoV-2 Gamma variant transmission in Manaus, Brazil: A test-negative case-control study. Lancet Reg Heal - Am 2021; 9:100025.. In addition, there is the fact that the various vaccines are technologically different, having varying effects on the immunological system and, consequently, their efficacy⁵5 Cerqueira-Silva T, Oliveira V de A, Pescarini JM, Bertoldo Júnior J, Machado TM, Flores-Ortiz R, Penna G, Ichihara MY, Barros JV, Boaventura VS, Barreto ML, Werneck GL, Barral-Netto M. Influence of age on the effectiveness and duration of protection in Vaxzevria and CoronaVac vaccines. medRxiv 2021.. Lastly, the lack of clear correlates of protection for COVID-19 should be highlighted (i.e., usually measurable antibodies which may serve markers of individual protection), although if found, may be different for specific vaccines.⁶6 Krammer F. COVID-19 vaccination and immune thrombocytopenia. Nat Med 2021; 27:1145-1153. These immunological correlates of protection enable individuals, or the fraction of the population which is adequately protected, to be quickly identified through the use of surveys on population samples, with biological sample tests. In their absence, or complementary to these, we have effectiveness studies, which are population-based studies, either purely observational or quasi-experimental, and aim to estimate the effect of the vaccine (effectiveness) during its use in the real world.

In this context, vaccine effectiveness (VE) studies should include appropriate methods to mimic the process of allocating intervention, seeking the most adequate balance between those who did, or did not, receive the intervention. The methods may be adaptations of classic observational studies (i.e., time-series, case-control and cohorts), or those called quasi-experimental. The quasi-experimental methods, also called `natural experiments`, are resources which are less used and disseminated, but that have a great potential of providing adequate and efficient solutions to solve the problem of imbalance on allocating the intervention among treated and control groups, and producing valid VE estimates.

Therefore, this manuscript has the aim of providing an overview of the methods used when assessing VE, with a greater emphasis on quasi-experimental studies. Also, we aim to emphazise the ecosystem of existing and accessible epidemiological data, to discuss the potential and limitations for conducting VE studies in Brazil.

Methods to evaluate vaccine efficacy and effectiveness

Halloran et al.³3 Halloran ME, Haber M, Longini Jr IM, Struchiner C. Direct and Indirect Effects in Vaccine Efficacy and Effectiveness. Am J Epidemiol 1991; 133:323-331. define four types of effects which may be considered in studies to evaluate vaccine effectiveness. An examination of these effects involves considering not only administration of the vaccine to individuals, but also the context of vaccination programmes, and the different levels of vaccination coverage in populations in particular. Schematically, considering two extreme alternatives, in which in one there is a population with a vaccination programme under development and, in the other there is no vaccination programme at population-level, the following effects of the vaccines may be defined:

Randomized clinical, cohort, and case-control studies are mainly used to examine the direct effects of a vaccine, while community-based intervention, or cluster randomized trials, can be used to evaluate the overall, total, or indirect effects. More recently, quasi-experimental methods have been used to evaluate the effects of vaccination programmes. They may also be used to evaluate indirect and total effects if there is adequate data on the vaccination status of individuals within a cohort of those eligible for a certain programe⁷7 Halloran ME, Struchiner C, Longini Jr IM. Study Designs for Rvauating Different Efficacy and Effectiveness Aspects of Vaccines. Am J Epidemiol 1997; 146: 789-803..

Observational epidemiological methods and their application in the evaluation of COVID-19 vaccine effectiveness

Case-control (test-negative) studies

In the last twenty years, a modification in the traditional case-control study design, called the “test-negative case-control” has been widely used in observational studies to evaluate the post-licensing effectiveness of influenza vaccines⁸8 Sofia Arriola C, El Omeiri N, Azziz-Baumgartner E, Thompson MG, Sotomayor-Proschle V, Fasce RA, Horoch MV, Olalla JEC, Almeida WAF de, Palacios J, Palekar R, Couto P, Descalzo M, Ropero-Álvarez AM. Influenza vaccine effectiveness against hospitalizations in children and older adults - Data from South America, 2013-2017. A test negative design. Vaccine X 2019; 3:1-8.^,99 Castilla J, Martínez-Artola V, Salcedo E, Martínez-Baz I, Cenoz MG, Guevara M, Alvarez N, Irisarri F, Morán J, Barricarte A. Vaccine effectiveness in preventing influenza hospitalizations in Navarre, Spain, 2010-2011: Cohort and case-control study. Vaccine 2012; 30:195-200.. More recently, the World Health Organization (WHO) proposed application of this study to evaluate COVID-19 vaccine effectiveness¹⁰10 World Health Organization (WHO). Evaluation of COVID-19 Vaccine Effectiveness. Geneva: WHO; 2021..

In COVID-19 studies, a case is defined as a patient who seeks health care and tests positive for SARS-CoV-2 infection through a reverse transcription polymerase chain reaction (RT-PCR) and/or antigen test. The control is a patient who followed the same case process, but receives a negative result for SARS-CoV-2 infection. VE is calculated in a similar way to traditional case-control studies; in other words, VE = (1 - the ratio of chance of vaccination among cases and controls) × 100%⁴4 Hitchings MDT, Ranzani OT, Torres MSS, Oliveira SB, Almiron M, Said R, Borg R, Schulz WL, Oliveira RD, Silva PV, Castro DB, Sampaio VS, Albuquerque BC, Ramos TCA, Fraxe SHH, Costa CF, Naveca FG, Siqueira AM, Araújo WN, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of CoronaVac among healthcare workers in the setting of high SARS-CoV-2 Gamma variant transmission in Manaus, Brazil: A test-negative case-control study. Lancet Reg Heal - Am 2021; 9:100025.^,1111 Ranzani OT, Hitchings M, Dorion Nieto M, D'Agostini TL, Paula RC de, Paula OF de, Villela EF de M, Torres MSS, Oliveira SB de, Schulz W, Almiron M, Said R, Oliveira RD de, Silva PV da, Araújo WN de, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of the CoronaVac vaccine in the elderly population during a P.1 variant-associated epidemic of COVID-19 in Brazil: A test-negative case-control study. BMJ 2021; 374:1-43..

Ranzani et al.¹¹11 Ranzani OT, Hitchings M, Dorion Nieto M, D'Agostini TL, Paula RC de, Paula OF de, Villela EF de M, Torres MSS, Oliveira SB de, Schulz W, Almiron M, Said R, Oliveira RD de, Silva PV da, Araújo WN de, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of the CoronaVac vaccine in the elderly population during a P.1 variant-associated epidemic of COVID-19 in Brazil: A test-negative case-control study. BMJ 2021; 374:1-43. conducted a test-negative case-control study in the elderly aged ≥70 from the State of São Paulo who received the CoronaVac vaccine. Vaccine effectiveness for symptomatic infection, adjusted for age and comorbidities, was 18.2% (CI 95% 0.0 to 33.2) and 41.6% (CI 95% 26.9 to 53.3) in a 0 to 13 and ≥14 day period, respectively, following the 2^nd dose¹¹11 Ranzani OT, Hitchings M, Dorion Nieto M, D'Agostini TL, Paula RC de, Paula OF de, Villela EF de M, Torres MSS, Oliveira SB de, Schulz W, Almiron M, Said R, Oliveira RD de, Silva PV da, Araújo WN de, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of the CoronaVac vaccine in the elderly population during a P.1 variant-associated epidemic of COVID-19 in Brazil: A test-negative case-control study. BMJ 2021; 374:1-43.. Hitchings et al. (2021)⁴4 Hitchings MDT, Ranzani OT, Torres MSS, Oliveira SB, Almiron M, Said R, Borg R, Schulz WL, Oliveira RD, Silva PV, Castro DB, Sampaio VS, Albuquerque BC, Ramos TCA, Fraxe SHH, Costa CF, Naveca FG, Siqueira AM, Araújo WN, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of CoronaVac among healthcare workers in the setting of high SARS-CoV-2 Gamma variant transmission in Manaus, Brazil: A test-negative case-control study. Lancet Reg Heal - Am 2021; 9:100025. also evaluated the effectiveness of the CoronaVac in health professionals in Manaus, State of Amazonas, Brazil. Vaccination with a minimum of one dose presented effectiveness, adjusted by other variables, of 49.6% (CI 95% 11.3 to 71.4) for symptomatic SARS-CoV-2 infection in the ≥14 day period following the first dose. However, low effectiveness was confirmed (36.8%; CI 95% 54.9 to 74.2) in the ≥14 day period following the second dose⁴4 Hitchings MDT, Ranzani OT, Torres MSS, Oliveira SB, Almiron M, Said R, Borg R, Schulz WL, Oliveira RD, Silva PV, Castro DB, Sampaio VS, Albuquerque BC, Ramos TCA, Fraxe SHH, Costa CF, Naveca FG, Siqueira AM, Araújo WN, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of CoronaVac among healthcare workers in the setting of high SARS-CoV-2 Gamma variant transmission in Manaus, Brazil: A test-negative case-control study. Lancet Reg Heal - Am 2021; 9:100025.. In the United Kingdom, Bernal et al.¹²12 Lopez Bernal J, Andrews N, Gower C, Gallagher E, Simmons R, Thelwall S, Stowe J, Tessier E, Groves N, Dabrera G, Myers R, Campbell CNJ, Amirthalingam G, Edmunds M, Zambon M, Brown KE, Hopkins S, Chand M, Ramsay M. Effectiveness of Covid-19 Vaccines against the B.1.617.2 (Delta) Variant. N Engl J Med 2021; 385:585-594. evaluated the effectiveness of Comirnaty and Vaxzevria vaccines with a test-negative case-control study. The results were similar for both vaccines, with over 70% effectiveness for the Alpha, and over 65% for the Delta variant¹²12 Lopez Bernal J, Andrews N, Gower C, Gallagher E, Simmons R, Thelwall S, Stowe J, Tessier E, Groves N, Dabrera G, Myers R, Campbell CNJ, Amirthalingam G, Edmunds M, Zambon M, Brown KE, Hopkins S, Chand M, Ramsay M. Effectiveness of Covid-19 Vaccines against the B.1.617.2 (Delta) Variant. N Engl J Med 2021; 385:585-594..

Cohort studies

Cohort studies have characteristics similar to those of clinical trials, except that they do not involve manipulating interventions. The longitudinal structure allows observation of the time sequence of events, and outcome exposure, which facilitates the causal inference process, and direct calculation of incidence and mortality measures.

A prospective cohort study was conducted on a national level in Chile to evaluate CoronaVac vaccine effectiveness, including approximately 80% of the population.¹³13 Jara A, Undurraga EA, González C, Paredes F, Fontecilla T, Jara G, Pizarro A, Acevedo J, Leo K, Leon F, Sans C, Leighton P, Suárez P, García-Escorza H, Araos R. Effectiveness of an Inactivated SARS-CoV-2 Vaccine in Chile. N Engl J Med 2021; 385:875-884. Using an extension of the Cox proportional-hazards model, considering vaccination status as a dependent time variable, vaccination effectiveness was estimated, associated with partial immunization (≥14 days following receipt of the first dose, and before receiving the second dose), and full immunization (≥14 days following receipt of the second dose). Among the fully immunized people, the adjusted effectiveness of the vaccine was 65.9% for SARS-CoV-2 infection, 87.5% for hospitalisation, 90.3% for ICU admission, and 86.3% for death. The results were maintained in the subgroup analyses by age, mainly among people of an age equal or superior to 60¹³13 Jara A, Undurraga EA, González C, Paredes F, Fontecilla T, Jara G, Pizarro A, Acevedo J, Leo K, Leon F, Sans C, Leighton P, Suárez P, García-Escorza H, Araos R. Effectiveness of an Inactivated SARS-CoV-2 Vaccine in Chile. N Engl J Med 2021; 385:875-884..

The SIREN study is based on a prospective multicentric cohort of public hospital employees in the United Kingdom, immunized with the BNT162b2 vaccine (Comirnaty)¹⁴14 Hall V, Foulkes S, Saei A, Andrews N, Oguti B, Charlett A, Wellington E, Stowe J, Gilson N, Atti A, Islam J, Karagiannis I, Munro K, Khawan J, Chand MA, Brown CS, Ramsay M, Lopez-Bernal J, Hopkins S. Effectiveness of BNT162b2 mRNA vaccine against infection and COVID-19 vaccine coverage in healthcare workers in England, multicentre prospective cohort study (the SIREN Study). SSRN Electron J 2021.. Risk factors, vaccination status, and symptoms, were registered at two week intervals, in addition to all of the RT-PCR and SARS-CoV-2 antibody test results. A Poisson mixed-effects proportional hazard model was used to calculate hazard ratios, to compare the time to infection in unvaccinated and vaccinated participants, and thereby estimate the impact of the vaccine on symptomatic and asymptomatic infections. Vaccination coverage in the period studied was 89%. Significantly lower coverage was associated with previous infection (OR 0.59; CI 95% 0.54-0.64), women (OR 0.72, CI 95% 0.63-0.82), those aged under 35, belonging to minority ethnic groups (especially black people) (OR 0.26, CI 95% 0.21-0.32), porters/security guards (OR 0.61, CI 95% 0.42-0.90), or midwives (OR 0.74, CI 95% 0.57-0.97), and employees living in more vulnerable neighbourhoods (OR 0.75, CI 95% 0.65-0.87)¹⁴14 Hall V, Foulkes S, Saei A, Andrews N, Oguti B, Charlett A, Wellington E, Stowe J, Gilson N, Atti A, Islam J, Karagiannis I, Munro K, Khawan J, Chand MA, Brown CS, Ramsay M, Lopez-Bernal J, Hopkins S. Effectiveness of BNT162b2 mRNA vaccine against infection and COVID-19 vaccine coverage in healthcare workers in England, multicentre prospective cohort study (the SIREN Study). SSRN Electron J 2021.. Vaccine effectiveness was 72% (CI 95% 58-86) 21 days following the first dose, and 86% (CI 95% 76-97) seven days after two doses¹⁴14 Hall V, Foulkes S, Saei A, Andrews N, Oguti B, Charlett A, Wellington E, Stowe J, Gilson N, Atti A, Islam J, Karagiannis I, Munro K, Khawan J, Chand MA, Brown CS, Ramsay M, Lopez-Bernal J, Hopkins S. Effectiveness of BNT162b2 mRNA vaccine against infection and COVID-19 vaccine coverage in healthcare workers in England, multicentre prospective cohort study (the SIREN Study). SSRN Electron J 2021..

A prospective cohort study conducted in Scotland¹⁵15 Vasileiou E, Simpson CR, Robertson C, Shi T, Kerr S, Agrawal U, Akbari A, Bedston S, Beggs J, Bradley D, Chuter A, Lusignan S de, Docherty A, Ford D, Hobbs R, Mark J, Katikireddi SV, Marple J, McCowan C, McGagh D, McMenamin J, Moore E, Murray JLK, Pan J, Ritchie L, Shah SA, Stock S, Torabi F, Tsang RSM, Wood R, Woolhouse M, Sheikh A. Effectiveness of First Dose of COVID-19 Vaccines Against Hospital Admissions in Scotland: National Prospective Cohort Study of 5.4 Million People. SSRN Electron J 2021., with a hospitalisation and mortality database of 5.4 million people, estimated the effectiveness of the first doses of Pfizer-BioNTech (Cominarty) and Oxford-AstraZeneca (Vaxzevria) vaccines against COVID-19 related hospital admissions. The time-dependent Cox model and Poisson regression models were used for this. The first dose of the BNT162b2 (Cominarty) vaccine was associated with 85% (CI 95%; 76-91) vaccination effectiveness to prevent COVID-19-related hospitalisation, 28-34 days after vaccination¹⁵15 Vasileiou E, Simpson CR, Robertson C, Shi T, Kerr S, Agrawal U, Akbari A, Bedston S, Beggs J, Bradley D, Chuter A, Lusignan S de, Docherty A, Ford D, Hobbs R, Mark J, Katikireddi SV, Marple J, McCowan C, McGagh D, McMenamin J, Moore E, Murray JLK, Pan J, Ritchie L, Shah SA, Stock S, Torabi F, Tsang RSM, Wood R, Woolhouse M, Sheikh A. Effectiveness of First Dose of COVID-19 Vaccines Against Hospital Admissions in Scotland: National Prospective Cohort Study of 5.4 Million People. SSRN Electron J 2021.. Vaccine effectiveness in the same time interval for the ChAdOx1 (Vaxzevria) vaccine was 94% (CI 95%; 73-99). The results of combined vaccine effectiveness to prevent COVID-19-related hospitalisation were comparable, by restricting the analysis to those at an age equal to or above 80 (81%; CI 95% 65-90), 28-34 days following vaccination)¹⁵15 Vasileiou E, Simpson CR, Robertson C, Shi T, Kerr S, Agrawal U, Akbari A, Bedston S, Beggs J, Bradley D, Chuter A, Lusignan S de, Docherty A, Ford D, Hobbs R, Mark J, Katikireddi SV, Marple J, McCowan C, McGagh D, McMenamin J, Moore E, Murray JLK, Pan J, Ritchie L, Shah SA, Stock S, Torabi F, Tsang RSM, Wood R, Woolhouse M, Sheikh A. Effectiveness of First Dose of COVID-19 Vaccines Against Hospital Admissions in Scotland: National Prospective Cohort Study of 5.4 Million People. SSRN Electron J 2021..

A longitudinal study was conducted in Brazil, to evaluate the effectiveness of CoronaVac and Vaxzevria in four different outcomes: SARS-CoV-2 virus infection, hospitalisation, ICU admission, and death. Due to the lack of data on the unvaccinated population, the period between the date of the first dose and the 13^th day was used as a reference to estimate VE. Full vaccination (14 days after the 2^nd dose) with Vaxzevria or CoronaVac displayed 78% and 53% vaccination effectiveness against SARS-CoV-2 infection, respectively. VE against hospitalisation, ICU admission, and death, was 91.4% (CI 95% 90.1-92.5), 91.1 (CI 95% 88.9-92.9) and 92.3% (CI 95% 90.5-93.7) respectively for Vaxzevria, and 71.2% (70.0-72.4), 72.2% (70.2-74.0) and 73.7% (72.1-75.2) for CoronaVac. The lower protection found for older adults is also highlighted, especially those over the age of 80, particularly with CoronaVac⁵5 Cerqueira-Silva T, Oliveira V de A, Pescarini JM, Bertoldo Júnior J, Machado TM, Flores-Ortiz R, Penna G, Ichihara MY, Barros JV, Boaventura VS, Barreto ML, Werneck GL, Barral-Netto M. Influence of age on the effectiveness and duration of protection in Vaxzevria and CoronaVac vaccines. medRxiv 2021..

Quasi-experimental methods and their applications in evaluating vaccine effectiveness

Observational studies (such as cohort, or case-control studies), which compare individuals who have been vaccinated with those who have not, can be used to evaluate the direct effect of the vaccines. However, individuals who have been vaccinated may systematically differ from those who have not, and it is often difficult, or impossible, to separate the effects of these differences from those related to the vaccine. On the other hand, quasi-experimental studies, while they are also fundamentally observational, since they stem from records of interventions conducted outside of the researcher`s control, however seek to mimic experimental studies by exogenous attribution to eligibility of a treatment, in this case, the vaccine, thereby avoiding endogenous sources of bias¹⁶16 Lopez Bernal JA, Andrews N, Amirthalingam G. The Use of Quasi-experimental Designs for Vaccine Evaluation. Clin Infect Dis 2019; 68:1769-1776..

There is a wide range of scenarios in which quasi-experimental methods can be used to evaluate vaccines¹⁶16 Lopez Bernal JA, Andrews N, Amirthalingam G. The Use of Quasi-experimental Designs for Vaccine Evaluation. Clin Infect Dis 2019; 68:1769-1776.. Firstly, when a vaccine has already been implemented, and, therefore, the use of randomized trials is no longer a viable option, or in circumstances where it would be unethical not to provide a vaccine to a specific group (control), such as during a pandemic. Among the quasi-experimental methods, we can cite the following: interrupted time series, difference-in-differences, propensity scores, instrumental variables, and regression discontinuity.

Interrupted time series

Interrupted Time Series (ITS) are indicated to evaluate a large-scale intervention applied at population-level, and with a well-defined start date,¹⁷17 Linden A. Conducting interrupted time-series analysis for single- and multiple-group comparisons. Stata J 2015; 15:480-500.^,1818 Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: A tutorial. Int J Epidemiol 2016; 0:1-8. where multiple pre- and post-intervention observations are used to examine a change in the tendency of the outcome following the intervention. ITS studies have been increasingly frequent to evaluate the effect of public health interventions, highlighting the impact of rotavirus¹⁹19 Hungerford D, Vivancos R, Read JM, Iturriza-Gómara M, French N, Cunliffe NA. Rotavirus vaccine impact and socioeconomic deprivation: An interrupted time-series analysis of gastrointestinal disease outcomes across primary and secondary care in the UK. BMC Med 2018; 16:1-14. and pneumonia²⁰20 Silaba M, Ooko M, Bottomley C, Sande J, Benamore R, Park K, Ignas J, Maitland K, Mturi N, Makumi A, Otiende M, Kagwanja S, Safari S, Ochola V, Bwanaali T, Bauni E, Gleeson F, Knoll MD, Adetifa I, Marsh K, Williams TN, Kamau T, Sharif S, Levine OS, Hammitt LL, Scott JAG. Effect of 10-valent pneumococcal conjugate vaccine on the incidence of radiologically-confirmed pneumonia and clinically-defined pneumonia in Kenyan children: an interrupted time-series analysis. Lancet Glob Heal 2019; 7:e337-e346. vaccination programmes.

ITS studies can be constructed using two approaches: single and multiple groups. In single group studies, there is no comparison group, and the effect can be estimated from the change in the pre- and post-intervention tendency, so that the pre-intervention tendency is counterfactual. However, in multiple group studies, there is at least one comparison group, and the change in the intergroup outcome tendency (i.e., in the pre- and post-intervention period), and between the groups (comparison and control group) can be evaluated¹⁷17 Linden A. Conducting interrupted time-series analysis for single- and multiple-group comparisons. Stata J 2015; 15:480-500.^,2121 Handley MA, Lyles CR, McCulloch C, Cattamanchi A. Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research. Annu Rev Public Health 2018; 39:5-25..

A minimum of three variables is required to analyse ITS regression: the time (t) elapsed since the start of the study with the unit that represents the frequency with which the observations are made; the variable that indicates the pre-intervention (Xt) period; and the outcome in each period of time (Yt)¹⁸18 Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: A tutorial. Int J Epidemiol 2016; 0:1-8.^,2121 Handley MA, Lyles CR, McCulloch C, Cattamanchi A. Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research. Annu Rev Public Health 2018; 39:5-25..

In order to prepare an interrupted time series, a number of assumptions should be fulfilled. Initially, that there is a clear differentiation between the pre- and post-intervention period needs to be confirmed, requiring a timeframe for the start of the intervention. These studies require a sequential measure of the outcomes before and after the intervention, they can be binary, continuous, or counts, and are indicated when the change occurs in a short space of time. In addition, sequential measures of the outcome over time should be similarly distributed both before and after the intervention¹⁸18 Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: A tutorial. Int J Epidemiol 2016; 0:1-8..

ITS has already been used in Brazil to evaluate rotavirus²²22 Carmo GMI Do, Yen C, Cortes J, Siqueira AA, Oliveira WK de, Cortez-Escalante JJ, Lopman B, Flannery B, Oliveira LH de, Carmo EH, Patel M. Decline in diarrhea mortality and admissions after routine childhood rotavirus immunization in Brazil: A time-series analysis. PLoS Med 2011; 8:11. and pneumococcal conjugate (PCV10)²³23 Andrade AL, Minamisava R, Policena G, Cristo EB, Domingues CMS, Brandileone MC de C, Almeida SCG, Toscano CM, Bierrenbach AL. Evaluating the impact of PCV-10 on invasive pneumococcal disease in Brazil: A time-series analysis. Hum Vaccines Immunother 2016; 12:285-292. vaccine effectiveness. However, no data is available on the use of ITS in COVID-19 vaccine effectiveness until this time.

ITS studies are useful when randomization is impossible, allowing a large-scale longitudinal evaluation. In addition, the possibility of working with aggregated population data contributes towards greater external validity¹⁶16 Lopez Bernal JA, Andrews N, Amirthalingam G. The Use of Quasi-experimental Designs for Vaccine Evaluation. Clin Infect Dis 2019; 68:1769-1776.. Quick and convenient access to routinely collected health data on database, and use of ITS, enables the production of information on vaccination effectiveness in real time¹⁸18 Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: A tutorial. Int J Epidemiol 2016; 0:1-8.. A further important point to be raised is the presentation of graphic and numerical results, which are easily understood by health professionals and managers, and may assist with decision-making, and the reallocation of resources.

On the other hand, there are limitations. The analysis may be subject to residual confounders, due to the unavailability of important covariables not registered on the databases used. Variables which change quickly in time need to be identified, and treated through multivariate models, since they may bias the association found¹⁸18 Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: A tutorial. Int J Epidemiol 2016; 0:1-8.. Seasonality and the unequal distribution of units of time before and after the intervention are also factors that need to be taken into consideration¹⁸18 Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: A tutorial. Int J Epidemiol 2016; 0:1-8..

Difference-in-differences

The difference-in-differences (DD) method has been used to evaluate the impact, considering interventions that vary in time²⁴24 Greene SK, Kulldorff M, Lewis EM, Li R, Yin R, Weintraub ES, Fireman BH, Lieu TA, Nordin JD, Glanz JM, Baxter R, Jacobsen SJ, Broder KR, Lee GM. Near real-time surveillance for influenza vaccine safety: Proof-of-concept in the Vaccine Safety Datalink Project. Am J Epidemiol 2009; 171:177-188.

25 Uddin MJ, Shamsuzzaman M, Horng L, Labrique A, Vasudevan L, Zeller K, Chowdhury M, Larson CP, Bishai D, Alam N. Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine 2016; 34:276-283.^-2626 Raymond S, Li L, Taioli E, Nash D, Liu B. The effect of the Affordable Care Act dependent coverage provision on HPV vaccine uptake in young adult women, National Health and Nutrition Examination Survey 2007-2016. Prev Med (Baltim) 2021; 148:1-8., due to its relative convenience, and wide use in evaluations of public health interventions²⁷27 Wing C, Simon K, Bello-Gomez RA. Designing Difference in Difference Studies: Best Practices for Public Health Policy Research. Annu Rev Public Health 2018; 39:453-469.. In order to measure the effect, DD considers that the treatment group estimate (counterfactual) is equal to the pre-intervention value of the treatment group, added to the control group post/pre-difference²⁸28 Fredriksson A, Oliveira GM de. Impact evaluation using Difference-in-Differences. RAUSP Manag J 2019; 54:519-532.. Although DD identifies the average effect in those treated, its meaning and identification conditions differ between data types²⁸28 Fredriksson A, Oliveira GM de. Impact evaluation using Difference-in-Differences. RAUSP Manag J 2019; 54:519-532..

Use of DD depends on supporting premises. Firstly, the value of a stable treatment unit and premises of “parallel tendencies”²⁸28 Fredriksson A, Oliveira GM de. Impact evaluation using Difference-in-Differences. RAUSP Manag J 2019; 54:519-532.

29 Athey S, Imbens GW. Identification and Inference in Nonlinear Difference-in-Differences Models. Econometrica 2006; 74:431-497.^-3030 Lee M jae, Kang C. Identification for difference in differences with cross-section and panel data. Econ Lett 2006; 92:270-276.; in other words, there should not be a spillover effect among the treatment and control groups, since the treatment effect will not be identified. Additionally, the control variables at an individual and/or aggregated level should be exogenous, and unaffected by the treatment. A typical approach is the use of covariables prior to the intervention²⁸28 Fredriksson A, Oliveira GM de. Impact evaluation using Difference-in-Differences. RAUSP Manag J 2019; 54:519-532.^,2929 Athey S, Imbens GW. Identification and Inference in Nonlinear Difference-in-Differences Models. Econometrica 2006; 74:431-497..

Raymond et al.²⁶26 Raymond S, Li L, Taioli E, Nash D, Liu B. The effect of the Affordable Care Act dependent coverage provision on HPV vaccine uptake in young adult women, National Health and Nutrition Examination Survey 2007-2016. Prev Med (Baltim) 2021; 148:1-8. used DD methodology to evaluate the effect of the provision of human papillomavirus (HPV) vaccination coverage in children, on infection by the virus in young women. The effect was estimated using linear probability regressions, adjusted for race/ethnicity, age, income, head of the family`s level of education, and family employment²⁶26 Raymond S, Li L, Taioli E, Nash D, Liu B. The effect of the Affordable Care Act dependent coverage provision on HPV vaccine uptake in young adult women, National Health and Nutrition Examination Survey 2007-2016. Prev Med (Baltim) 2021; 148:1-8.. To the best of our knowledge, no reports on the use of DD to evaluate COVID-19 vaccine effectiveness are available.

All of the DD analyses should carefully consider any possible violations of assumptions, many of which appear to be probable, due to the COVID-19 dynamic, such as record quality, the non- linearity that arises from person-to-person transmission, and the probability that control policies at population-level have different effects over time³¹31 Goodman-bacon A, Marcus J. Using Difference-in-Differences to Identify Causal Effects of COVID-19 Policies. Surv Res Methods 2020; 14:153-158.. By including control groups, DD provides important advantages over methods such as before and after ITS comparisons.³¹31 Goodman-bacon A, Marcus J. Using Difference-in-Differences to Identify Causal Effects of COVID-19 Policies. Surv Res Methods 2020; 14:153-158. In addition, the graphic and parametric tools developed for DD in recent years allow an evaluation of the plausibility of assumptions.

Propensity score

The propensity score (PS) is the probability of allocating treatment, conditional to the confounding variables observed. This score could be used to control confounding, making treatment groups more similar in relation to the confounding variables observed³²32 Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011; 46:399-424.. In VE evaluation studies, the confounding variables that require control are, for example, age, sex, occupational profile, health conditions, confidence, and availability of the vaccine in specific contexts, among other aspects. Thus, this information may be used to estimate the probability of an individual being vaccinated.

The PS is commonly estimated using logistics regression, being the status of treatment predicted by the set of confounding variables observed³²32 Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011; 46:399-424.. The selection of independent variables that will be included in the PS estimation model is an important step in the use of this methodology. It is recommended that both exposure-related variables and the outcome being investigated (confounding variables) are included³²32 Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011; 46:399-424.^,3333 Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: A systematic review and suggestions for improvement. J Thorac Cardiovasc Surg 2007; 134:1128-1135..

Following score estimation, they can be used in the analysis to control the confounding variables observed in four main forms: (i) model adjustment, where the PS is included as an adjustment variable in a regression model that relates the outcome and exposure of interest; (ii) analysis stratification, where the PS is used to divide the study population into groups, or lower, more similar strata in relation to the confounding variables observed; (iii) participant matching, which involves matching an exposed with an unexposed individual who has a similar PS; and (iv) model weighting, where PSs are used to calculate the statistical weights for each individual, so that when considering the study population, the exposed and unexposed groups become similar in relation to the confounding variables observed, allowing an impartial estimate of the relation between the exposure and outcome.³²32 Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011; 46:399-424.^,3333 Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: A systematic review and suggestions for improvement. J Thorac Cardiovasc Surg 2007; 134:1128-1135. The last two have been highlighted in literature in recent years, especially in studies that evaluate VE.

The main assumptions to estimate the causal effect in PS based methods include (i) ignorability, which means that no confounders are observed³⁴34 Thoemmes F, Ong AD. A Primer on Inverse Probability of Treatment Weighting and Marginal Structural Models. Emerg Adulthood 2016; 4:40-59.; (ii) positivity, which means that every individual should have a probability other than zero to receive any treatment³⁴34 Thoemmes F, Ong AD. A Primer on Inverse Probability of Treatment Weighting and Marginal Structural Models. Emerg Adulthood 2016; 4:40-59.; (iii) the correct specification of the model used to calculate the PS, considering that its calculation is based on measured characteristics (variables). If unmeasured factors influence the selection of treatment, the resultant PS will not remove all of the bias from the confounding variables. In some cases, the residual bias could remain, and incorrect specifications could increase the bias.³⁴34 Thoemmes F, Ong AD. A Primer on Inverse Probability of Treatment Weighting and Marginal Structural Models. Emerg Adulthood 2016; 4:40-59.

The use of PS has been widely implemented in observational studies, particularly to evaluate the effectiveness of various medicines, highlighting VE studies for influenza³⁵35 Coles C, Zell E, Chen W-J, Richardson A, Arnold J, Fairchok M, Schofield C, Clair KJS, Danaher P, Rajnik M, Mcdonough E, Mor D, Ridore M, Burgess T, Millar E, Jackson HM. Estimates of Influenza Vaccine Effectiveness Among Highly Immunized United States Military Members andBeneficiaries: Utilization of Propensity Score Matching. Open Forum Infect Dis 2015; 2:S149.

36 Nation ML, Moss R, Spittal MJ, Kotsimbos T, Kelly PM, Cheng AC. Influenza vaccine effectiveness against influenza-related mortality in Australian hospitalized patients: a propensity score analysis. Clin Infect Dis 2021; 72:99-107.^-3737 Walzer P, Estève C, Barben J, Menu D, Cuenot C, Manckoundia P, Putot A. Impact of influenza vaccination on mortality in the oldest old: A propensity score-matched cohort study. Vaccines 2020; 8:1-9. and rotavirus.³⁸38 Hungerford D, Vivancos R, Read JM, Bonnett LJ, Bar-Zeev N, Iturriza-Gómara M, Cunliffe NA, Frech N. Mitigating bias in observational vaccine effectiveness studies using simulated comparator populations: Application to rotavirus vaccination in the UK. Vaccine 2018; 36:6674-6682. Therefore, use of PS could be an important methodological alternative to evaluate COVID-19 vaccine effectiveness. However, since the method controls exclusively for the confounding variables observed, it requires a significant number of covariables that are associated both with the vaccination (e.g., distribution, vaccination coverage, prioritization criteria - such as age, comorbidities, profession, and vulnerable populations, among others), for the outcome of interest, whether infection, hospitalisation, or death by COVID-19.

Until this time, the use of PS has only been identified in one study, which paired individuals receiving a minimum of one dose of any of the vaccines with unvaccinated individuals, according to demographic data, location (post code), and the number of previous PCR SARS-CoV-2 tests.³⁹39 Pawlowski C, Lenehan P, Puranik A, Agarwal V, Venkatakrishnan AJ. FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system. Med (NY) 2021 13(8):979-992. The authors highlighted that the administration of two doses of the COVID-19 vaccine (mRNA-1273 [Moderna], or BNT162b2 [Pfizer/BioNTech]), was 88.7% effective in preventing SARS-CoV-2 (CI 95%: 68.4-97.1%) infection. In addition, the administration of a minimum of one dose of the vaccines reduced hospital admission rates by 14 days for patients who were subsequently diagnosed with COVID-19 following immunization, in relation to those who were not vaccinated (3.7% vs. 9.2%; relative risk: 0.4; p-value: 0.007).³⁹39 Pawlowski C, Lenehan P, Puranik A, Agarwal V, Venkatakrishnan AJ. FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system. Med (NY) 2021 13(8):979-992. Therefore, it is emphasised that evaluation of the effectiveness of COVID-19 vaccines using the PS method is limited in literature.

Instrumental variables

Instrumental variables have been used increasingly as a strategy to control confounding in non-randomized study designs. Unlike the propensity score, instrumental variables use an exogenous form of variation, or “instrument”. The instrument should: (1) be the cause or “proxy” for exposure; (2) only be outcome related through exposure; and (3) not be associated with any confounding variable not measured in the study⁴⁰40 Cousens S, Hargreaves J, Bonell C, Armstrong B, Thomas J, Kirkwood BR, Hayes R. Alternatives to randomisation in the evaluation of public-health interventions: Statistical analysis and causal inference. J Epidemiol Community Health 2009; 65:576-581..

In instrumental variable analysis, the effect of the intervention on the outcome of interest is measured by comparing the magnitude of the association between the instrument and outcome with that between the instrument and intervention exposure⁴⁰40 Cousens S, Hargreaves J, Bonell C, Armstrong B, Thomas J, Kirkwood BR, Hayes R. Alternatives to randomisation in the evaluation of public-health interventions: Statistical analysis and causal inference. J Epidemiol Community Health 2009; 65:576-581.. A sequence of two regression analyses is generally used, which depends on the nature of the exposure and the outcome. Selection of an instrument is a fundamental part of the design, and is not simple. Even if the three above assumptions are met, the association between the instrument and intervention exposure should be strong. Inadequate instruments which establish a weak association with the intervention may produce inflated estimates, with broad confidence intervals⁴⁰40 Cousens S, Hargreaves J, Bonell C, Armstrong B, Thomas J, Kirkwood BR, Hayes R. Alternatives to randomisation in the evaluation of public-health interventions: Statistical analysis and causal inference. J Epidemiol Community Health 2009; 65:576-581..

The use of instrumental variables has not been widely used in the context of evaluating VE. We highlight the study by Wong et al.⁴¹41 Wong K, Campitelli MA, Stukel TA, Kwong JC. Estimating influenza vaccine effectiveness in community-dwelling elderly patients using the instrumental variable analysis method. Arch Intern Med 2012; 172:484-491., which evaluated the effectiveness of the influenza vaccine in the hospitalisation and death of elderly patients, using influenza vaccination coverage among people aged 65 or over as the instrument. A study by Chakrabarti et al.⁴²42 Chakrabarti A, Grépin KA, Helleringer S. The impact of supplementary immunization activities on routine vaccination coverage: An instrumental variable analysis in five low-income countries. PLoS One 2019; 14:1-11., also evaluated the effect of supplementary immunization activities in routine health service vaccination coverage. The authors evaluated if the child had obtained the main routine vaccination, or not, using the child`s age in the first supplementary immunization campaign as the instrument.⁴²42 Chakrabarti A, Grépin KA, Helleringer S. The impact of supplementary immunization activities on routine vaccination coverage: An instrumental variable analysis in five low-income countries. PLoS One 2019; 14:1-11. Use of methodologies using instrumental variables to evaluate COVID-19 vaccination effectiveness has not been documented in literature until this time.

When applied to evaluating COVID-19 vaccine effectiveness as instrumental variables, they could have great analytical power, given the difficulty in obtaining individualized data that contains clinical information and comorbidities, strongly associated with hospitalisation and mortality caused by the disease. Exogenous sources of variation, which could be used as instruments include age, the location of residence (municipality or state), and vaccination coverage, as conducted in the study by Wang et al.⁴¹41 Wong K, Campitelli MA, Stukel TA, Kwong JC. Estimating influenza vaccine effectiveness in community-dwelling elderly patients using the instrumental variable analysis method. Arch Intern Med 2012; 172:484-491..

The advantage of this method is due to the fact that the selection of a good instrument allows a non-biased estimate of association between intervention and outcome, without the need to measure all the confounding variables. This becomes crucial when conducting studies with secondary bases, where the data has already been collected. However, the main methodological limitation related to the instrumental variable approach, is the difficulty in finding good instruments, which may reduce the capacity to detect small effects, even in studies with large databases⁴³43 Staiger D, Stock JH. Instrumental Variables Regression with Weak Instruments. Econom Soc 1997; 65:557-586..

Regression discontinuity design

Regression discontinuity design (RDD) has been used increasingly in research in the area of epidemiology and public health.⁴⁴44 Bor J, Moscoe E, Mutevedzi P, Newell ML, Bärnighausen T. Regression discontinuity designs in epidemiology: Causal inference without randomized trials. Epidemiology 2014; 25:729-737.

45 Sekhon JS, Titiunik R. Understanding Regression Discontinuity Designs As Observational Studies. Obs Stud 2017; 3:174-182.

46 Moscoe E, Bor J, Bärnighausen T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: A review of current and best practice. J Clin Epidemiol 2015; 68:132-143.^-4747 Oldenburg CE, Moscoe E, Bärnighausen T. Regression Discontinuity for Causal Effect Estimation in Epidemiology. Curr Epidemiol Reports 2016; 3:233-241. It allows the effect of an intervention to be measured when the eligibility rules are based on a previously defined threshold (cut-off point). In other words, in regression discontinuity exposure to a treatment, or intervention, is determined by a continuous variable, such as the income or age of an individual, called the attribute variable, and the cut-off point for this variable defines intervention eligibility (or treatment). Generally speaking, RDD assumes that the intervention attribute (or treatment) in a small neighbourhood around the cut-off point is ignorable, and the potential response could be assumed independent of the attribute, which occurs in a randomized study.

Regression discontinuity design can be described in a deterministic or probabilistic way. For example, if all of the individuals above the cut-off point receive the vaccine, and nobody receives the vaccine under the cut-off point, then regression discontinuity is considered deterministic, known as sharp regression discontinuity. In contrast, if the probability of receiving the vaccine on one side of the cut-off point is greater in relation to the other, the design is said to be probabilistic, known as fuzzy regression discontinuity. Thus, in sharp RDD the intervention (or treatment) attribute rule perfectly determines the exposure, while in fuzzy RDD, the attribute rule causes a discontinuous change in the probability of exposure around the cut-off point, and this discontinuity is used to estimate the local causal effect of the change on the policy, or intervention, among the individuals who are around the cut-off point⁴⁶46 Moscoe E, Bor J, Bärnighausen T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: A review of current and best practice. J Clin Epidemiol 2015; 68:132-143.^,4848 Smith LM, Lévesque LE, Kaufman JS, Strumpf EC. Strategies for evaluating the assumptions of the regression discontinuity design: A case study using a human papillomavirus vaccination programme. Int J Epidemiol 2017; 46:939-949..

RDD has assumptions which can make its use inadequate, if they cannot be verified. The assumptions associated with discontinuous regression design are: 1) the existence of discontinuity in the probability of exposure; 2) non-manipulation of attribute variable values; 3) interchangeability; and 4) continuity in the probability of the response at the cut-off point. The first assumption evaluates if there is a discontinuous change in the probability of exposure around the cut-off point. The assumption of non-manipulation of the attribute variable suggests that the individuals in the study did not alter the information on this variable. Manipulation of this information would violate the assumption that the groups are attributed to the intervention (or treatment), which occurs in randomized studies. The assumption of interchangeability suggests that the exposure groups could be interchanged around the cut-off point. If this assumption is valid, it is expected that the individuals around the cut-off point are similar in relation to the distribution of all the baseline variables. The last assumption of RDD could be seen as an extension of the assumption of interchangeability, and suggests that any discontinuity in the probability of the response could solely be attributed to exposure. We highlight that these assumptions are considered weak, and can be confirmed through graphic analyses, or statistical tests⁴⁶46 Moscoe E, Bor J, Bärnighausen T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: A review of current and best practice. J Clin Epidemiol 2015; 68:132-143.^,4747 Oldenburg CE, Moscoe E, Bärnighausen T. Regression Discontinuity for Causal Effect Estimation in Epidemiology. Curr Epidemiol Reports 2016; 3:233-241..

For example, in studies that evaluate VE, the criteria (or eligibility rule) which makes an individual eligible to receive the vaccine, is usually age. Thus, individuals who are over a previously-defined age limit, are considered eligible for the intervention. Therefore, in this type of study, age would be our attribute variable. Use of discontinuous regression in studies on VE is still in the early stages, but we can cite some recent work which made use of this type of study design. For example, Frio and França⁴⁹49 Frio GS, França MTA. Human papillomavirus vaccine and risky sexual behavior: Regression discontinuity design evidence from Brazil. Econ Hum Biol 2021; 40:1-10. used fuzzy RDD to evaluate if the HPV vaccination affected the start of a sex life in girls in the age range close to the cut-off point (14 years old), which is defined by the public vaccination campaign⁴⁹49 Frio GS, França MTA. Human papillomavirus vaccine and risky sexual behavior: Regression discontinuity design evidence from Brazil. Econ Hum Biol 2021; 40:1-10.. The authors also investigated if the teenagers who had already started to have a sex life, stopped using condoms since they had been vaccinated⁴⁹49 Frio GS, França MTA. Human papillomavirus vaccine and risky sexual behavior: Regression discontinuity design evidence from Brazil. Econ Hum Biol 2021; 40:1-10.. The studies by Van Ourti and Bouckaert (2020)⁵⁰50 Van Ourti T, Bouckaert N. The Dutch influenza vaccination policy and medication use, outpatient visits, hospitalization and mortality at age 65. Eur J Public Health 2021; 30:275-280. and Anderson et al.⁵¹51 Anderson ML, Dobkin C, Gorry D. The effect of influenza vaccination for the elderly on hospitalization and mortality an observational study with a regression discontinuity design. Ann Intern Med 2020; 172: 445-452., used RDD to evaluate vaccine effectiveness against the influenza virus. In Van Ourti and Bouckaert`s⁵⁰50 Van Ourti T, Bouckaert N. The Dutch influenza vaccination policy and medication use, outpatient visits, hospitalization and mortality at age 65. Eur J Public Health 2021; 30:275-280. study, the authors estimated the impact of the Dutch vaccination programme on the use of medication, outpatient consultations, hospitalisation and mortality at the age of 65 (cut-off point), and concluded that there was an increase in the vaccination rate, but they did not find a relation with a possible reduction in hospitalisation, or mortality rates in the population analysed. In the study by Anderson et al.⁵¹51 Anderson ML, Dobkin C, Gorry D. The effect of influenza vaccination for the elderly on hospitalization and mortality an observational study with a regression discontinuity design. Ann Intern Med 2020; 172: 445-452., the authors evaluated vaccine effectiveness in the reduction of hospitalisations and mortality among older adults in the 65 year old age range (cut-off point), and concluded that there was an increase in vaccination rates, but they did not find a relation with a possible reduction in hospitalisation, or mortality rates, in the population analysed.

Application of RDD to evaluate the effectiveness of COVID-19 has been discussed⁵²52 Basta NE, Halloran ME. Evaluating the effectiveness of vaccines using a regression discontinuity design. Am J Epidemiol 2019; 188:987-990., but its application is still in the early stages⁵³53 Bermingham C, Morgan J, Ayoubkhani D, Glickman M, Islam N, Sheikh A, Sterne J, Walker AS, Nafilyan V. Estimating the effectiveness of first dose of COVID-19 vaccine against mortality in England: a quasi-experimental study. medRxiv 2021.07.12.21260385; doi: https://doi.org/10.1101/2021.07.12.21260385
https://doi.org/10.1101/2021.07.12.21260... ^,5454 Greene SK, Levin-Rector A, McGibbon E, Greene SK, Levin-Rector A, McGibbon E, Baumgartner J, Devinney K, Ternier A, Sell J, Kahn R, Kishore N. Reduced COVID-19 Hospitalizations among New York City Residents Following Age-Based SARS-CoV-2 Vaccine Eligibility: Evidence from a Regression Discontinuity Design. medRxiv 2021.06.30.21259491; DOI http://doi.org/10.1101/2021.06.30.21259491
http://doi.org/10.1101/2021.06.30.212594... . RDD was used in 2020 to evaluate if the BCG vaccination, applied to protect against tuberculosis, could reduce COVID-19 infection.⁵⁵55 Fukui M, Kawaguchi K, Matsuura H. Does TB Vaccination Reduce COVID-19 Infection? No Evidence from a Regression Discontinuity Analysis. medRxiv 2020.04.13.20064287; DOI: https://doi.org/10.1101/ 2020.04.13.20064287
https://doi.org/10.1101/ 2020.04.13.2006... However, an association between greater BCG vaccination coverage, and a lower chance of age-specific COVID-19 infection, was not found using data in the five countries applying this design⁵⁵55 Fukui M, Kawaguchi K, Matsuura H. Does TB Vaccination Reduce COVID-19 Infection? No Evidence from a Regression Discontinuity Analysis. medRxiv 2020.04.13.20064287; DOI: https://doi.org/10.1101/ 2020.04.13.20064287
https://doi.org/10.1101/ 2020.04.13.2006... . In England, effectiveness of the first dose of the COVID-19 vaccine was evaluated using age, defined as the priority criterion for the vaccine in this country, as a treatment attribute variable⁵³53 Bermingham C, Morgan J, Ayoubkhani D, Glickman M, Islam N, Sheikh A, Sterne J, Walker AS, Nafilyan V. Estimating the effectiveness of first dose of COVID-19 vaccine against mortality in England: a quasi-experimental study. medRxiv 2021.07.12.21260385; doi: https://doi.org/10.1101/2021.07.12.21260385
https://doi.org/10.1101/2021.07.12.21260... . Another study evaluated the effectiveness of the COVID-19 vaccination in hospitalisation rates for the disease in New York, also applying the age criterion as an attribute variable⁵⁴54 Greene SK, Levin-Rector A, McGibbon E, Greene SK, Levin-Rector A, McGibbon E, Baumgartner J, Devinney K, Ternier A, Sell J, Kahn R, Kishore N. Reduced COVID-19 Hospitalizations among New York City Residents Following Age-Based SARS-CoV-2 Vaccine Eligibility: Evidence from a Regression Discontinuity Design. medRxiv 2021.06.30.21259491; DOI http://doi.org/10.1101/2021.06.30.21259491
http://doi.org/10.1101/2021.06.30.212594... . However, we highlight that the rather broad age ranges may violate RDD assumptions, and should be verified.

Vaccines in use and with potential use in Brazil

The COVID-19 vaccines developed until this time (using data updated on 8th September, 2021) are based on four main technologies: 1) inactivated SARS-CoV-2 virus vaccines (CoronaVac); 2) recombinant vaccines which use adenovirus viral vectors, expressing the SARS-CoV-2 Spike protein (S) (the Janssen vaccine uses a human adenovirus, and Vaxzevria uses a chimpanzee adenovirus); 3) messenger RNA (mRNA) vaccines, which codify the SARS-CoV-2 (Comirnaty) S protein; and 4) protein subunit vaccines, which use SARS-CoV-2 S protein nanoparticles, or fragments of this protein (Novavax)⁵⁶56 Brasil. Ministério da Saúde (MS). Plano Nacional de Operacionalização da Vacina contra a Covid-19. Brasília: MS; 2021..

Chart 1 presents characteristics of the four vaccines authorised by the National Sanitary Surveillance Agency (Anvisa) for emergency or definitive use in the country until September 2021. They are, Vaxzevria (ChAdOx1-S; Oxford-AstraZeneca-Fiocruz, produced in partnership with the Serum Institute of India), CoronaVac (Butantan/Sinovac Biotech), Comirnaty (Pfizer/BioNTech/Wyeth), and Janssen-Cilag (Johnson & Johnson). They are all indicated for people aged 18 or over, except for Comirnaty, which is authorised for those over the age of 12, and have a two dose vaccination scheme, except for Janssen, which is applied in a single dose. The administration of Vaxzevria was temporarily interrupted for pregnant women and those who have recently given birth.

Until now, the simultaneous administration of COVID-19 vaccines with others on the national immunization calendar is not recommended, to improve monitoring of adverse events following vaccination, and due to a lack of knowledge on the possibility of antigenic competition.

The administration of additional doses, or interchangeability between different vaccines has been discussed, and may be recommended in specific situations (e.g., pregnant women and the immunosuppressed). Although attractive as a public health policy, a combination of vaccines awaits proof of efficacy and effectiveness. The Center for Disease Control Prevention (CDC/EUA) and Public Health England only advocate this in exceptional situations, or if there is a lack of vaccines. In Brazil, based on the possible association of viral vector vaccines with rare thrombotic events⁵⁷57 Brasil. Trata-se de atualização das referentes a vacinação contra a covid-19 em gestantes e puérperas até 45 dias pós-parto. Nota Técnica no 2/2021-SECOVID/GAB/SECOVID/MS. 1.1. Rio de Janeiro: Ministério da Saúde (MS); 2021., a combination is indicated for pregnant women who receive a first dose of Vaxzevria, with the second dose of Comirnaty or CoronaVac. For the same reason, a number of European countries, and Canada, have recommended the second dose with mRNA vaccines for people under the age of 55 who received their first dose with Vaxzevria⁵⁸58 Reuters. German experts say under 60s should not get second AstraZeneca vaccine. 2021. [cited 2021 Set 7]. Available from: https://www.reuters.com/article/us-health-coronavirus-germany-astrazenec-idUSKBN2 BO6KZ.
https://www.reuters.com/article/us-healt... ^,5959 Reuters. France recommends second mRNA COVID-19 dose for under- 55s who got first dose with Astra. 2021. [cited 2021 Set 7]. Available from: https://www.reuters.com/article/us-health-coronavirus-germany-astrazenec-idUSKBN2BO6KZ.
https://www.reuters.com/article/us-healt... . Considering the change in guidelines in these countries, data on combined VE will be available in the near future.

Existing and accessible data in Brazil - Limitations and possibilities

In a context of increasing vaccination, there is a growing need for rapid answers on crucial questions, in order to orientate vaccination programmes. Among these, we highlight (1) the effect of new variants of the virus on vaccine effectiveness; (2) the optimum interval between doses; (3) the effect of the vaccines on asymptomatic infection, against serious disease; (4) the decline in immunity over time, and (5) the need, or usefulness, of combining different vaccines, or booster doses. In this context, the use of data originating from large databases of data routinely collected by the health system, associated with appropriate methods of analysis, has been one of the solutions presented in various countries. For any chosen investigation, both classic observational studies, and the use of quasi-experimental designs, require high quality information, whether related to the vaccination status of each individual, health outcomes, or the covariables required.

At the start of the pandemic, Scotland built a large integrated data system⁶⁰60 Simpson CR, Robertson C, Vasileiou E, McMenamin J, Gunson R, Ritchie LD, Woolhouse M, Morrice L, Kelly D, Stagg HR, Marques D, Murray J, Sheikh A. Early Pandemic Evaluation and Enhanced Surveillance of COVID-19 (EAVE II): Protocol for an observational study using linked Scottish national data. BMJ Open 2020; 10:1-7., which has been widely used for epidemiological and COVID-19 clinical studies and is currently employed to evaluate BNT162b2 vaccine effectiveness⁶¹61 Vasileiou E, Simpson CR, Shi T, Kerr S, Agrawal U, Akbari A, Bedston S, Beggs J, Bradley D, Chuter A, Lusignan S de, Docherty AB, Ford D, Hobbs R, Joy M, Katikireddi SV, Marple J, McCowan C, McGagh D, McMenamin J, Moore E, Murray JLK, Pan J, Ritchie L, Shah AS, Stock S, Torabi F, Tsang RSM, Wood R, Woolhouse M, Robertson C, Sheikh A. Interim findings from first-dose mass COVID-19 vaccination roll-out and COVID-19 hospital admissions in Scotland: a national prospective cohort study. Lancet 2021; 397:1646-1657.. In Israel, a follow-up study used the databases of four health organizations, which cover more than half of the population, to combine health history information, RT-PCR test results, outpatient, hospitalisation and vaccination data, to estimate the effectiveness of the first and second doses of the BNT162b2 vaccine at protecting from infection, symptomatic COVID-19, hospitalisation, serious disease, and death.⁶²62 Haas EJ, Angulo FJ, McLaughlin JM, Anis E, Singer SR, Khan F, Brooks N, Smaja M, Mircus G, Pan K, Southern J, Swerdlow DL, Jodar L, Levy Y, Alroy-Preis S. Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data. Lancet 2021; 397:1819-1829. Outside of the circle of developed countries, Chile has presented a major study on the effectiveness of the main vaccine in use in the country (CoronaVac), also utilising data which is routinely collected on national databases¹³13 Jara A, Undurraga EA, González C, Paredes F, Fontecilla T, Jara G, Pizarro A, Acevedo J, Leo K, Leon F, Sans C, Leighton P, Suárez P, García-Escorza H, Araos R. Effectiveness of an Inactivated SARS-CoV-2 Vaccine in Chile. N Engl J Med 2021; 385:875-884.. In Brazil, two effectiveness studies based on routine data were conducted using the case-control approach. The first, of health professionals from Manaus⁴4 Hitchings MDT, Ranzani OT, Torres MSS, Oliveira SB, Almiron M, Said R, Borg R, Schulz WL, Oliveira RD, Silva PV, Castro DB, Sampaio VS, Albuquerque BC, Ramos TCA, Fraxe SHH, Costa CF, Naveca FG, Siqueira AM, Araújo WN, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of CoronaVac among healthcare workers in the setting of high SARS-CoV-2 Gamma variant transmission in Manaus, Brazil: A test-negative case-control study. Lancet Reg Heal - Am 2021; 9:100025., and the other with the elderly population in the state of São Paulo¹¹11 Ranzani OT, Hitchings M, Dorion Nieto M, D'Agostini TL, Paula RC de, Paula OF de, Villela EF de M, Torres MSS, Oliveira SB de, Schulz W, Almiron M, Said R, Oliveira RD de, Silva PV da, Araújo WN de, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of the CoronaVac vaccine in the elderly population during a P.1 variant-associated epidemic of COVID-19 in Brazil: A test-negative case-control study. BMJ 2021; 374:1-43., both evaluating the effectiveness of CoronaVac, or CoronaVac and Vaxzevria, throughout the country⁵5 Cerqueira-Silva T, Oliveira V de A, Pescarini JM, Bertoldo Júnior J, Machado TM, Flores-Ortiz R, Penna G, Ichihara MY, Barros JV, Boaventura VS, Barreto ML, Werneck GL, Barral-Netto M. Influence of age on the effectiveness and duration of protection in Vaxzevria and CoronaVac vaccines. medRxiv 2021..

In Brazil, a series of databases register and provide complete or partial data that is potentially useful to study COVID-19 vaccine effectiveness. Of specific interest are databases with individualized, identified, or pseudonymised data, allowing analyses that bring together different markers for exposure and outcomes, and relevant covariables. Unfortunately, we do not have a unified system with all of the relevant, required information, since it is dispersed throughout different records.

We will now provide a brief description of the databases available until this time. In the first group are databases which were created, or extended, to register COVID-19 related events, while the others are for regular use, and include important information for COVID-19 related studies. They all provide dictionaries and open data on the OpenDataSUS site.

Databases produced for COVID-19 related events

Regularly collected data that could be the source of COVID-19 outcomes

a) SUS Hospital Information System (SUS/HIS): Access to Hospitalisation Authorisations (HA) requested and authorised during the period of study are obtained through this system;

b) Mortality Information System (MIS): access to Death Certificates (DC) registered on the MIS on a monthly basis, independent of any review or criticism, registered during the period of study. The update could be monthly, through access to files in the DataSUS native dissemination format (.dbc), with the full details registered on the DC. This information is important to monitor the reduction in deaths as a result of vaccination. However, the registration of revised mortality experiences a delay of at least six months, which imposes a limitation on its use on evaluating vaccination effectiveness.

Data linkage

Within the context of the Brazilian health data ecosystem, the isolated use of the above-mentioned databases is, in general, inadequate for a valid assessment of vaccine effectiveness, since the relevant variables are dispersed throughout different databases. For example, we have a database that provides information on the vaccine doses of every individual, their age, and sex, among other details, but other databases are required to discover if this individual was infected, or became a COVID-19 clinical case. Therefore, linkage strategies for these different databases allow for the capture and incorporation of relevant information which is available on only one of the databases, or which have a varying quality of records and level of completeness between them. This process provides new possibilities for investigation with national databases⁶³63 Sanni Ali M, Ichihara MY, Lopes LC, Barbosa GCG, Pita R, Carreiro RP, Santos DB dos, Ramos D, Bispo N, Raynal F, Canuto V, Almeida B de A, Fiaccone RL, Barreto ME, Smeeth L, Barreto ML. Administrative data linkage in Brazil: Potentials for health technology assessment. Front Pharmacol 2019;10:1-20. and prospects for investigation into health, particularly in VE evaluation studies.

The objective of linking databases is finding records for the same individual on different databases, in order to combine the various items of information for each of them. In the presence of an unmistakable identification variable between the databases, the deterministic method is used, and records are related by comparing this variable on the different databases. In the absence of a variable of this nature, the probabilistic record linkage method is used, which involves estimating the probability of agreement and disagreement between the common variables on the databases for the record pairing process. While the deterministic method involves a single stage of comparison, the probabilistic method may involve a series of stages, such as standardisation, blocking, and matching.

Conclusions and Recommendations

In the complex and dynamic context that has characterized the development of this pandemic, vaccination effectiveness studies are gaining importance, as part of the resources which are able to produce evidence that subsidises the relevant decisions that are required to control the pandemic.

Although it is important to highlight the results of the efficacy of COVID-19 vaccines, understanding VE is a more complex task, particularly in a pandemic context with a reduced number of doses available, various types of vaccines, and the emergence and circulation of different strains of the SARS-CoV-2 virus. Classic observational studies, particularly test-negative case-control and longitudinal studies are approaches which have been used frequently, due to convenience, their designs are widely known, and analytical methods are standardized on various software programs. While quasi-experimental (or natural experiments) studies are strictly observational, they include a diversified series of designs, which seek to reduce the chance of biases introduced by non-randomized vaccination, since the decision to be vaccinated, or not, in the real world depends on various, non-random factors.

There are a wide range of scenarios in which quasi-experimental studies can be used for vaccine evaluation. The possibility of using data with exogenous attributes, primarily without the requirement of high quality data, avoids sources of bias, and makes quasi-experimental designs an excellent choice to evaluate COVID-19 vaccine effectiveness. Methods such as ITS, using aggregated data, PS, DD, Instrumental Variable and RDD on an individual level, may be used to evaluate vaccine effectiveness. However, their widespread use faces a series of limitations. For example, the main limitation of the PS is that it only controls measured variables, and is dependent on the availability of databases with covariables, which is not always the case. Instrumental variables may not necessarily be good analysis options, since the assumptions can easily be violated, given the difficulty of obtaining good instruments in a context of constant changes in eligibility criteria, and the rapid expansion of vaccination coverage.

The evaluation of COVID-19 vaccine effectiveness, although estimated by quasi-experimental methods, should also take other sources of variation into account, which may hinder the attainment of unbiased measures, such as 1) the introduction of different strains of SARS-CoV-2; 2) the administration of different vaccines and multiples immunization strategies (number, time interval between doses, and combination of vaccines); 3) stage of the epidemic in each location evaluated; and 4) difficulties with gathering records in different Brazilian municipalities.

In this review, we have presented the immense variety of options that exist for any researcher who is interested in following VE in populations, particularly COVID-19 vaccines. The ecosystem of data available is crucial for selecting the best evaluation strategies. It should be clarified that whichever option selected, there will be limitations that, possibly, may not be able to satisfy all of the assumptions. The researcher is responsible for selecting the analytical strategies that are best suited to the context, are the most robust possible, and always include sensitivity tests that support (or not) their findings.

Acknowledgments

We would like to thank the teams of Rede CoVida and Vigivac.

References

Publication Dates

History