ABSTRACT:

Objective:

To describe the methodological and operational aspects of an epidemiological and an evaluation of Rede Cegonha in Santa Catarina, Brazil.

Methods:

The research carried out in 2019 was composed of two sub-studies. Regarding the first, whose design was epidemiological and analyzed prenatal, delivery and immediate puerperium care addressed to pregnant women, puerperal women and children assisted at SUS, the instruments used for data collection and the organization of the field of the study are described. The other sub-study was a normative assessment of municipal management in prenatal and postpartum care within the scope of Rede Cegonha. It began with an evaluability assessment followed by the assessment itself. The different methodological strategies adopted are described, with the involvement of stakeholders and experts.

Results:

The response rate of the epidemiological sub-study was 97.7%. Women residing in 82.7% of Santa Catarina's municipalities were interviewed. The sample was similar to that registered in SINASC for the same period, and the characteristics of the sub-sample interviewed after six months was similar to the global sample. The evaluation study improved and applied a model with 32 indicators that allowed to analyze the municipalities considering the political-organizational and tactical-operational aspects. Two hundred and four municipalities answered the questionnaire (69.1%); they were evaluated according to their respective population size.

Conclusion:

The availability of methodological procedures of studies that articulate epidemiological and evaluation methods allows generating more accurate and complete information and contribute with the design and evaluation of health policies, programs and actions.

Keywords:
Epidemiology; Health evaluation; Health services research; Methods

INTRODUCTION

To improve the health and prevent mortality of women and infants are two of the main global objectives, which promote recurring discussions about the necessary and efficient measures to reach them¹1. Countdown to 2030 Collaboration. Countdown to 2030: tracking progress towards universal coverage for reproductive, maternal, newborn, and child health. Lancet 2018; 391(10129): 1538‐48. https://doi.org/10.1016/s0140-6736(18)30104-1
https://doi.org/https://doi.org/10.1016/... . Even if Brazil was able to reduce mortality rates in childhood and maternal death rates, the numbers are still high, which has mobilized different sectors of government and civil society to propose policies, programs and actions to face these public health issues²2. Brasil. Instituto de Pesquisa Econômica Aplicada e Secretaria de Planejamento e Investimentos Estratégicos. Objetivos de Desenvolvimento do Milênio: relatório nacional de acompanhamento. Brasília: IPEA; 2014.^,33. Leal MC, Szwarcwald CL, Almeida PVB, Aquino EML, Barreto ML, Barros F, et al. Saúde reprodutiva, materna, neonatal e infantil nos 30 anos do Sistema Único de Saúde (SUS). Ciênc Saúde Colet 2018; 23(6): 1915-28. https://doi.org/10.1590/1413-81232018236.03942018
https://doi.org/https://doi.org/10.1590/... .

Rede Cegonha was established in 2011, in Brazil, to contribute with the reduction of mother and child morbidity and mortality rates. Its objectives are to structure and organize the health services as a care network, presenting four strategic action components:

From the set of activities to be carried out to improve the health conditions of women and children, the high potential of work of Primary Health Care (PHC) teams stands out, in terms of reducing mother and child morbidity and mortality through qualified cared during pregnancy and postpartum⁵5. Espanha. Grupo de trabajo de Guía de práctica clínica de atención em el embarazo y puerperio. Guía de práctica clínica de atención em el embarazo y puerperio. Andalucía: Ministerio de Sanidad, Servicios Sociales e Igualdad, Agencia de Evaluación de Tecnologías Sanitarias de Andalucía; 2014.^,66. Viellas EF, Domingues RMSM, Dias MAB, Gama SGN, Theme Filha MM, Costa JV, et al. Assistência pré-natal no Brasil. Cad Saúde Pública 2014; 30(Supl.1): S85-S100. https://doi.org/10.1590/0102-311X00126013
https://doi.org/https://doi.org/10.1590/... ^,77. Correa MSM, Feliciano KVO, Pedrosa EM, Souza AI. Acolhimento no cuidado à saúde da mulher no puerpério. Cad Saúde Pública 2017; 33(3): e00136215. https://doi.org/10.1590/0102-311x00136215
https://doi.org/https://doi.org/10.1590/... . However, despite the increase in prenatal care coverage and in the number of appointments attended by pregnant women in Brazil³3. Leal MC, Szwarcwald CL, Almeida PVB, Aquino EML, Barreto ML, Barros F, et al. Saúde reprodutiva, materna, neonatal e infantil nos 30 anos do Sistema Único de Saúde (SUS). Ciênc Saúde Colet 2018; 23(6): 1915-28. https://doi.org/10.1590/1413-81232018236.03942018
https://doi.org/https://doi.org/10.1590/... ^,88. Mallmann MB, Boing AF, Tomasi YT, Anjos JC, Boing AC. Evolução das desigualdades socioeconômicas na realização de consultas de pré-natal entre parturientes brasileiras: análise do período 2000-2015. Epidemiol Serv Saúde 2018; 27(4): e2018022. https://doi.org/10.5123/s1679-49742018000400014
https://doi.org/https://doi.org/10.5123/... ^,99. Brasil. Sistema de Informações sobre Nascidos Vivos [Internet]. Brasil: Ministério da Saúde ; 2020 [acesso em 12 maio 2020]. Disponível em: Disponível em: http://tabnet.datasus.gov.br/cgi/deftohtm.exe?sinasc/cnv/nvuf.def
http://tabnet.datasus.gov.br/cgi/deftoht... , indicators such as the incidence of congenital syphilis and systemic arterial hypertension remain high, which leads to the questioning of care quality¹⁰10. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Atenção ao pré-natal de baixo risco. Brasília: Ministério da Saúde; 2012.^,1111. Pacagnella RC, Nakamura-Pereira M, Gomes-Sponholz F, Aguiar RALP, Guerra GVQL, Diniz CSG, et al. Maternal Mortality in Brazil: Proposals and Strategies for its Reduction. Rev Bras Ginecol Obstet 2018; 40(9): 501-6. https://doi.org/10.1055/s-0038-1672181
https://doi.org/https://doi.org/10.1055/... .

The elaboration and implementation of assessment instruments for the prenatal and postpartum care actions carried out by the cities can collaborate to identify weak and strong aspects in PHC. Likewise, empirical data of probability samples regarding care addressed to pregnant women during prenatal and postpartum care can subsidize the definition of public actions and policies for the improvement of care quality. Investigations with such a design are scarce in Brazil, especially epidemiological studies articulated with other assessment instruments, with a wide set of analyses and indicators of health services related to the health of women and children. Therefore, presenting and discussing methodological paths, challenges and solutions of studies that articulate tools and concepts of epidemiology and health assessment in investigations about the health of women and children can contribute with the dissemination of such analyses in the field.

This article aims at describing the methodological aspects of a study that associated tools from epidemiology and health assessment to carry out a diagnosis of prenatal and postpartum care in the state of Santa Catarina, with primary data from puerperal women and an evaluation of the municipal management in two components of Rede Cegonha.

METHODS

The study was divided in two complementary sub-studies and allied epidemiology and health assessment methods. The former collected primary data from puerperal women in Santa Catarina, aiming at identifying primary health care during prenatal and postpartum periods. The latter built and applied an assessment model of the municipal management in prenatal and postpartum care. The different methods of each investigation will be presented separately.

SUB-STUDY 1: DATA COLLECTION FROM PUERPERAL WOMEN

Reference population for the study

The study population was composed of puerperal women who:

The investigation included all hospitals in Santa Catarina which, in 2016, performed at least 50 deliveries through SUS, which resulted in the total of 31 hospitals distributed in 30 different cities. This set of institutions corresponded to 86.2% of all births in the State funded by SUS in 2016. The sampling calculation was performed with Stata 15. The sample included 3,665 omen, with 95% confidence interval, population of 50 thousand women, margin of error of 1.6 percentage points, estimated prevalence of 50 and 5% to recompose losses. The number of interviews in each hospital was in accordance with the proportional distribution of births observed in 2016.

Data collection instrument

The data collection instrument was built based on periodical meetings that included professors and post-graduate students. Since the study had a strong focus on assessing the state’s health services, the discussions to build the questionnaire also included different professionals from the State Secretariat of Health.

The final version of the questionnaire was composed of 365 questions, some from closed and validated instruments brought from other analyses with similar goals, and others designed by the surveyors according to the project’s objectives. This instrument was divided in twelve blocks:

The complete questionnaire was tested in 35 people to verify the understanding of each question and the structure of the entire instrument.

The interviewer’s guide presented detailed information about what each question aimed at measuring, the behavior the surveyor should present during questioning, the possible doubts that could be manifested by the interviewees, and the explanations to be given. It also contemplated ethical matters and instructions about the behavior to be adopted to handle the hospital staff, puerperal women and chaperones. Interviewers were trained to apply the poll in the hospital environment, with an intensive training dynamic, decentralized from the State.

Team of interviewers

Thirty-five interviewers were selected. They had at least incomplete higher education in the health field. Such a strategy allowed the selection of surveyors who were familiar with the subject and lived near the hospitals.

Data collection and consistency control

The questionnaire was programmed in the Research Electronic Data Capture (REDCap) Brazil (https://www.redcapbrasil.com.br/), installed in tablets. At the end of each day, the interviewers connected to a wi-fi network and transmitted data to the study server, from Universidade Federal de Santa Catarina. On a daily basis, the research coordination tested the consistency and quality of the data using a descriptive analysis. The results were sent to the regional coordinators, who then sent them to the interviewers and reinforced the relevant orientations. Then, a new visit to the hospital allowed the correction of inconsistencies in some hours or a few days.

The information about puerperal women were extracted from their pregnancy book records (when available) and hospital charts. The access to these three sources allowed to obtain the necessary data amplitude for the study and, in case of data repetition in several sources, enabled the comparison between the self-reported information and those reported in the documents. Data collection occurred between January and August, 2019.

Field logistics

The collection instrument was submitted to a pilot study in four hospitals from the study, with the equivalent to 5% of the sample. This stage allowed to verify the need for adjustments in the questionnaire, as well as to measure the time of application of the poll - which ranged between 40 and 60 minutes -, and allowed to test the dynamics of the record and transmission of data using REDCap to the central server.

Quality Control

The quality control of the data was performed weekly, in a random sample of 10% of the interviewees. The selected women were contacted by telephone by the regional supervisors, who applied a reduced questionnaire with eight questions, whose answers would not be subject to change between the two moments of the interview. With practice, the objective was to identify possible fraud (creation of non-existing interviews by an interviewer) and measure the reproducibility of measures. The questions included in quality control referred to type of labor, number of previous pregnancies, presence in a prenatal dental appointment, smoking during pregnancy, visit to the maternity hospital before labor, maximum schooling, use of ferrous sulphate during pregnancy and vaccination during prenatal care.

Second interview after six months

A second interview was performed six months after delivery, with the objective of verifying the service provided to women and children in this period, focusing on puerperium. For that, 975 women who participated in the first collection were interviewed again over the telephone. In this stage, the collection instrument was a structured questionnaire, with 41 questions divided in eight blocks: identification, child status, woman and child care in puerperium, breastfeeding and infant feeding, follow-up with the health team, reproductive planning and sexual education, behaviors and discrimination in the puerperium. This phase had the data programming dynamic used before, and seven interviewers participated in the collection. Before that stage, the poll team stored a wide variety of telephone contacts during the first wave, in order to minimize losses.

SUB-STUDY 2: EVALUATION OF PRIMARY CARE IN THE CITIES

This is a normative evaluation carried out in two stages, and the study object was municipal administration in prenatal and puerperal care in the scope of Rede Cegonha, in Santa Catarina.

The first stage consisted on an availability study¹²12. Centers for Disease Control and Prevention (CDC). Framework for Program Evaluation in Public Health. MMWR Recomm Rep 1999; 48(11): 1-40.^,1313. Thurston W, Ramaliu A. Evaluability assessment of a survivors of torture program: lessons learned. Can J Program Eval 2005; 20(2): 1-25.^,1414. Trevisan MS, Walser TM. Evaluability assessment. Estados Unidos: Sage; 2015. 181 p., which adopted the following as methodological strategies: document analysis, meetings, seminars and consensus conferences including stakeholders and experts in evaluation and primary health care.

Document analysis included the reading of technical materials that guide the implementation of Rede Cegonha at a federal level, available at the website of the Ministry of Health (Chart 1). The following step was the critical analysis of the materials, in order to improve the understanding about the object, emphasizing the aspects that compose mother and child care in primary health care in the municipal scope of health services, its goals, objectives, activities, products, results, impact and role in municipal administration.

Two meetings were held with the participation of PHC professionals (physicians and nurses), and representatives of the state and municipal health administrations (n = 10) to discuss the objectives, activities, products and expected results, identified in document analysis.

Based on the results of this analysis and meetings, assessment questions were defined, and the Theoretical Model (TM) and the Logical Model (LM) were elaborated. They consist of the schematic representation of the assessment object, increasing the understanding of the problem nature, the intervention and the relation between the object and the context. The preliminary proposals of TM and LM were the object of discussion and adjustments in an academic seminar promoted by a health assessment research group, with the participation of scholars in the themes of prenatal care, delivery and birth and puerperium, as well as the administration of primary health care.

Consensus conferences¹⁵15. Hartz ZMA, Vieira-da-Silva LM. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Rio de Janeiro: Fiocruz; 2005. 275 p. were performed including experts identified by their professional and academic work in the fields of epidemiology, planning and health assessment. They also included the stakeholders related to primary health care administration in SUS, at a state and municipal level¹⁵15. Hartz ZMA, Vieira-da-Silva LM. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Rio de Janeiro: Fiocruz; 2005. 275 p.. To establish consensus, the participants (n=14) received the preliminary proposal of the assessment model through electronic media with a 15-day deadline for analysis, and were able to answer whether or not they agreed totally, partially or disagreed of each one of the model components, justifying their choice. The proposed suggestions and changes in TM and LM were presented and debated in two on-site workshops, which lasted four hours each. Then, two on-site workshops were performed for the appreciation of the Analysis and Judgment Matrix (AJM), with the same duration and fewer participants (n=10).

The content of the on-site workshops was described and systematized by three surveyors who organized a new proposal based on the group decisions. The restructured assessment model was resent to the participants for revision and validation, and then the final consensus of the proposal was reached, with suggestions of strategies for its application.

The second stage of the evaluation consisted on the collection and analysis of data from the municipal administration, based on the products obtained in the availability study. The sources of evidence of the AJM elements were identified, initially by the analysis of the availability of information in research data bases and in official websites. It was observed that most of the information was available in secondary data sources for 2017, but there was the need for complementation with the collection of primary data from municipal administrators.

The next step was the elaboration and validation of the questionnaire sent to all municipal health administrators in Santa Catarina (n = 295) using the online poll platform Survey Monkey. Awareness strategies to send and monitor the fulfillment of the form by the municipal administrators were defined and applied in partnership with the State Health Secretariat. In addition, reinforcement measures were adopted by the surveyors and addressed to non-respondent municipal administrators, or those who began, but did not conclude, the sending of the questionnaire.

A protocol with the detailed identification of the AJM components was elaborated in an electronic spreadsheet to guide the organization and analysis of data according to dimensions, subdimensions, indicators, measurements, variables, source of evidence, calculation basis and judgment parameters.

For the analysis, the cities were grouped according to population size, in six strata (per number of residents): up to 3 thousand; from 3,001 to 5,999; from 6,000 to 9,999; from 10,000 to 19,999; from 20,000 to 49,999; and equal to or higher than 50 thousand.

For the issuance of value judgment oriented by the sum of assessment elements, reaching at least 75% of the maximum possible limit in each one of the components of the assessment matrix was considered as adequate; reaching at least 50% was inadequate; and others were partially adequate.

ETHICAL ASPECTS OF THE STUDY

The study was approved by the Human Research Ethics Committee (HREC) at UFSC. All of the ethical precepts of the National Health Council resolution n. 510/2016 were followed. All of the interviewers signed a Confidentiality Form, and the interviewees signed an Informed Consent Form.

RESULTS

SUB-STUDY 1

The study included 3,580 women interviewed in the 31 hospitals that participated in the study (Figure 1A). Therefore, refusal rate was 2.3% (n = 85) of the sample that was initially planned. The response rate was higher than 83% in all hospitals - in 25 of them, it was 100%, and in three others it ranged between 98 and 99%. The questionnaires built in the study and applied 48 hours and six months after delivery are available in the supplementary material.

Women living in 244 cities in Santa Catarina, of the total of 295 composing the study, were part of the sample (Figure 1B). By comparing the data in this study with those in the Live Birth Information System (SINASC) of 2019, referring to the same hospitals, it is possible to observe the similarity of values of newborn sex, age and schooling of the puerperal woman, number of prenatal appointments and type of delivery (Table 1). The minor observed variations can be related to the fact that the data from SINASC contemplate women who underwent prenatal care in a public-private alternance, or exclusively in the private sector, even if the delivery was performed in SUS.

No fraud was identified in the interviews, and all of the variables to control quality presented good or almost perfect agreement; six of the eight analyzed variables presented Cohen’s kappa coefficient higher than 0.68. The analysis of the characteristics of the women interviewed six months after delivery also corresponded with the global sample of the study, interviewed in the hospitals (Table 1).

SUB-STUDY 2

The strategies adopted in sub-study 2 allowed a comprehensive understanding of the object - of the involvement of the parties interested in the assessment, the definition of the evaluation plan and its execution. The description of the responsibilities of the municipal administration regarding prenatal care and puerperium in the scope of Rede Cegonha was obtained, and included the identification of goals, objectives, necessary resources, intended activities, expected results and causal relations, synthetized in TM and LM, besides the consensus about the assessment elements.

The elaborated assessment model, based on the reference of Networks and Primary Health Care, considered the context, the health policies, the legislation and rules related to the theme under different points of view, in the light of shared knowledge. It allowed to analyze the administration under two points of view: political-organizational and tactical-operational aspects. The former includes the conditions provided by the municipal administration to ensure care actions to pregnant women, puerperal women and children in prenatal care and puerperal period in Primary Health Care. The latter is characterized by the organization of care and actions/activities developed by PHC professionals for the care addressed to pregnant women, puerperal women and children.

AJM was composed of two dimensions, six subdimensions, 32 indicators and 90 measures, and requires 94 variables for its application. Most information (57.4%) is available in the data base of cycle 3 in the external evaluation of the National Primary Care Access and Quality Improvement Program (PMAQ_AB), and in official databases of public access, or made available after a formal request to competent parties.

The synthesis of the assessment model, with the sources of evidence used for each one of the indicators, and the questions addressed for municipal administrators in an electronic platform, are available in the Supplementary Material.

The primary data collection methodology had to be adjusted throughout the study. At first, the awareness and partnership strategies with the State Secretariat of Health enabled a response rate lower than 30%. With the redefinition of the strategy and responsibilities, and the increase of the collection period for six months, this rate rose to 69.1% (n = 204).

The 204 cities that participated in the study represent the different regions of the state, being: 38 (18.6%) with population size of up to 3 thousand residents; the same number was found in the stratum between 3,001 and 5,999 residents; 36 (17.6%) in the stratum between 6,000 and 9,999 residents; 41 (20.1%), from 10,000 to 19,999 residents; 25 (12.3%), from 20,000 to 49,999; and 26 (12.7%) with population equal to or higher than 50 thousand residents.

DISCUSSION

The collection of primary data in a large probability sample, regarding the care received in prenatal and postpartum periods by women and children, allows the state to look more carefully at the strong and weak aspects of health care offered to this audience in SUS. As a consequence, the strengthening of the decision-making processes and the design of public policies and actions is expected. The definition of a baseline will also allow the administrators and the civil society to count on elements for the future assessment of the actions implemented in this period. The articulation between the academia and health services, from the stage of study planning, was essential so that the results could reflect questionings from the administration and the professionals of SUS, with higher chances of being used by the health services.

In operational terms, the incorporation of REDCap to the poll brought several advantages that should be highlighted. As in every electronic form, it allowed the inclusion of controls that reduced the probability of typos and automatic space, which streamlines the record of data. Besides, it eliminated the stage of typing the printed questionnaires. Finally, the possibility to access all the data in the electronic format at the end of the interview, in a central server, made consistency control faster, and allowed the continuous training of the team of interviewers.

An important step of an assessment study is the enhancement of the knowledge about the object, oriented by a theoretical or normative referential, identifying its operational dynamic, expectation of products and results, plausibility and viability of objectives, in order to direct the choice of the assessment components¹³13. Thurston W, Ramaliu A. Evaluability assessment of a survivors of torture program: lessons learned. Can J Program Eval 2005; 20(2): 1-25.^,1414. Trevisan MS, Walser TM. Evaluability assessment. Estados Unidos: Sage; 2015. 181 p.. These stages, which come prior to the assessment per se, are not always presented in detail in the publications of the field. The involvement of different players, with a diverse participative strategy, deep knowledge about the study object and the analysis of the potential of the assessment contributed to increase the feasibility, the credibility and the use of the results obtained in the assessment per se, as well as for the rational use of resources¹⁶16. Leviton LC, Khan LK, Rog D, Dawkins N, Cotton D. Evaluability Assessment to Improve Public Health Policies, Programs, and Practices. Ann Rev Public Health 2010; 31: 213-33. https://doi.org/10.1146/annurev.publhealth.012809.103625
https://doi.org/https://doi.org/10.1146/... ^,1717. Baratieri T, Nicolloti C, Natal S, Lacerda JT. Aplicação do Estudo de Avaliabilidade na área da saúde: uma revisão integrativa. Saúde Debate 2019; 43(120): 240-55. https://doi.org/10.1590/0103-1104201912018
https://doi.org/https://doi.org/10.1590/... .

The developed assessment model, with reduced number of indicators, mostly extracted from secondary data, increases its potential of reproducibility in other national contexts, allowing a comparative analysis of the conformation of Rede Cegonha. A methodological limitation that requires adjustments to increase the potential that was previously described is the bond of the object to Rede Cegonha, a government proposal that can be disorganized with the alternance of terms. It is worth to mention the care aspects used for the selection of indicators, oriented by the referential of Primary Health Care and Care Networks, requiring punctual adjustments in a few measures, without impairing the model.

References

Publication Dates

History