ABSTRACT

OBJECTIVE

To confirm the diagram of the program’s impact pathways and Estratégia Amamenta e Alimenta Brasil (EAAB - Brazilian Breastfeeding and Complementary Feeding Strategy) core functions, and to determine indicators for evaluating EAAB.

METHODS

This is a qualitative study within the field of implementation research. Data collection included two focus groups with EAAB implementers, document analysis, and literature review. The analysis included a review of the participants’ suggestions and two stages of reviewing the diagram and the names of the impact pathways and core functions. Questions for evaluating the EAAB were then constructed based on the confirmed diagram. The indicators of the RE-AIM framework (reach, effectiveness, adoption, implementation, maintenance) were adapted to the EAAB context. The evaluation questions were revised according to the RE-AIM domains and, finally, indicators were determined for each evaluation question.

RESULTS

This study advanced the analysis of EAAB implementation, defining 22 indicators for its monitoring and evaluation. Most of the indicators are already used in implementation, however, the analysis with the RE-AIM framework allowed the indicators to be updated to be more specific, measurable, and relevant to the desired objectives.

CONCLUSION

The results of the study support the pragmatic application of RE-AIM evaluation in health programs and encourage the planning of evaluation indicators for other child health and development programs in primary care.

Breast Feeding; Infant Nutritional Physiological Phenomena; Primary Health Care; Implementation Science

INTRODUCTION

In Brazil, primary care is the population’s gateway to the Unified Health System (SUS), which offers universal access to a network of care to protect, promote, and support health¹1. Ministério da Saúde (BR). Portaria nº 2.436, de 21 de setembro de 2017. Aprova a Política Nacional de Atenção Básica, estabelecendo a revisão de diretrizes para a organização da Atenção Básica, no âmbito do Sistema Único de Saúde (SUS). Diário Oficial União. 22 set 2017.. A large number of families are served in primary care: in 2020, 54,105 health teams worked in more than 44,000 basic health units (UBS) and served 76.1% of Brazil’s population, equivalent to more than 159.9 million people²2. Ministério da Saúde (BR). Desempenho da Atenção Primária à Saúde no Brasil é alvo de pesquisa inédita. Brasília, DF: Ministério da Saúde; 2020[cited 2023 June 10] Available from: https://aps.saude.gov.br/noticia/10136
https://aps.saude.gov.br/noticia/10136... ,³3. Ministério da Saúde (BR). E-Gestor. Cobertura da Atenção Básica; Brasília, DF: Ministério da Saúde; 2021[cited 2023 June 10] Available from: https://egestorab.saude.gov.br/paginas/acessoPublico/relatorios/relHistoricoCoberturaAB.xhtml
https://egestorab.saude.gov.br/paginas/a... . Adequate and healthy infant feeding is one of the health guidelines promoted in primary care. Data from the Estudo Nacional de Alimentação e Nutrição Infantil (Enani-2019 - Brazilian National Survey on Child Nutrition) showed that 45.8% of Brazilian children under six months were exclusively breastfed and 86.3% between six and eight months were being introduced to food⁴4. Universidade Federal do Rio de Janeiro. Aleitamento materno: prevalência e práticas de aleitamento materno em crianças brasileiras menores de 2 anos. ENANI 2019. Rio de Janeiro: UFRJ, 2021[cited 2024 Abr 3]. Available from: https://enani.nutricao.ufrj.br/index.php/relatorios/
https://enani.nutricao.ufrj.br/index.php... ,⁵5. Universidade Federal do Rio de Janeiro. Alimentação infantil I: prevalência de indicadores de alimentação de crianças menores de 5 anos. ENANI 2019. Rio de Janeiro: UFRJ, 2021[cited 2024 Abr 3]. Available from: https://enani.nutricao.ufrj.br/index.php/relatorios/
https://enani.nutricao.ufrj.br/index.php... . The prevalence of overweight in children under five was 7.0%, while 18.3% of children were at risk of overweight⁶6. Universidade Federal do Rio de Janeiro. Estado nutricional antropométrico da criança e da mãe: prevalência de indicadores antropométrico de crianças brasileiras menores de 5 anos de idade e suas mães biológicas. ENANI 2019. Rio de Janeiro, RJ: UFRJ, 2022[cited 2024 Apr 3]. Available from: https://enani.nutricao.ufrj.br/index.php/relatorios/
https://enani.nutricao.ufrj.br/index.php... . Raising breastfeeding rates and improving the quality of food introduction in the first years of life is key to combating overweight and promoting children’s development to its full potential. To achieve these outcomes, the Brazilian Ministry of Health has been implementing the Estratégia Amamenta e Alimenta Brasil (EAAB - Brazilian Breastfeeding and Complementary Feeding Strategy) in primary care since 2013⁷7. Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Estratégia nacional para promoção do aleitamento materno e alimentação complementar saudável no Sistema Único de Saúde: manual de implementação. Brasília, DF: Ministério da Saúde; 2015..

The EAAB aims to qualify actions to promote and support breastfeeding and complementary feeding practices in the UBS. EAAB is implemented with multi-level governance by managers in the Ministry of Health and coordinators in all states and municipalities in Brazil⁷7. Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Estratégia nacional para promoção do aleitamento materno e alimentação complementar saudável no Sistema Único de Saúde: manual de implementação. Brasília, DF: Ministério da Saúde; 2015.,⁸8. Melo D, Venancio S, Buccini G. brazilian strategy for breastfeeding and complementary feeding promotion: a program impact pathway analysis. Int J Environ Res Public Health. 2022 Aug;19(16):9839. https://doi.org/10.3390/ijerph19169839
https://doi.org/10.3390/ijerph19169839... . Based on the cascade of continuing education for primary care professionals, the EAAB trains tutors to lead workshops with health teams in the UBS. The teams develop an action plan to help families with children under two. As a result, EAAB supports the training of professionals in the UBS and promotes better monitoring of food consumption markers for children under two⁹9. Relvas GR, Buccini G, Potvin L, Venancio S. Effectiveness of an educational manual to promote infant feeding practices in primary health care. Food Nutr Bull. 2019 Dec;40(4):544-61. https://doi.org/10.1177/0379572119855308
https://doi.org/10.1177/0379572119855308... ,¹⁰10. Bonini TP, Lino CM, Sousa ML, Mota MJ. Implantação e efeitos da Estratégia Amamenta Alimenta Brasil nas Unidades de Saúde de Piracicaba/SP. Res Soc Develop. 2021;10(14):e91101421528 https://doi.org/10.33448/rsd-v10i14.21528
https://doi.org/10.33448/rsd-v10i14.2152... .

By 2018, more than 5,500 tutors had been trained, and more than 3,100 workshops had been held at the UBS¹¹11. Ministério da Saúde (BR). Coordenação-Geral de Alimentação e Nutrição. Área Técnica de Saúde da Criança e Aleitamento Materno. Estratégia amamenta e alimenta Brasil: relatórios. 2023[cited 2023 Apr 23]. Available from: http://sisaps.saude.gov.br/eaab/Relatorios/relatorios.php
http://sisaps.saude.gov.br/eaab/Relatori... . Considering that there were more than 50,000 health teams in the country³3. Ministério da Saúde (BR). E-Gestor. Cobertura da Atenção Básica; Brasília, DF: Ministério da Saúde; 2021[cited 2023 June 10] Available from: https://egestorab.saude.gov.br/paginas/acessoPublico/relatorios/relHistoricoCoberturaAB.xhtml
https://egestorab.saude.gov.br/paginas/a... , these results showed that the expansion of the EAAB was less than expected. The Ministry of Health therefore began a project to strengthen implementation with strategies on a national scale in four areas: support for coordinators; expansion of tutor training; monitoring of implementation in the municipalities; and research and evaluation¹²12. Venancio SI, Relvas GR, Melo DS, Souza CB, Moreira HOM, Veras JJP, et al. Implementation strategies for a brazilian policy aimed at promoting breastfeeding and healthy complementary feeding in primary care. Glob Implement Res Appl. 2023;3(4):355-66. https://doi.org/10.1007/s43477-023-00098-z
https://doi.org/10.1007/s43477-023-00098... . As part of the research and evaluation axis, an analysis of the pathways to achieve the impact of the EAAB was conducted. Researchers interactively developed a diagram of the program’s impact pathway theory and analyzed barriers and facilitators to implementation. The results documented six critical control points of EAAB implementation, assumptions, facilitators, and contextual factors, described in another publication⁸8. Melo D, Venancio S, Buccini G. brazilian strategy for breastfeeding and complementary feeding promotion: a program impact pathway analysis. Int J Environ Res Public Health. 2022 Aug;19(16):9839. https://doi.org/10.3390/ijerph19169839
https://doi.org/10.3390/ijerph19169839... . The analysis of these elements pointed to seven implementation core functions, i.e. planned activities that need to be followed to ensure successful implementation on a large scale⁸8. Melo D, Venancio S, Buccini G. brazilian strategy for breastfeeding and complementary feeding promotion: a program impact pathway analysis. Int J Environ Res Public Health. 2022 Aug;19(16):9839. https://doi.org/10.3390/ijerph19169839
https://doi.org/10.3390/ijerph19169839... .

Experts recommend that after conducting the analysis of the program’s impact pathways diagram, it is important to consider implementing a robust impact evaluation system that allows for an objective understanding of whether the program is achieving its stated goals and whether it is following the pathways described in the diagram¹³13. Pérez-Escamilla R, Segura-Pérez S, Damio G. Applying the Program Impact Pathways (PIP) evaluation framework to school-based healthy lifestyles programs: Workshop Evaluation Manual. Food Nutr Bull. 2014 Sep;35(3 Suppl):S97-107. https://doi.org/10.1177/15648265140353S202
https://doi.org/10.1177/15648265140353S2... . The choice of indicators should be informed by the analysis of the program’s impact pathways and can use conceptual frameworks that support the operationalization of the indicators¹³13. Pérez-Escamilla R, Segura-Pérez S, Damio G. Applying the Program Impact Pathways (PIP) evaluation framework to school-based healthy lifestyles programs: Workshop Evaluation Manual. Food Nutr Bull. 2014 Sep;35(3 Suppl):S97-107. https://doi.org/10.1177/15648265140353S202
https://doi.org/10.1177/15648265140353S2... . The RE-AIM evaluation framework (reach, effectiveness, adoption, implementation, maintenance), for example, has a long history of use for planning and evaluating health programs¹⁴14. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019 Mar;7:64. https://doi.org/10.3389/fpubh.2019.00064
https://doi.org/10.3389/fpubh.2019.00064... . In summary, the “reach” domain assesses how far the program reaches the beneficiary population; the “effectiveness” domain analyzes the benefits produced for the beneficiary population; the “adoption” domain verifies where and who adopts the program, with a greater focus on implementers, communities, organizations, and systems; the “implementation” domain assesses the consistency of implementation, costs, and adaptations in the delivery of actions; finally, the “maintenance” domain analyzes the sustainability of results over time, both for the beneficiary population and at the organizational level¹⁴14. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019 Mar;7:64. https://doi.org/10.3389/fpubh.2019.00064
https://doi.org/10.3389/fpubh.2019.00064... ,¹⁵15. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, Resnick B. Implementation fidelity in community-based interventions. Res Nurs Health. 2010 Apr;33(2):164-73. https://doi.org/10.1002/nur.20373
https://doi.org/10.1002/nur.20373... . In this type of analysis, it is important to look at the indicators already in use by the programs, which can be consulted by reviewing the literature and discussing them with program workers and decision-makers. It is also useful to suggest new indicators that are needed, using evaluation models and frameworks to facilitate the systematized identification of indicators of implementation, effectiveness, and sustainability¹⁶16. Grembowski D. The practice of health program evaluation. 2nd ed. Thousand Oaks: Sage; 2015..

Investment in EAAB implementation research in recent years has helped to clarify the pathways and activities central to its successful implementation. In order to advance the analysis of EAAB implementation, this study aimed to 1) confirm the diagram of program impact pathways and EAAB core functions, and 2) define monitoring and evaluation indicators to compose a robust EAAB evaluation scheme based on the RE-AIM framework.

METHODS

The description of the methodological steps was guided by the Standards for Reporting Implementation Studies (StaRI)¹⁷17. Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017 Mar;356. https://doi.org/10.1136/bmj.i6795
https://doi.org/10.1136/bmj.i6795... and consolidated criteria for reporting qualitative research (COREQ)¹⁸18. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007 Dec;19(6):349-57. https://doi.org/10.1093/intqhc/mzm042
https://doi.org/10.1093/intqhc/mzm042... .

Study Design

This is a qualitative observational study within the field of implementation research. Data collection included two focus groups with EAAB implementers, document analysis, and literature review. Figure 1 describes the stages of data collection and analysis in the two phases of the study: 1) confirming the impact pathways and core functions of the EAAB, and 2) determining indicators for evaluating the EAAB. This study was approved by the Research Ethics Committee of the School of Public Health at the University of São Paulo (USP), under protocol number 15184019.2.0000.5421. All participants signed an Informed Consent Form to register their intention to participate.

Confirmation of EAAB Impact Paths and Core Functions

Selection of Participants

Key informants implementing the EAAB in the Ministry of Health, states, and municipalities in the country’s five macro-regions (North, Northeast, Midwest, Southeast, and South) were selected. A non-probabilistic purposive sample was selected. In order to capture the perspectives of different implementers, key informants who had already taken part in the study analyzing the program’s impact pathways were excluded⁸8. Melo D, Venancio S, Buccini G. brazilian strategy for breastfeeding and complementary feeding promotion: a program impact pathway analysis. Int J Environ Res Public Health. 2022 Aug;19(16):9839. https://doi.org/10.3390/ijerph19169839
https://doi.org/10.3390/ijerph19169839... . The inclusion criteria and selected participants are described in Chart 1. States and municipalities with advanced stages of implementation were selected because they had more experience in implementing workshops and certifying UBS. The people selected were invited by e-mail to take part in the study. Only one municipality in the Northeast region was unavailable at the time, so key informants from the second municipality with the most advanced stage of EAAB in that region were invited.

Relations with Participants

Two authors, D.S.M. and S.I.V., were part of the national project to strengthen the EAAB and participated in remote meetings with EAAB implementers in states and municipalities across the country between 2020 and 2022¹²12. Venancio SI, Relvas GR, Melo DS, Souza CB, Moreira HOM, Veras JJP, et al. Implementation strategies for a brazilian policy aimed at promoting breastfeeding and healthy complementary feeding in primary care. Glob Implement Res Appl. 2023;3(4):355-66. https://doi.org/10.1007/s43477-023-00098-z
https://doi.org/10.1007/s43477-023-00098... .

Data Collection

Two online focus groups were held in November 2022, one group with coordinators from the Ministry of Health and states and another group with municipal coordinators and tutors. A total of nine people took part in the first group and six people in the second, as well as a moderator (D.S.M.), an observer (S.I.V.), and a technical supporter (C.B.S.) experienced in qualitative methodology. A discussion guide was used, drawn up by a co-author (D.S.M) and revised by three co-authors (D.S.M., S.I.V., and G.B.). The groups were conducted in two stages: 1) to confirm the pathways diagram, and 2) to confirm the core functions. To make it easier to confirm the activities described in the EAAB program impact pathway diagram identified in a previous study⁹9. Relvas GR, Buccini G, Potvin L, Venancio S. Effectiveness of an educational manual to promote infant feeding practices in primary health care. Food Nutr Bull. 2019 Dec;40(4):544-61. https://doi.org/10.1177/0379572119855308
https://doi.org/10.1177/0379572119855308... , the decision was made to adapt the information into six diagrams: 1) pathways for EAAB coordination and funding, 2) pathways for tutor training, 3) pathways for tutor support for UBS teams, 4) pathways for EAAB monitoring, 5) pathways for certification criteria, and 6) pathways for the UBS certification process. The first four diagrams of the EAAB program’s impact pathways were presented, and diagrams 5 and 6 were not presented because they were activities that were temporarily suspended in the implementation of EAAB at the time of the research. Secondly, a chart of the core functions of the EAAB was presented: 1) the presence of coordinators at the three levels of government and interfederative and intersectoral coordination; 2) the allocation of resources for the implementation of the EAAB; 3) the planning of implementation at the three levels of government; 4) the establishment of a group of national facilitators, the training of tutors and of UBS professionals; 5) the development of activities to obtain certification; 6) the monitoring and evaluation of implementation; 7) the dissemination of the EAAB. After the presentation of each diagram and the chart of core functions, the participants confirmed whether the descriptions were adequate or needed to be modified.

Data Analysis

The notes and audio and video recordings of the two focus groups were revised by the first author to highlight the suggestions made by the key informants in the diagrams of the program’s impact pathways and EAAB’s core functions. Two co-authors (D.S.M. and S.I.V.) once again reviewed the materials to polish the texts. Then, three co-authors (D.S.M., S.I.V., and G.B.) organized the formatting of the final diagram and the names of the EAAB implementation pathways and core functions.

Determining Indicators for Monitoring and Evaluating the EAAB

Documentary Analysis

A document analysis was carried out to identify the indicators already used to monitor and evaluate the EAAB. We consulted the EAAB Implementation Manual⁷7. Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Estratégia nacional para promoção do aleitamento materno e alimentação complementar saudável no Sistema Único de Saúde: manual de implementação. Brasília, DF: Ministério da Saúde; 2015., the monitoring reports for the project to strengthen the EAAB in 2021 and 2022¹²12. Venancio SI, Relvas GR, Melo DS, Souza CB, Moreira HOM, Veras JJP, et al. Implementation strategies for a brazilian policy aimed at promoting breastfeeding and healthy complementary feeding in primary care. Glob Implement Res Appl. 2023;3(4):355-66. https://doi.org/10.1007/s43477-023-00098-z
https://doi.org/10.1007/s43477-023-00098... , Ordinance No. 1.124 of May 19, 2022 (which established new indicators for monitoring the EAAB)¹⁹19. Ministério da Saúde (BR). Portaria GM/MS nº 1.124, de 19 de maio de 2022. Institui, em caráter excepcional e temporário, o incentivo financeiro de custeio para as ações de promoção, proteção e apoio ao aleitamento materno e da alimentação complementar adequada e saudável para crianças menores de 2 (dois) anos de idade no âmbito da Estratégia Amamenta e Alimenta Brasil (EAAB), na Atenção Primária à Saúde (2022). Diário Oficial União. 20 maio 2022., and the primary care monitoring forms for recording food consumption and nutritional status markers and individual and collective activities in the UBS²⁰20. Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Orientações para avaliação de marcadores de consumo alimentar na atenção básica. Brasília, DF: Ministério da Saúde; 2015..

Literature Review

The RE-AIM evaluation framework was chosen to support the determination of EAAB evaluation indicators because it has a long history of use for planning and evaluating health programs¹⁴14. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019 Mar;7:64. https://doi.org/10.3389/fpubh.2019.00064
https://doi.org/10.3389/fpubh.2019.00064... . A literature review was conducted to identify definitions of the RE-AIM domains with the aim of adapting them to evaluate EAAB. Peer-reviewed articles, supplementary texts, and official checklists available on the “https://re-aim.org/assessments/” website²¹21. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. 2018 Jan;15:170271. https://doi.org/10.5888/pcd15.170271
https://doi.org/10.5888/pcd15.170271... were included.

Analysis

The confirmed EAAB impact pathways diagram (Figure 2) and the indicators used to evaluate EAAB were consulted to guide the construction of EAAB evaluation questions. The evaluation questions were drafted in an interactive discussion process considering three main structures of the diagram: the implementation process, the outcomes in the service, and the outcomes in the population. Then, in an interactive discussion process, the authors adapted the definitions of the RE-AIM implementation outcome indicators to the EAAB context. The evaluation questions were then revised and allocated to the RE-AIM domains in the following order: adoption, implementation, reach, effectiveness, and maintenance. Finally, the indicators already used in the EAAB were reorganized, some grouped, others excluded or adapted, and so indicators were defined for each evaluation question and organized into levels: organizational (states, municipalities, and UBS), professional (tutors and health teams), and individual (beneficiary target population)²²22. Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci. 2021 May;5(1):e126. https://doi.org/10.1017/cts.2021.789
https://doi.org/10.1017/cts.2021.789... .

RESULTS

Confirmation of EAAB Impact Pathways and Core Functions

The diagram of the EAAB program’s impact pathways after confirmation and review (Figure 2) is organized into three structures: “implementation process”, “service outcomes”, and “population outcomes”. Within these structures, the activities of “inputs”, “processes”, “outputs” (which include short-term and intermediate results), and “impacts”, which are the expected long-term results, are highlighted. The diagram then details four impact pathways: 1) pathways for coordination articulation, planning, and financing EAAB implementation; 2) pathways for training tutors and workshops; 3) pathways for complementary activities and promotion of healthy infant feeding; and 4) pathways for monitoring, evaluating, and disseminating the EAAB. Finally, the facilitators at the bottom of each pathway diagram are presented, which are activities or contexts that facilitate successful implementation.

Participants agreed on the definitions of the seven EAAB core functions. In order to make the names of the core functions more objective without compromising their content, the core functions were reorganized into eight items: 1) coordination, 2) political and legislative support, 3) financing and resources, 4) planning and goals, 5) training and workshops, 6) delivery and promotion, 7) research, monitoring and evaluation, and 8) social communication. This reorganization was based on the Breastfeeding Gear Model for programs in low- and middle-income countries²⁵25. Pérez-Escamilla R, Curry L, Minhas D, Taylor L, Bradley E. Scaling up of breastfeeding promotion programs in low- and middle-income countries: the "breastfeeding gear" model. Adv Nutr. 2012 Nov;3(6):790-800. https://doi.org/10.3945/an.112.002873
https://doi.org/10.3945/an.112.002873... . The EAAB’s core functions are structural objectives and important activities for successful implementation, so they have been incorporated into the diagrams of the four impact pathways, as shown in Figure 2.

Determining Indicators for Monitoring and Evaluating the EAAB

Eleven indicators were identified in the EAAB Implementation Manual, eleven in the EAAB strengthening project reports, and three in Ordinance No. 1.124/2022. Most of the indicators have been retained with adaptations to their description to make them more specific and measurable. Only three of them were excluded because they were specific activities in the certification criteria for actions in the UBS. A new indicator was added, which deals with assessing the stages of implementation of the EAAB. As a result, 22 indicators were determined to assess the activities that are in place in the implementation of the EAAB. Chart 2 describes: the definition of the RE-AIM indicators adapted for evaluating the EAAB, the evaluation questions, the evaluation indicators, and the data sources. In the “adoption” domain, four indicators were determined relating to the training of tutors and the presence of coordinators in the states and municipalities. In the “implementation” domain, seven indicators were determined relating to financial resources, the stage of implementation, and the consistency of implementation for the protection, promotion, and support of healthy infant feeding in the UBS. In the “reach” domain, three indicators were determined, focusing on the coverage of the target population of children under two years of age with EAAB actions. To assess the “effectiveness” domain, two indicators were determined: the percentage of records and the annual prevalence of infant feeding markers. Finally, in the “maintenance” domain, six indicators were determined to assess the institutionalization of the implementation of the EAAB in the municipalities, the continuity of the tutors’ work, and the continuous improvement of the effectiveness indicators.

DISCUSSION

This study advanced the analysis of the EAAB program impact pathways, interactively confirming the the activities and core functions of implementation with implementers from different levels and locations. The results of the initial analysis and the RE-AIM evaluation framework guided the unprecedented determination of indicators of EAAB adoption, implementation, reach, effectiveness, and maintenance.

Interactive confirmation of the program’s impact pathway diagram is described in the literature as an important strategy for health program coordinators to have a common understanding of program activities from the macro to the local level¹³13. Pérez-Escamilla R, Segura-Pérez S, Damio G. Applying the Program Impact Pathways (PIP) evaluation framework to school-based healthy lifestyles programs: Workshop Evaluation Manual. Food Nutr Bull. 2014 Sep;35(3 Suppl):S97-107. https://doi.org/10.1177/15648265140353S202
https://doi.org/10.1177/15648265140353S2... ,²⁶26. Slater E, Lasco ML, Capelli J, Pen G. Health in action program, Brazil: a Program Impact Pathways (PIP) analysis. Food Nutr Bull. 2014 Sep;35(3 Suppl):S108-16. https://doi.org/10.1177/15648265140353S203
https://doi.org/10.1177/15648265140353S2... ,²⁷27. Mukhina M, Novikova I. BeHealthy Charities Aid Foundation Program, Russia: a Program Impact Pathways (PIP) analysis. Food Nutr Bull. 2014 Sep;35(3 Suppl):S139-44. https://doi.org/10.1177/15648265140353S207
https://doi.org/10.1177/15648265140353S2... . In line with the results described in other studies²⁶26. Slater E, Lasco ML, Capelli J, Pen G. Health in action program, Brazil: a Program Impact Pathways (PIP) analysis. Food Nutr Bull. 2014 Sep;35(3 Suppl):S108-16. https://doi.org/10.1177/15648265140353S203
https://doi.org/10.1177/15648265140353S2... ,²⁷27. Mukhina M, Novikova I. BeHealthy Charities Aid Foundation Program, Russia: a Program Impact Pathways (PIP) analysis. Food Nutr Bull. 2014 Sep;35(3 Suppl):S139-44. https://doi.org/10.1177/15648265140353S207
https://doi.org/10.1177/15648265140353S2... , in this study the process of interaction between researchers and implementers helped to confirm the core activities of the program and gain an understanding of the indicators for measuring the objectives.

This study contributed with adjustments to the definitions of impact pathways, core functions, and indicators, aligning them with the taxonomy recommended in implementation science²⁸28. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009 Jan;36(1):24-34. https://doi.org/10.1007/s10488-008-0197-4
https://doi.org/10.1007/s10488-008-0197-... . Thus, the 22 indicators proposed make it possible to monitor and evaluate implementation processes (existence of coordinators and training of tutors), outcomes in the service (operationalization of workshops and actions in the UBS), and outcomes in the population (registration and improvement of breastfeeding and complementary feeding markers). Conceptualizing and measuring implementation results, as presented in our research, is a critical step for understanding implementation processes, for increasing effectiveness in implementation research, and for opening avenues for comparative studies on implementation strategies²⁹29. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011 Mar;38(2):65-76. https://doi.org/10.1007/s10488-010-0319-7
https://doi.org/10.1007/s10488-010-0319-... .

This study presented a comprehensive description of the methodological step-by-step process for determining indicators for the EAAB. This is an important contribution to the literature, as there is a lack of publications detailing the methods used to determine evaluation indicators based on the analysis of the program’s impact pathways²⁶26. Slater E, Lasco ML, Capelli J, Pen G. Health in action program, Brazil: a Program Impact Pathways (PIP) analysis. Food Nutr Bull. 2014 Sep;35(3 Suppl):S108-16. https://doi.org/10.1177/15648265140353S203
https://doi.org/10.1177/15648265140353S2... ,²⁷27. Mukhina M, Novikova I. BeHealthy Charities Aid Foundation Program, Russia: a Program Impact Pathways (PIP) analysis. Food Nutr Bull. 2014 Sep;35(3 Suppl):S139-44. https://doi.org/10.1177/15648265140353S207
https://doi.org/10.1177/15648265140353S2... . In our study, the use of the RE-AIM evaluation framework to guide the definition of indicators ensured the robustness of the methods and results and confirmed the usefulness and applicability of this framework to guide the evaluation of large-scale programs such as the EAAB³⁰30. Lee RE, Galavíz KI, Soltero EG, Rosales Chavez J, Jauregui E, Lévesque L, et al. Applying the RE-AIM conceptual framework for the promotion of physical activity in low- and middle-income countries. Rev Latino-Am Enfermagem. 2017;25:e2923. https://doi.org/10.1590/1518-8345.1894.2923
https://doi.org/10.1590/1518-8345.1894.2... .

The RE-AIM framework is a useful tool for determining adjustments and setting goals during the process of implementing health service improvement projects³¹31. Glasgow RE, Battaglia C, McCreight M, Ayele RA, Rabin BA. Making implementation science more rapid: use of the RE-AIM framework for mid-course adaptations across five health services research projects in the Veterans Health Administration. Front Public Health. 2020 May;8:194. https://doi.org/10.3389/fpubh.2020.00194
https://doi.org/10.3389/fpubh.2020.00194... . However, the studies are predominantly from high-income countries and the application of the framework has been more pragmatic, i.e. not all RE-AIM domains are usually assessed³²32. D'Lima D, Soukup T, Hull L. Evaluating the application of the RE-AIM planning and evaluation framework: an updated systematic review and exploration of pragmatic application. Front Public Health. 2022 Jan;9:755738. https://doi.org/10.3389/fpubh.2021.755738
https://doi.org/10.3389/fpubh.2021.75573... . Studies have shown that, during program planning, most implementers focus more on the “reach” and “implementation” domains and less on the “effectiveness” and “maintenance” domains³¹31. Glasgow RE, Battaglia C, McCreight M, Ayele RA, Rabin BA. Making implementation science more rapid: use of the RE-AIM framework for mid-course adaptations across five health services research projects in the Veterans Health Administration. Front Public Health. 2020 May;8:194. https://doi.org/10.3389/fpubh.2020.00194
https://doi.org/10.3389/fpubh.2020.00194... ,³³33. Kwan BM, McGinnes HL, Ory MG, Estabrooks PA, Waxmonsky JA, Glasgow RE. RE-AIM in the Real World: Use of the RE-AIM Framework for program planning and evaluation in clinical and community settings. Front Public Health. 2019 Nov;7:345. https://doi.org/10.3389/fpubh.2019.00345
https://doi.org/10.3389/fpubh.2019.00345... , the latter being considered by members as the least important for proposing adjustments during project implementation. The present study fills these gaps by determining indicators for the five RE-AIM domains to plan the evaluation of a large-scale child nutrition promotion program in a middle-income country.

Indicators were defined to monitor and evaluate the EAAB, most of which were already being used. The analysis made it possible to update the indicators to be more specific, i.e. clearer to capture the desired changes in key outcomes, as recommended in the literature¹³13. Pérez-Escamilla R, Segura-Pérez S, Damio G. Applying the Program Impact Pathways (PIP) evaluation framework to school-based healthy lifestyles programs: Workshop Evaluation Manual. Food Nutr Bull. 2014 Sep;35(3 Suppl):S97-107. https://doi.org/10.1177/15648265140353S202
https://doi.org/10.1177/15648265140353S2... . Experts recommend that the indicators informed by the analysis of the program’s impact pathways should be specific, measurable, achievable, relevant, and time-bound, noting whether they are already in use by the program or whether they are planned¹³13. Pérez-Escamilla R, Segura-Pérez S, Damio G. Applying the Program Impact Pathways (PIP) evaluation framework to school-based healthy lifestyles programs: Workshop Evaluation Manual. Food Nutr Bull. 2014 Sep;35(3 Suppl):S97-107. https://doi.org/10.1177/15648265140353S202
https://doi.org/10.1177/15648265140353S2... . One example was updating the indicator “number of tutors trained” to “number of tutors in the municipality in relation to the number of tutors planned in the implementation plan”, so the indicator is more specific to adoption at the municipal level and measures the existence of a tutor training plan.

Some EAAB indicators were specific to certification criteria that were under review by the Ministry of Health. A previous study showed that UBS teams found it difficult to organize and send supporting documents for certain activities for certification, and the Ministry of Health had an overload of tasks which delayed analysis of the documents sent⁸8. Melo D, Venancio S, Buccini G. brazilian strategy for breastfeeding and complementary feeding promotion: a program impact pathway analysis. Int J Environ Res Public Health. 2022 Aug;19(16):9839. https://doi.org/10.3390/ijerph19169839
https://doi.org/10.3390/ijerph19169839... . Therefore, it was justified to keep indicators of the certification criteria in the “implementation” domain that were measurable (quantitative and with tracking of progress over time) and achievable (realistically measurable with the data and resources available)³⁴34. Center for Disease Control and Prevention. Writing SMART objectives. 2018 [cited 2024 July 3]. Available from: https://www.cdc.gov/healthyyouth/evaluation/pdf/brief3b.pdf
https://www.cdc.gov/healthyyouth/evaluat... .

Aoki et al.³⁵35. Aoki A, Mochida K, Kuramata M, Sadamori T, Bhandari AK, Freitas HR, et al. The RE-AIM framework-based evaluation of the implementation of the Maternal and Child Health Handbook program in Angola: a mixed methods study. BMC Health Serv Res. 2022 Aug;22(1):1071. https://doi.org/10.1186/s12913-022-08454-9
https://doi.org/10.1186/s12913-022-08454... used 14 indicators based on the RE-AIM framework to assess the stage of implementation of a maternal and child health promotion program in health institutions in Angola. The institutions were classified into optimal or suboptimal implementation groups, while interviews made it possible to complement the assessment and identify barriers and facilitators to implementation. In contrast, EAAB first underwent an analysis of barriers and facilitators to implementation and had its implementation strategies documented. This research generated positive results, especially in multilevel governance and tutor training⁸8. Melo D, Venancio S, Buccini G. brazilian strategy for breastfeeding and complementary feeding promotion: a program impact pathway analysis. Int J Environ Res Public Health. 2022 Aug;19(16):9839. https://doi.org/10.3390/ijerph19169839
https://doi.org/10.3390/ijerph19169839... ,¹²12. Venancio SI, Relvas GR, Melo DS, Souza CB, Moreira HOM, Veras JJP, et al. Implementation strategies for a brazilian policy aimed at promoting breastfeeding and healthy complementary feeding in primary care. Glob Implement Res Appl. 2023;3(4):355-66. https://doi.org/10.1007/s43477-023-00098-z
https://doi.org/10.1007/s43477-023-00098... . It is understood that the indicators determined in this study could be used in the future to assess stages of implementation and identify priority municipalities for monitoring both the quality and effectiveness of EAAB implementation.

A strength of the indicator framework presented in this study is that it favors the application of the RE-AIM framework in the real world, agreeing with experts in the field who encourage the expansion of the use of RE-AIM in community-based programs³¹31. Glasgow RE, Battaglia C, McCreight M, Ayele RA, Rabin BA. Making implementation science more rapid: use of the RE-AIM framework for mid-course adaptations across five health services research projects in the Veterans Health Administration. Front Public Health. 2020 May;8:194. https://doi.org/10.3389/fpubh.2020.00194
https://doi.org/10.3389/fpubh.2020.00194... . In Brazil, few studies have used the RE-AIM framework to evaluate large-scale health programs and policies³⁶36. Buccini G, Gubert MB, Palmeira PA, Godoi L, Santos LD, Esteves G, et al. Scaling up a home-visiting program for child development in Brazil: a comparative case studies analysis. Lancet Reg Health Am. 2024 Jan;29:100665. https://doi.org/10.1016/j.lana.2023.100665
https://doi.org/10.1016/j.lana.2023.1006... ,³⁷37. Buccini G, Venancio SI, Pérez-Escamilla R. Scaling up of Brazil's Criança Feliz early childhood development program: an implementation science analysis. Ann N Y Acad Sci. 2021 Aug;1497(1):57-73. https://doi.org/10.1111/nyas.14589
https://doi.org/10.1111/nyas.14589... . It is known that funding for the application of RE-AIM is often dependent on external program resources, such as national agencies, cooperation between academics, and health services and foundations³³33. Kwan BM, McGinnes HL, Ory MG, Estabrooks PA, Waxmonsky JA, Glasgow RE. RE-AIM in the Real World: Use of the RE-AIM Framework for program planning and evaluation in clinical and community settings. Front Public Health. 2019 Nov;7:345. https://doi.org/10.3389/fpubh.2019.00345
https://doi.org/10.3389/fpubh.2019.00345... . This study was conducted as part of doctoral research funded by a national research agency. It was therefore possible to devote time to an in-depth analysis of the program’s impact pathways and RE-AIM domains and to propose a systematic EAAB evaluation framework. This highlights the importance of cooperation between academia and decision-makers in public policy and complex health programs, in order to make the pragmatic application of RE-AIM feasible.

This study has limitations and strengths. The data collected in the focus groups is limited to the perceptions of the participants who attended the meetings, and who were only from municipalities with an advanced implementation context. The second focus group saw the absence of the municipal coordinator and tutor from the North-East region and the absence of tutors from the North and South regions. On the other hand, in the interactive process of confirming the impact pathways diagram, it was observed that most of the suggestions from the second group were similar to those from the first group, demonstrating the saturation and robustness of the results presented. Some of the indicators determined for the EAAB require the Ministry of Health to periodically send online forms to municipalities, as they are not available in official primary health care information systems. We understand that future adaptations to the data sources for these indicators may be necessary. Despite this limitation, we believe that these indicators should be maintained as part of the EAAB evaluation system, as they refer to the existence of coordinators, funding, and the implementation plan, which are critical points for control, identified in a previous study analyzing the implementation of the EAAB⁸8. Melo D, Venancio S, Buccini G. brazilian strategy for breastfeeding and complementary feeding promotion: a program impact pathway analysis. Int J Environ Res Public Health. 2022 Aug;19(16):9839. https://doi.org/10.3390/ijerph19169839
https://doi.org/10.3390/ijerph19169839... .

CONCLUSIONS

This study advanced the analysis of EAAB implementation by determining indicators for evaluation with reference to implementation research theories and frameworks. The theory of impact pathways and the confirmed core functions act as gears to guide the implementation of EAAB on a large scale in a consistent manner and achieve successful implementation. The knowledge gained through interaction with EAAB coordinators from the country’s five macro-regions supports the generalization of the results. Indicators were defined to monitor and evaluate the EAAB which, for the most part, were already being used in the implementation. However, the analysis with the RE-AIM framework allowed the indicators to be updated to be more specific, measurable and relevant to the desired objectives. Overall, the results of the study support the pragmatic application of RE-AIM evaluation in health programs and encourage the planning of evaluation indicators for other child health and development programs in primary care.

REFERENCES

Publication Dates

History