Abstract

Health promotion has a set of strategies for advancing health and reducing inequalities. However, evaluating the effectiveness of health promotion programs has been a challenge. This paper shows the development and application of the Evaluation Matrix, constructed with qualitative-quantitative and multidimensional indicators supported by public policies targeting socially vulnerable territories. This is a cross-sectional study with the implementation of a health promotion program in order to develop an Evaluation Matrix to be applied in two distinct socio-environmentally vulnerable areas. The Evaluation Matrix proved to be easily applicable and enabled the detection of strengths and weaknesses of health promotion programs applied in different territories. The participation of managers, teams, population and multiple sectors of society was decisive for the success of the program. Furthermore, community health workers stood out as essential stakeholders due to their linkages with the population. Contributions include a tool and methodology for evaluating health promotion programs to be applied in different territories and modified according to the territory.

Key words
Primary Health Care; Inequities; Territory

Introduction

In 1946, Sigerist referred to the term “health promotion” in defining essential medical tasks such as health promotion, disease prevention, patient recovery and rehabilitation¹1. Sigerist HE. The University at the Crossroads. Addresses and Essays. Nova Iorque: Henry Schuman;1946..

In the 1970s, with the revival of nineteenth-century social medical thinking, the term health promotion was once again used mainly in Canada and in Western European countries, from the discussion about effective high-level curative health care technology, increasing health medicalization and the need to reduce the cost of the current biomedical model²2. Buss PM, Carvalho AI. Desenvolvimento da Promoção da Saúde no Brasil nos últimos vinte anos (1988-2008). Cien Saude Colet 2009; 14(6):2305-2316..

In the 1980s, the International Conferences of Ottawa (1986), Adelaide (1988) and Sundsvall (1991) initiated the global health promotion movement and brought to the discussion the concepts of health, risk, social vulnerability, territory, intersectoriality, participation and surveillance, as well as the current conceptual and political bases for health promotion³3. Czeresnia D, Freitas, CM, organizadores. Promoção da Saúde: conceitos, reflexões, tendências. 2ª ed. rev. e amp. Rio de Janeiro: Fiocruz; 2009..

In Brazil, this discussion was inspired by the progressive model, with a scientific stance and critical analysis of the relationship between health and society. Thus, there was an extensive production of papers aimed at understanding this relationship and the origin of the different epidemiological profiles found in our society, characterized by inequality. Therefore, health promotion began being perceived through its determinants and the understanding of the health-disease process was expanded³3. Czeresnia D, Freitas, CM, organizadores. Promoção da Saúde: conceitos, reflexões, tendências. 2ª ed. rev. e amp. Rio de Janeiro: Fiocruz; 2009..

In the last two decades, some Latin American and Caribbean countries have implemented reforms in their health systems that have fostered inclusion, citizen participation and equitable access to health care⁴4. United Nations Children's Fund Health (Unicef). Equity Report: Analysis of reproductive, maternal, newborn, child and adolescent health inequities in Latin America and the Caribbean to inform policymaking. Panama City: Unicef; 2016.. Despite these initiatives, huge inequalities in the coverage of health interventions persist in most countries of the region.

In Brazil, the economic advances observed during the military regime have disproportionately benefited the privileged segments of society and the movements for democracy provided a broad discussion on the needs of the population, thus catalyzing the Health Reform⁵5. Paim J, Travassos C, Almeida C, Bahia L, Macinko J. O Sistema de Saúde Brasileiro: histórias, avanços e desafios. Lancet 2011; 377 (9779):1778-1797.. The construction of the Unified Health System (SUS), based on the principles of universalization, equity and integrality was a commitment of the State to comply with its duty to provide health promotion policies. It is in this context that Primary Health Care of SUS, primarily through the Family Health Strategy (FHS), acquired the important role of responsible for providing humanized access, coordinated care, comprehensive services and equity in its actions⁶6. Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Secretaria de Atenção à Saúde. Política Nacional de Promoção da Saúde (PNPS):revisão da Portaria MS/GM n° 687, de 30 de março de 2006. Brasília: MS; 2015.. Health promotion programs have been implemented by SUS with the aim of reducing inequalities in different territories⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626., but there is no evidence of an evaluation tool that originates in territorialized social realities⁸8. Minayo MCS. Saúde e Ambiente Sustentável: estreitando nós. Rio de Janeiro: Fiocruz; 2011.^,99. Kusma SZ, Moysés ST, Moysés SJ. Promoção da saúde: perspectivas avaliativas para a saúde bucal na atenção primária em saúde. Cad Saude Publica 2012; 28(Supl.):S9-S19. that considers the participation of societal sectors and stakeholders⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626..

The need to follow-up on the quality of health systems through monitoring and systematic evaluations has been emphasized by the World Health Organization (WHO)¹⁰10. World Health Organ (WHO). The World Health Report 2008: Primary Health Care. Geneva: WHO. [acessado 2015 Maio 8]. Disponível em: http://www.who.int/whr/2008/en/
http://www.who.int/whr/2008/en/... , but they generally use disease outcomes such as mortality and morbidity. Previous studies have shown that health promotion has the potential to act effectively in the health-disease process in a way consistent with the reality of each territory²2. Buss PM, Carvalho AI. Desenvolvimento da Promoção da Saúde no Brasil nos últimos vinte anos (1988-2008). Cien Saude Colet 2009; 14(6):2305-2316.^,33. Czeresnia D, Freitas, CM, organizadores. Promoção da Saúde: conceitos, reflexões, tendências. 2ª ed. rev. e amp. Rio de Janeiro: Fiocruz; 2009.^,77. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626.; but evaluating health promotion programs is a challenge, since health promotion has a differentiated vision based on territoriality and is unrelated to the exclusively biomedical vision of disease. In collaboration with WHO, since 2002, the International Union for Health Promotion and Education (IUHPE) has started to support initiatives to assess and show the effectiveness of health promotion to health system managers¹⁰10. World Health Organ (WHO). The World Health Report 2008: Primary Health Care. Geneva: WHO. [acessado 2015 Maio 8]. Disponível em: http://www.who.int/whr/2008/en/
http://www.who.int/whr/2008/en/... . In 2004, the Pan American Health Organization (PAHO) engaged in this debate, which emphasized the importance of establishing health promotion policies and/or programs built on participatory methodology and agreement of guiding values and principles for the evaluation of health promotion¹⁰10. World Health Organ (WHO). The World Health Report 2008: Primary Health Care. Geneva: WHO. [acessado 2015 Maio 8]. Disponível em: http://www.who.int/whr/2008/en/
http://www.who.int/whr/2008/en/... .

At the national level, SUS’ performance is evaluated with PROADESS¹¹11. Brasil. Ministério da Saúde (MS). Departamento de Monitoramento e Avaliação do SUS. O Programa de Avaliação para a Qualificação do SUS (PROADESS). Proposta de Avaliação de Desempenho do Sistema de Saúde: indicadores para monitoramento. [relatório final]. Rio de Janeiro: LISIC/ICT/Fiocruz; 2011. and the quality of Primary Health Care services is evaluated with the PMAQ¹²12. Brasil. Ministério da Saúde (MS). Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ). Brasília: MS; 2011.. However, SUS does not have a tool that evaluates its proposal for the qualification of health promotion actions.

Academia is also addressing this challenge. Restrepo¹³13. Restrepo H. Generalidades sobre evaluación de experiências y proyectos de promoción de la salud. In: Restrepo H, Málaga H, organizadores. Promoción de la salud: como construir vida saludable. Bogotá: Editorial Médica Panamericana; 2001. p. 212-217. considers that the evaluation of health promotion should be part of planning and developed through social participation and sustainable actions. Pedrosa⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626. stresses this characteristic of health promotion evaluation when affirming that it has to be participatory, where stakeholders negotiate, agree and decide collectively in order to achieve the desired changes. According to the author⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626., evaluation thought of in this form acts between the established and the transformative, and is transdisciplinary and multicultural, highlighting the need to be institutionalized in order to be effective, since issues, criteria and evaluation parameters are built from the articulation of the evaluation and its object.

These trends are also found in the quality assessment model, the “Systemic Evaluation Model” proposed by Donabedian¹⁴14. Donabedian A. The quality of care: How can it be assessed? Arch Pathol Lab Med 1997; 121(11): 1145-1150.^,1515. Donabedian A. La calidad de la atención médica: definición y métodos de evaluación. México: La Prensa Mexicana; 1984., which is composed of the analysis of the realms: structure, process and results. For each realm several indicators are selected and used jointly with the intention of enabling stakeholders involved in producing health to appropriate the methods and tools both to make a diagnosis about the organization and operation of services and practices and to build intervention projects for the identified challenges.

The challenge of evaluating health promotion programs is even greater when considering the complex health-disease process in areas of socio-environmental vulnerability, where poverty-related diseases coexist with chronic non-communicable diseases, external causes of injury, inadequate sanitation conditions and difficult access to prophylactic measures, including treatment and educational measures¹⁶16. Prata PR. The Epidemiologic Transition in Brazil. Cad Saude Publica 1992; 8(2):168-175.

17. Omram AR. The epidemiologic transition: a theory of the epidemiology of population change. Bulletin of the World Health Organization 2001; 79(2):161-170.

18. Araújo-Jorge T. Embasamento técnico e sugestões para ações de controle das Doenças da Pobreza no Programa de Erradicação da Pobreza Extrema no Brasil. Rio de Janeiro: Instituto Oswaldo Cruz 2011. [Nota Técnica N. 1/2011/IOC-FIOCRUZ/ Diretoria]. [acessado 2017 Ago 30]. Disponível em: http://www.fiocruz.br/ioc/media/NotaTecnica_1_2011_IOC.pdf.
http://www.fiocruz.br/ioc/media/NotaTecn...

19. Malta DC, SilvaJúnior BJ. O Plano de Ações Estratégicas para o Enfrentamento das Doenças Crônicas Não Transmissíveis no Brasil e a definição das metas globais para o enfrentamento dessas doenças até 2025: uma revisão. Epidemiol. Serv. Saúde 2013; 22(1):151-164.^-2020. United Nations Children's Fund (Unicef). Health Equity Report. Analysis of reproductive, maternal, newborn, child and adolescent health inequities in Latin America and the Caribbean to inform policymaking. Summary. Report. New York: Unicef; 2016.. In this context, intestinal parasitic infections (IPI) perpetuate the disease-poverty-disease cycle¹⁸18. Araújo-Jorge T. Embasamento técnico e sugestões para ações de controle das Doenças da Pobreza no Programa de Erradicação da Pobreza Extrema no Brasil. Rio de Janeiro: Instituto Oswaldo Cruz 2011. [Nota Técnica N. 1/2011/IOC-FIOCRUZ/ Diretoria]. [acessado 2017 Ago 30]. Disponível em: http://www.fiocruz.br/ioc/media/NotaTecnica_1_2011_IOC.pdf.
http://www.fiocruz.br/ioc/media/NotaTecn... by impairing cognitive function and school performance and, consequently, employability conditions⁶6. Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Secretaria de Atenção à Saúde. Política Nacional de Promoção da Saúde (PNPS):revisão da Portaria MS/GM n° 687, de 30 de março de 2006. Brasília: MS; 2015.^,2121. Mello Jorge MHP, Koizumi MS, Tono VL. External causes: what they are, how they affect the health sector, how they can be measured, and a few subsidies to prevent them. Revista Saúde UNG 2008; 1(1):37-47.. The impact of IPIs has been largely ignored in Brazil and in other developing countries, neglecting these diseases even more¹⁸18. Araújo-Jorge T. Embasamento técnico e sugestões para ações de controle das Doenças da Pobreza no Programa de Erradicação da Pobreza Extrema no Brasil. Rio de Janeiro: Instituto Oswaldo Cruz 2011. [Nota Técnica N. 1/2011/IOC-FIOCRUZ/ Diretoria]. [acessado 2017 Ago 30]. Disponível em: http://www.fiocruz.br/ioc/media/NotaTecnica_1_2011_IOC.pdf.
http://www.fiocruz.br/ioc/media/NotaTecn... ^,2222. Cordón JA. Dificuldades, Contradições e Avanços, na Inserção da Odontologia no SUS. Divulg. saúde debate 1996; 13:44.^,2323. Smeke ELM, Oliveira NLS. Educação em saúde e concepções de sujeito. In: Vasconcelos EM. A saúde nas palavras e nos gestos: reflexões da rede educação popular e saúde. São Paulo: Hucitec; 2001. p. 115-136..

This study builds on the hypothesis that qualitative-quantitative and multidimensional health indicators contribute to the evaluation of the effectiveness of health promotion programs based on public policies, the promotion of perceived health from its determinants and the expanded conception of the health-disease process³3. Czeresnia D, Freitas, CM, organizadores. Promoção da Saúde: conceitos, reflexões, tendências. 2ª ed. rev. e amp. Rio de Janeiro: Fiocruz; 2009..

Thus, this study aimed to develop an Evaluation Matrix based on the implementation of a health promotion program, with the theme of confronting intestinal parasites (IPs) in vulnerable territories, using the theoretical bases of health promotion, the quality assessment model¹⁴14. Donabedian A. The quality of care: How can it be assessed? Arch Pathol Lab Med 1997; 121(11): 1145-1150.^,1515. Donabedian A. La calidad de la atención médica: definición y métodos de evaluación. México: La Prensa Mexicana; 1984., health planning⁸8. Minayo MCS. Saúde e Ambiente Sustentável: estreitando nós. Rio de Janeiro: Fiocruz; 2011.^,1313. Restrepo H. Generalidades sobre evaluación de experiências y proyectos de promoción de la salud. In: Restrepo H, Málaga H, organizadores. Promoción de la salud: como construir vida saludable. Bogotá: Editorial Médica Panamericana; 2001. p. 212-217. and evaluation⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626. for its construction.

Methodology

This cross-sectional qualitative-quantitative study was carried out through the implementation of a health promotion program in the period of 2013-2015, using participant observation, interviews and census in the 559 households enrolled in the Family Health Strategy (FHS) of the municipality of Laje do Muriaé, RJ, aiming at the construction of the Evaluation Matrix and subsequent testing of its applicability in territories with different realities. Program implementation counted on the partnership of the Municipal Health Secretariat (SMS) and adherence of FHS and Endemic teams of the Municipal Culture and Education Secretariats. Secondary school students from the municipality, members of Scientific Pre-Initiation Program for Young Science Talents, from the Foundation for Research Support of the State of Rio de Janeiro (FAPERJ) also participated in the initiative.

Study areas

The health promotion program was developed in the municipality of Laje do Muriaé, which is located in the northeast of Rio de Janeiro, Brazil (21°12’24”S, 42°7’57”O). It has the second highest Social Vulnerability Index (IVSop = 0.82, 1 being the most vulnerable)²⁴24. Barata MML, Confalonieri UEC, Lima ACLL, Marinho DP, Luigi G, De Simone GC, Ferreira IB, Pinto IV. Vulnerability map of the state's population of Rio de Janeiro to the impacts of climate Buss. [S.l.] :[s.n.]; 2011., the highest rate of declining population in the State (-0.53% per year)²⁵25. Brasil. Instituto Brasileiro de Geografia e Estatística (IBGE). Censo 2010, 2011, 2015. [acessado 2017 Ago 30]. Disponível em: www.ibge.gov.br
www.ibge.gov.br... , the second lowest GDP of the State²⁵25. Brasil. Instituto Brasileiro de Geografia e Estatística (IBGE). Censo 2010, 2011, 2015. [acessado 2017 Ago 30]. Disponível em: www.ibge.gov.br
www.ibge.gov.br... and a large force enrolled in the Bolsa Família (Family Grant) Program (4.208 inhabitants, 60.0%)²⁵25. Brasil. Instituto Brasileiro de Geografia e Estatística (IBGE). Censo 2010, 2011, 2015. [acessado 2017 Ago 30]. Disponível em: www.ibge.gov.br
www.ibge.gov.br... .

Laje Muriaé has a total area of 250 km² and a population of 7,487 inhabitants²⁵25. Brasil. Instituto Brasileiro de Geografia e Estatística (IBGE). Censo 2010, 2011, 2015. [acessado 2017 Ago 30]. Disponível em: www.ibge.gov.br
www.ibge.gov.br... ; 3,126 (42%) have some occupation, of which only 940 (30%) have a formal employment relationship. The municipality has poor sanitation conditions, piped sewage, but which is exhausted in natura in the River Muriaé that traverses all its urban area.

The Matrix was later applied to the Manguinhos Complex of Favelas located in the northern part of the city of Rio de Janeiro (22°52’47.04”S, 43°14’57.18”W), with approximately 40,000 people²⁶26. Ignacio FC, Silva MEC, Handam NB, Alencar MFL, Sotero-Martins A, Barata MML, Moraes Neto AHA. Socioenvironmental conditions and intestinal parasitic infections in Brazilian urban slums: a cross-sectional study. Rev Inst Med Trop São Paulo 2017; 59:e56.. The Complex is composed mostly of salaried, underemployed and/or unemployed workers, poor sanitation and water supply conditions and defective public services²⁷27. Matos M, Bruno P. Programa de Controle da Dengue em Manguinhos (PCDM): reflexões sobre uma experiência coletiva no campo da saúde. In: Matos M, Bruno P. Território, participação popular e saúde: Manguinhos em debate. Rio de Janeiro: Fiocruz; 2010.. Since Manguinhos is not a municipality and does not fit into district boundaries, the Manguinhos Complex of Favelas has limited official population data. However, it is a region with issues related to poverty and increasing violence²⁸28. Lima CM, Bueno LB, organizadores. Território, participação popular e saúde: Manguinhos em debate. Rio de Janeiro: Fiocruz; 2010. p. 51-60.. Communities also coexist with air, water and soil contamination resulting from polluting industries, namely, the Manguinhos refinery.

Both locations have determinant environmental conditions in the development of health-disease processes.

Stages of the Health Promotion Program

The program carried out in Laje do Muriaé was structured in three stages (Figure 1):

1. Situational diagnosis (pre-test), aiming to identify the frequency and profile of intestinal parasitoses; knowledge, attitudes and practices about IPs and socio-environmental conditions²⁹29. Moraes Neto AHA, Pereira APMF, Alencar MFL, Souza-Júnior PRB, Dias RC, Fonseca JG, Santos CP, Almeida JCA. Prevalence of intestinal parasites versus knowledge, attitudes, and practices of inhabitants of low-income communities of Campos dos Goytacazes, RJ. Parasitol Res 2010; 107(2):295-307..

2. Educational intervention: health education actions with the population based on the identified problem situation, using integrative practices that consider local culture and knowledge and continuing education actions.

3. Epidemiological, socio-environmental and educational re-evaluation (post-test)²⁹29. Moraes Neto AHA, Pereira APMF, Alencar MFL, Souza-Júnior PRB, Dias RC, Fonseca JG, Santos CP, Almeida JCA. Prevalence of intestinal parasites versus knowledge, attitudes, and practices of inhabitants of low-income communities of Campos dos Goytacazes, RJ. Parasitol Res 2010; 107(2):295-307. and the evaluation of the quality of drinking water and peridomiciliar soil³⁰30. Sotero-Martins A, Duarte NA, Carvajal E, Sarquis MIM, Fernandes OCC. Microbiological and Quality Control of Recreation Areas. Rev. Eletrónica Gestão & Saúde 2013; 4(1):1075-1092.; with distribution of educational material on the proper care of drinking water, water reservoirs and filters.

At all stages, observations made with the population and the FHS team were recorded²⁸28. Lima CM, Bueno LB, organizadores. Território, participação popular e saúde: Manguinhos em debate. Rio de Janeiro: Fiocruz; 2010. p. 51-60., aiming to identify qualitative-quantitative indicators in a multidimensional participatory perspective for the construction of an Evaluation Matrix.

All stages were also carried out in partnership with community health workers. Participants were enrolled through Informed Consent Form (ICF) signed by the legal representative of the family. The partnership with the SMS started after program objectives were agreed, when the Deed of Undertaking and Terms of Agreement were signed with the managers and the FHS team, respectively.

Building the Evaluation Matrix

Monitoring of the health promotion program adapted from Moraes Neto et al.²⁹29. Moraes Neto AHA, Pereira APMF, Alencar MFL, Souza-Júnior PRB, Dias RC, Fonseca JG, Santos CP, Almeida JCA. Prevalence of intestinal parasites versus knowledge, attitudes, and practices of inhabitants of low-income communities of Campos dos Goytacazes, RJ. Parasitol Res 2010; 107(2):295-307. allowed for the categorization of the program's stages in the following realms: structure, processes and results¹⁴14. Donabedian A. The quality of care: How can it be assessed? Arch Pathol Lab Med 1997; 121(11): 1145-1150.^,1515. Donabedian A. La calidad de la atención médica: definición y métodos de evaluación. México: La Prensa Mexicana; 1984., and the identification of the need to map the work processes of each realm¹⁴14. Donabedian A. The quality of care: How can it be assessed? Arch Pathol Lab Med 1997; 121(11): 1145-1150.^,1515. Donabedian A. La calidad de la atención médica: definición y métodos de evaluación. México: La Prensa Mexicana; 1984.. The analysis of this mapping allowed us to outline the flowchart of the stages of the aforementioned program³¹31. Bethlem AS. Estratégia Empresarial: conceitos, processo e Administração Estratégica. 4ª ed. São Paulo: Atlas; 2002. (Figure 1) and to identify the efficiency and effectiveness³²32. Lima GBA, Carvalho NC, Herkenhoff DA. Avaliação de desempenho baseada na ISO 9004:2000: Estudo de caso em uma empresa de manutenção. INGEPRO - Inovação, Gestão e Produção 2010; 2(6)97-107. of each realm (strengths and weaknesses)³¹31. Bethlem AS. Estratégia Empresarial: conceitos, processo e Administração Estratégica. 4ª ed. São Paulo: Atlas; 2002.^,3232. Lima GBA, Carvalho NC, Herkenhoff DA. Avaliação de desempenho baseada na ISO 9004:2000: Estudo de caso em uma empresa de manutenção. INGEPRO - Inovação, Gestão e Produção 2010; 2(6)97-107., thus making the necessary adjustments. These results allowed to evaluate the articulation between structure, processes and results³²32. Lima GBA, Carvalho NC, Herkenhoff DA. Avaliação de desempenho baseada na ISO 9004:2000: Estudo de caso em uma empresa de manutenção. INGEPRO - Inovação, Gestão e Produção 2010; 2(6)97-107. and to identify the two additional realms not proposed by the systemic model of Donabedian¹⁴14. Donabedian A. The quality of care: How can it be assessed? Arch Pathol Lab Med 1997; 121(11): 1145-1150.^,1515. Donabedian A. La calidad de la atención médica: definición y métodos de evaluación. México: La Prensa Mexicana; 1984.:

1. Context, since the program started with negotiation and agreement with the FHS team and stakeholders;

2. Continuity, due to the discontinuity of the program and its non-inclusion in the Municipal Health Plan of Laje do Muriaé.

It also allowed for the realization that the program considered nine principles of the PNPS⁶6. Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Secretaria de Atenção à Saúde. Política Nacional de Promoção da Saúde (PNPS):revisão da Portaria MS/GM n° 687, de 30 de março de 2006. Brasília: MS; 2015. and, therefore, that the construction of research descriptors should be bound to these principles.

Evaluation tool (questionnaire)

The evaluation tool (questionnaire) was prepared through the following:

1. Identification of 23 evaluation descriptors linked to the principles of the policy (Chart 1);

2. Allocation of questions for each descriptor;

3. Establishing the typology of these questions:

   1. –

      Closed with scale: Yes, No, I don't know;

   2. –

      Semi-open that allow for the addition of comments and;

   3. –

      Developed from some important issues in order to obtain better information quality³²32. Lima GBA, Carvalho NC, Herkenhoff DA. Avaliação de desempenho baseada na ISO 9004:2000: Estudo de caso em uma empresa de manutenção. INGEPRO - Inovação, Gestão e Produção 2010; 2(6)97-107..

Validation of applicability

The validation process of the evaluation tool (questionnaire) used micro and macro analyses based on its applicability. The tool was evaluated through the adapted Delphi Technique, with the collaboration of specialists in education, health promotion and health services using face-to-face meetings³³33. Pinheiro R, Silva Júnior AG. Práticas avaliativas e as mediações com a integralidade na saúde: uma proposta para estudos de processos avaliativos na atenção básica. In: Pinheiro R, Silva Júnior AG, Mattos RA, organizadores. Atenção básica e integralidade: contribuições para estudos de práticas avaliativas em saúde. Rio de Janeiro: CEPESC; 2008. p. 17-42. (Figure 2).

Submission to the Delphi Technique generated reformulations of the evaluation questionnaire³³33. Pinheiro R, Silva Júnior AG. Práticas avaliativas e as mediações com a integralidade na saúde: uma proposta para estudos de processos avaliativos na atenção básica. In: Pinheiro R, Silva Júnior AG, Mattos RA, organizadores. Atenção básica e integralidade: contribuições para estudos de práticas avaliativas em saúde. Rio de Janeiro: CEPESC; 2008. p. 17-42., with each specialist contributing to improve the tool's applicability:

1. The Education specialist carried out an indepth analysis of the tool by broadening its qualitative character;

2. The health promotion specialist shortened the questionnaire, implementing improvements in the statement and reducing the application time to approximately 15 minutes;

3. The health service specialist stimulated the scope of the issues in order to adapt the instrument according to the needs of different territories with varying epidemiological profiles.

The differences in focus of analyses were not stimulated and emerged spontaneously from the evaluators. After obtaining the consensus of experts, the questionnaire that composed the Evaluation Matrix was established.

Then, questions received scores:

1. Closed questions with scale: yes = 2, no = 1, I don't know = 0;

2. Developing questions, one (1) point for each selected option;

3. Semi-open questions: one (1) point for each element added;

4. Time of application of questionnaires following the educational action:

Intervals and the classification of scores obtained with the application of the Evaluation Matrix (Chart 2) were established following the assignment of scores.

Theoretical and potential alignment of the Evaluation Matrix Intervention

The Evaluation Matrix's potential for intervention and alignment with the theoretical foundations was discussed and evaluated in three workshops, which included the participation of public health researchers, FHS health professionals and municipal health managers.

Workshop 1: Theoretical alignment

* The Relevance of Evaluation and Monitoring of Health Promotion Programs, within the framework of the IOC/FIOCRUZ Study Center: with the presentation of three lectures and discussion by the plenary.

1. –

   Health promotion and creativity.

2. –

   The evaluation and monitoring of health program and the management of cities.

3. –

   National Program for Improving Access and Quality of Primary Health Care (PMAQ).

Workshops 2 and 3: Potential for intervention

* Health promotion programs: relevance of evaluation and monitoring for public management.

* Health promotion programs: relevance of evaluation and monitoring for the population.

Workshops 2 and 3 were recorded and suggestions were registered by two rapporteurs and read at the end of the plenary. They were then incorporated into the Evaluation Matrix (Chart 2) following the participants’ approval.

The Evaluation Matrix was then applied to the coordinator and to the health workers who participated in the programs carried out in Laje do Muriaé and the Manguinhos Complex of Favelas, the latter by free choice of the FHS team of the Victor Valla Family Clinic, with authorization from the Education and Research Coordination of the Germano Sinval Faria School Health Center, National School of Public Health, Fiocruz, RJ. The results of these applications were submitted to a descriptive analysis (distribution of variables’ frequencies).

Ethical Considerations

The Fiocruz Ethics Committee on Human Research: CEP/IOC/Fiocruz 2013 approved this study in the Plataforma Brasil Platform.

Results

Observations recorded during the implementation of the health promotion program in Laje do Muriaé identified community health workers as the FHS health professionals involved in all of the stages of the program and with the greatest proximity to the population. The other members of the health team adhered to the program; however, they remained within the health facility, providing support for health promotion actions directed by community health workers. In 2015, endemic disease agents joined the FHS³⁴34. Brasil. Presidência da República. Casa Civil. Decreto n° 8474, de 22 de junho de 2015. Dispõe sobre as atividades de Agente Comunitário de Saúde e de Agente de Combate às Endemias. Diário Oficial da União 2015; 24 jun. team and, therefore, joined the program in stage two, participating in health education and continuing education actions, and stage three (in the posttest), in the collection of drinking water and peridomicile soil for physical-chemical and cholimetric analyses.

Coordinators and managers of the municipality of Laje do Muriaé supported the actions by increasing the governability of health workers to implement the program and solve problem situations, such as positive results for Taenia sp, which required specific actions by the FHS teams. The population was actively involved throughout the program, participating in the planned activities and seeking continuity with the health workers. The inclusion of secondary school students of the Pre-Scientific Initiation Program favored the population's adherence, since they were FHS users.

The fact that the educational activities depended on the participation of the population, managers and stakeholders and reflected the specific characteristics of the study area, facilitated the construction of knowledge by all stakeholders and favored the joint identification of mechanisms for the promotion, prevention and control of the positive association between A. lumbricoides and Taenia sp (2/11 (18.2%) vs. 5/764 (0.7%), p = 0.003 (Fisher's exact test).

The analysis of residents’ knowledge, attitudes and practices (KAP) in the post-test, when compared to pre-test showed that 88.3% considered IPIs as “disease / contamination caused by worms”, and no more: “It's natural, everyone has it, we're already born with it”. The result of the post-test epidemiological profile was better than the pre-test, since no association between parasites (A. lumbricoides and Taenia sp) was observed.

However, although 99.3% of the households received treated water, participant observation revealed that the population had a habit of drinking water from a well or a mine, and therefore, the post-test drinking water and peridomicile soil sample analysis was performed, showing that 84.2% of the water collected in households, water mines and wells were unfit for human consumption due to evidence of fecal coliforms E. coli and/or Salmonella sp. The prevalence of IPs in the soil was 82.7%.

The Evaluation Matrix proved to be easy to apply and facilitated the detection of strengths and weaknesses of health promotion programs carried out in territories with different characteristics: Laje do Muriaé and the Manguinhos Complex of Favelas.

The main strengths and weaknesses identified in the evaluation carried out by the Laje do Muriaé team were:

The Evaluation Matrix's application time with the Laje do Muriaé team averaged 15 minutes. The mean score was 102 (78.5%), corresponding to the classification “Very good”.

The main strengths and weaknesses identified in the evaluation carried out by the Manguinhos Complex of Favelas team were:

The Evaluation Matrix's application time with the Manguinhos Complex of Favelas team averaged 13 minutes. The mean score was 75 (57.7%), corresponding to the classification “Good”.

Discussion

The elaboration of the Evaluation Matrix built on the hypotheses that qualitative-quantitative and multidimensional health indicators contribute to the evaluation of the effectiveness of health promotion programs and that these programs are effective in the promotion, prevention and control of poverty-related diseases. It also considered that the construction of health indicators should be based on social reality⁸8. Minayo MCS. Saúde e Ambiente Sustentável: estreitando nós. Rio de Janeiro: Fiocruz; 2011., participation and negotiation among stakeholders as suggested by Pedrosa⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626. and Restrepo¹³13. Restrepo H. Generalidades sobre evaluación de experiências y proyectos de promoción de la salud. In: Restrepo H, Málaga H, organizadores. Promoción de la salud: como construir vida saludable. Bogotá: Editorial Médica Panamericana; 2001. p. 212-217..

Pinheiro & Silva-Junior³³33. Pinheiro R, Silva Júnior AG. Práticas avaliativas e as mediações com a integralidade na saúde: uma proposta para estudos de processos avaliativos na atenção básica. In: Pinheiro R, Silva Júnior AG, Mattos RA, organizadores. Atenção básica e integralidade: contribuições para estudos de práticas avaliativas em saúde. Rio de Janeiro: CEPESC; 2008. p. 17-42. affirmed the need for a collaborative relationship between managers and evaluators for the institutionalization of health promotion programs. The results of the application of the Evaluation Matrix in the two locations also showed the importance of the adherence of health managers and stakeholders to the negotiations, but also the need for their inclusion in the Municipal Health Plan in order to ensure their sustainability and consequent institutionalization.

Pedrosa⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626. argues that the lack of institutionalization occurs because municipal managers fail to consider health promotion as a public policy, since they believe that treatment is what matters, understanding health promotion as a chronologically anticipated prevention³⁵35. Carvalho AI, Bodstein RC, Hartz Z, Matida AH. Concepções e abordagens na avaliação em promoção da saúde. Cien Saude Colet 2004; 9(3):521-529.. Felisberto³⁶36. Felisberto E. Monitoramento e avaliação na atenção básica: novos horizontes. Rev. Brasileira de Saúde Materno Infantil 2004; 4(3):317-321. says that institutionalizing evaluations should be understood as contributing to the quality of Primary Health Care. It helps to trigger a process for building knowledge and better practices, promoting the construction of structured and systematic processes consistent with the principles of SUS, in its various realms of management, care and intervention in the territory.

The proposal of a Health Promotion Evaluation Matrix has been discussed by WHO and authors such as Pedrosa⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626., who, in his proposed health promotion evaluation, identified the construction of three major matrices: health promotion linked to the conception of epidemiological and social risk, health promotion articulated to broad proposals for sustainable development, and a third, intermediate matrix focused on actions of intersectoriality, active participation of the population with a priority for local development⁷7. Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626.. The Evaluation Matrix shown in this paper synthesizes the three strands of the proposed Matrices.

It also is in line with the discussion established by Carvalho et al.³⁵35. Carvalho AI, Bodstein RC, Hartz Z, Matida AH. Concepções e abordagens na avaliação em promoção da saúde. Cien Saude Colet 2004; 9(3):521-529. based on the challenges associated with the conceptual field of Health Promotion and the requirements of evidence of effectiveness and efficiency faced by managers, evaluators and agents in the development of intersectoral health actions. These authors understand that the need for evidence in public policies can lead to an increased gap between complex health promotion interventions and their conceptions, and that the determination of changes in this area requires collaborative processes that consider the stakeholders involved in the implementation and evaluation of interventions.

Thus, the evaluation cannot be a purely technical procedure, it must have a methodological design which aims for social stakeholders to participate with decision-making power and for the evaluation results to be incorporated into management³⁷37. Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z. Avaliação: conceitos e métodos. Rio de Janeiro: Editora Fiocruz; 2011.^,3838. Akerman M, Mendes R, Bógus CM. É possível avaliar um imperativo ético? Cien Saude Colet 2004; 9(3):605-615., allowing managers to get acquainted with different problem situations and contribute to the well-being and improvement of the quality of life of the population. The second stage of the program (educational intervention) held in Laje do Muriaé, with workshops and round tables with the population, stakeholders, health managers and the FHS team brought to the forefront the need to outline the methodological design of the program in order to consolidate the results and favor the identification of qualitative-quantitative and multidimensional indicators for the Evaluation Matrix³⁷37. Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z. Avaliação: conceitos e métodos. Rio de Janeiro: Editora Fiocruz; 2011.^,3939. Thiollent M. Pesquisa-ação nas organizações. 2ª ed. São Paulo: Atlas; 2009..

We could also observe that the three stages of the program carried out in Laje do Muriaé, in addition to contributing to the epidemiological diagnosis in the six micro areas covered by the local Primary Health Care services, the educational intervention, treatment and follow-up, further strengthened interactions between the population and the FHS team, as well as those with the local managers. In considering the results of the socio-environmental, educational and epidemiological surveys in the shared construction of educational actions, the program linked the specific characteristics of the territory to health education and to the preparation of the teams to engage in health education activities during home visits and throughout the entire process of care²³23. Smeke ELM, Oliveira NLS. Educação em saúde e concepções de sujeito. In: Vasconcelos EM. A saúde nas palavras e nos gestos: reflexões da rede educação popular e saúde. São Paulo: Hucitec; 2001. p. 115-136..

Besen et al.⁴⁰40. Besen CB, Souza Netto M, Ros MA, Silva FW, Pires MF. The Family Health Strategy as an object of Health Education. Saúde Soc 2007; 16(1):57-68. say that the FHS does not yet include health education as a major focus, since managers are unaware of the health promotion rationale and most have vertical educational practices, in a short-sighted and curativist relationship with the population, still inserted in the biomedical model.

Rootman et al.⁴¹41. Rootman I, Goodstadt M, Hyndman B, McQueen DV, Potvin L, Springett J, Ziglio E. Evaluation in Health Promotion: Principles and Perspectives. Copenhagen: World Health Organization; 2001. stressed that there is a need to use multiple strategies to promote health and it would be necessary to be supported by principles of empowerment, integrality, participation, intersectoriality, equity and sustainability, principles found in the 2006 National Health Promotion Policy, which were extended with the introduction of intrasectoriality, territoriality and autonomy principles when reformulated in 2015⁶6. Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Secretaria de Atenção à Saúde. Política Nacional de Promoção da Saúde (PNPS):revisão da Portaria MS/GM n° 687, de 30 de março de 2006. Brasília: MS; 2015.. This need to have the policy's principles as a strategic basis in the determination of evaluation indicators was identified at the end of the implementation of the program carried out in a participatory manner with all the segments involved in local Primary Health Care and facilitated the establishment of the Evaluation Matrix based on 23 descriptors elaborated from the nine principles of said Policy⁶6. Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Secretaria de Atenção à Saúde. Política Nacional de Promoção da Saúde (PNPS):revisão da Portaria MS/GM n° 687, de 30 de março de 2006. Brasília: MS; 2015. and linked to the realms of the expanded systemic model of Donabedian¹⁴14. Donabedian A. The quality of care: How can it be assessed? Arch Pathol Lab Med 1997; 121(11): 1145-1150.^,1515. Donabedian A. La calidad de la atención médica: definición y métodos de evaluación. México: La Prensa Mexicana; 1984., aiming to enable the use of multiple strategies required to meet the characteristics of each territory, integrated by different vulnerable groups.

Thus, the Evaluation Matrix was established, corroborating the hypothesis that qualitative-quantitative and multidimensional health indicators contribute to the evaluation of the effectiveness of health promotion programs and that the participatory and integrated evaluation process has the potential to boost the management of quality and to promote the internalization of good public management practices and the continuous improvement of the teams’ work processes⁸8. Minayo MCS. Saúde e Ambiente Sustentável: estreitando nós. Rio de Janeiro: Fiocruz; 2011., aiming at confronting situations of exclusion and inequities, based on the establishment of the determinants of the disease-health process.

Conclusion

The rapid urbanization of cities evidences important situations of exclusion and great inequities. The development of health promotion programs and evaluation of its effectiveness are essential strategies for the promotion of equity and the right to the city.

Thus, contributions of this work include a health promotion program model and an evaluation methodology, namely, the Evaluation Matrix, built from a case study in a municipality with high socio-environmental vulnerability and which has been shown to be applicable in both territories with different characteristics by favoring:

1. –

   Self-assessment by teams and identification of weaknesses in their work processes;

2. –

   The identification of individual and collective health problems and the consequent need to act with integrity and equity;

The Evaluation Matrix mainly showed the potential to boost quality management and favor the internalization of good public management practices and the continuous improvement of work processes, aiming to cope with situations of exclusion and inequities from the identification of the determinants of the health-disease process.

Acknowledgments

This study was funded by the Foundation for Research Support of the State of Rio de Janeiro (FAPERJ), Coordination for the Improvement of Higher Education Personnel (CAPES), and by the Laboratory of Innovations in Therapies, Education and Bioproducts - LITEB/IOC/Fiocruz. We would like to thank Anna Cristina Calçada Carvalho, Diana Pinheiro Marinho, Elisete Casotti, Felipe de Carvalho Vommaro Marincola, Júlio Cesar Pegado Bordignon, Lívia Marinho da Silva, Maria de Fátima Leal Alencar, Patrícia Ocampo, Sandra Escovedo Selles and Tania Cremonini de Araújo Jorge.

References

Publication Dates

History