Abstract

Objective

To develop and validate of an instrument to evaluate interventions in relation to Health Promotion principles in Brazil.

Methods

A cross-sectional study of the development and validation of an instrument was carried out, based on the assessment of the first version by 20 health workers, the final version by 19, analysis of reliability by 31, and content evaluation of the first version by ten experts and of the final version by 12 experts. The content validity index, Cronbach’s alpha (α) and intraclass correlation coefficient were used.

Results

The indicators achieved 89.9% adequacy, 82.5% clarity, internal consistency was α = 0.80, and the test-retest correlation was 0.93.

Conclusion

The instrument showed acceptable validity and reliability and can be used for the evaluation of Health Promotion interventions.

Keywords:
Health Promotion; Validity Studies; Surveys and Questionnaires; Health Care Models

Introduction

Health Promotion is an expression that holds many concepts, associated with diverse and wide-ranging values and principles, such as quality of life, solidarity, democracy, citizenship, development, social participation and intersectoriality, among others.¹1. Sicoli JL, Nascimento PR. Promoção de saúde: concepções, princípios e operacionalização. Interface (Botucatu). 2003;7(12):101-22. doi: http://doi.org/10.1590/S1414-32832003000100008.
http://doi.org/10.1590/S1414-32832003000...

2. Heidmann ITSB, Almeida MCP, Boehs AE, Wosny AM, Monticelli M. Promoção à saúde: trajetória histórica de suas concepções. Texto contexto- Enferm. 2006;15(2): 352-8. doi: http://doi.org/10.1590/S0104-07072006000200021.
http://doi.org/10.1590/S0104-07072006000...

3. Czeresnia D, Freitas CM, organizadores. Promoção da saúde: conceitos, reflexões, tendências. Rio de Janeiro: Editora Fiocruz; 2009.

4. Westphal MF. Promoção da saúde e prevenção de doenças. In: Campos GWS, Minayo MCS, Akerman N, Drumond Júnior M, Carvalho YM, organizadores. Tratado de saúde coletiva. 2a ed. São Paulo: Hucitec; 2014. p. 681-717.

5. Bezerra IMP, Sorpreso ICE. Concepts and movements in health promotion to guide educational practices. J Hum Growth Dev. 2016;26(1):11-16. doi: http://doi.org/10.7322/jhgd.113709.
http://doi.org/10.7322/jhgd.113709....

6. Mendes R, Fernandez JCA, Sacardo DP. Promoção da saúde e participação: abordagens e indagações. Saude Debate. 2016; 40(108):190-203. doi: http://doi.org/10.1590/0103-1104-20161080016.
http://doi.org/10.1590/0103-1104-2016108...

7. Souza RKT, Loch MR. Promoção da saúde. In: Andrade SM, Cordoni Júnior L, Carvalho BG, Gonzáles AD, Silva AMR, organizadores. Bases da saúde coletiva. 2ª ed. Londrina: 2017. p. 29-46.

8. Colin ECS, Pelicioni CF. Territorialidade, desenvolvimento local e promoção da saúde: estudo de caso em uma vila histórica de Santo André, São Paulo. Saude Soc. 2018;27(4):1246-60. doi: http://doi.org/10.1590/s0104-12902018170850.
http://doi.org/10.1590/s0104-12902018170... ^-99. Chiari APG, Ferreira RC, Akerman M, Amaral JHL, Machado KM, Senna MIB. Rede intersetorial do Programa Saúde na Escola: sujeitos, percepções e práticas. Cad. Saude Publica. 2018.34(5):e00104217. doi: http://doi.org/10.1590/0102-311x00104217.
http://doi.org/10.1590/0102-311x00104217... In Brazil, an important milestone for the institutionalization of Health Promotion was the implementation of the National Health Promotion Policy, published in 2006 and reformulated in 2014.¹⁰10. Brasil. Portaria n. 2446, de 11 de novembro de 2014. Redefine a Política Nacional de Promoção da saúde (PNPS). Brasília, DF: Diário Oficial da União. 13 nov. 2014 [acesso 18 jun. 2019]: Seção 1, p. 68. Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2014/prt2446_11_11_2014.html .
http://bvsms.saude.gov.br/bvs/saudelegis...

11. Malta DC, Reis CAA, Jaime PC, Moraes Neto OL, Silva MMA, Akerman Met. O SUS e a política nacional de promoção da saúde: perspectiva resultados, avanços e desafios em tempos de crise. Cienc Saude Colet. 2018;23(6):1799-809. doi: http://doi.org/10.1590/1413-81232018236.04782018.
http://doi.org/10.1590/1413-81232018236.... ^-1212. Rocha DG, Alexandre VP, Marcelo VC, Rezende R, Nogueira JD, Sá RF. Processo de revisão da política nacional de promoção da saúde: múltiplos movimentos simultâneos. Cienc Saude Colet . 2014;19(11):4313-22. doi: http://doi.org/10.1590/1413-812320141911.11232014.
http://doi.org/10.1590/1413-812320141911... The main objective of the Policy is to,

And it intends to trigger changes in the ways of organizing, planning, performing, analyzing and evaluating health work.¹³13. Malta DC, Morais Neto OL, Silva MMA, Rocha D, Castro AM, Reis AAC, et al. Política nacional de promoção da saúde (PNPS): capítulos de uma caminhada ainda em construção. Cienc Saude Colet . 2016.21(6):1683-94. doi: http://doi.org/10.1590/1413-81232015216.07572016.
http://doi.org/10.1590/1413-81232015216....

The various stakeholders involved in interventions that seek to promote health need to consider and seek to promote certain principles. Creating instruments that help in the evaluation of interventions aimed at promoting health can be of great relevance, including for assessing the greater or lesser closeness of each intervention to the principles of Health Promotion. Therefore, it is necessary to guarantee a process whereby these instruments are adequately developed and validated jointly with the target group.¹⁴14. DeVellis RF. Scale development: theory and applications, 4th ed. publisher unknown: SAGE: 2017.

The objective of this study was to develop and validate an instrument to evaluate how close interventions that seek to promote health are to the principles of the Brazilian National Health Promotion Policy. The instrument’s target group is comprised of health workers who undertake interventions focused on Health Promotion. It is expected that they themselves evaluate these programs and analyze their interventions and their closeness to the principles of the National Health Promotion Policy. And, considering the specificities of each intervention and context, that they adapt their action programs for them to be aligned with those principles.

Methods

This was an observational study, with a descriptive and cross-sectional design. The context in which the study was conducted was Primary Health Care (PHC) in Brazil, characterized by health workers and residents working directly with health promotion in PHC. To ensure greater representativeness of the opinion of health professionals, we chose to include different categories of them, distributed over different regions and cities of the country. Psychometrics was used as a methodological reference. This is a science aimed at measuring phenomena that cannot be observed directly, although they can be characterized using other measurable attributes.¹⁴14. DeVellis RF. Scale development: theory and applications, 4th ed. publisher unknown: SAGE: 2017.

The sample selection was intentional, to ensure the participation of experts in the content of the instrument to be developed. The sample size was defined according to indications found in the literature.¹⁴14. DeVellis RF. Scale development: theory and applications, 4th ed. publisher unknown: SAGE: 2017. Figure 1 show the methodological steps adopted in the study.

Stage 1

Structuring of the theoretical basis of the scale and elaboration of the instrument’s items

In this step, we sought to identify the principles of Health Promotion and design their respective indicators.

To identify these principles, the National Health Promotion Policy¹⁰10. Brasil. Portaria n. 2446, de 11 de novembro de 2014. Redefine a Política Nacional de Promoção da saúde (PNPS). Brasília, DF: Diário Oficial da União. 13 nov. 2014 [acesso 18 jun. 2019]: Seção 1, p. 68. Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2014/prt2446_11_11_2014.html .
http://bvsms.saude.gov.br/bvs/saudelegis... and the Charters of International Conferences on Health Promotion¹⁵15. World Health Organization. The Ottawa charter for health promotion: the first international conference on health promotion, 1986 [acesso 7 fev 2020] Geneva: WHO; [1987?]. Disponível em: https://www.who.int/healthpromotion/conferences/previous/ottawa/en /
https://www.who.int/healthpromotion/conf...

16. World Health Organization. Adelaide recommendations on healthy public policy 1988. [acesso 7 fev. 2020] Disponível em: http://www.who.int/hpr/docs/adelaide.html
http://www.who.int/hpr/docs/adelaide.htm...

17. World Health Organization. Sundsvall Statement on supportive environments for health 1991 [acesso 7 fev. 2020] Disponível em: https://archive.org/details/WHOHED92.1/page/n2/mode/2up
https://archive.org/details/WHOHED92.1/p...

18. World Health Organization. Jakarta declaration on leading health promotion into the 21st Century 1997. Rev Salud Publica. 1998 [acesso 7 fev. 2020]];3(1):58-61. Disponível em: https://scielosp.org/pdf/rpsp/1998.v3n1/58-61/en
https://scielosp.org/pdf/rpsp/1998.v3n1/...

19. [Organização Mundial da Saúde]. Declaração do México promoção da saúde: rumo à maior equidade, 2000 [acesso 7 fev. 2020]. Disponível em: http://bvsms.saude.gov.br/bvs/publicacoes/declaracao_mexico_2000.pdf
http://bvsms.saude.gov.br/bvs/publicacoe...

20. World Health Organization. The Bangkok charter for health promotion in a globalized world 2005 [acesso 7 fev. 2020]. Disponível em: http://www.iasaude.pt/attachments/article/66/Carta_de_BangKoK-2005.pdf
http://www.iasaude.pt/attachments/articl...

21. World Health Organization. Nairobi call to action, 2009. Geneva: WHO ; 2009 [acesso 7 fev. 2020] Disponível em: http://www.ngos4healthpromotion.net/wordpressa4hp/wp-content/uploads/2017/04/Nairobi_Call_to_Action_Nov09.pdf
http://www.ngos4healthpromotion.net/word... ^-2222. World Health Organization. The Helsinki statement on health in all policies: the 8th global conference on health promotion, 2013. Geneva: WHO ; 2013 [acesso 7 fev. 2020]. Disponível em: http://www.who.int/healthpromotion/conferences/8gchp/en /.
http://www.who.int/healthpromotion/confe... were used. A first version of the instrument was designed, containing 19 indicators based on 16 principles: (i) Equity; (ii) Social Participation; (iii) Autonomy; (iv) Empowerment; (v) Intrasectoriality; (vi) Intersectoriality; (vii) Sustainability; (viii) Integrality; (ix) Territoriality; (x) Popular Education in Health; (xi) Guideline Comprehensiveness; (xii) Reorientation of Health Services; (xiii) Culture of Peace and Human Rights; (xiv) Continuing Education; (xv) Creating Favorable Environments; and (xvi) Positive Approach.

Stage 2

Administration of the initial version of the instrument with the target group (health intervention workers)

In this stage, the objective was to verify the target group’s understanding of the central idea of the instrument and their impressions regarding its applicability, before content validation (a later stage). The initial instrument was administered with a second year Family Health Residency class at the Londrina State University - ten residents from eight different health areas (Nursing, Physical Education, Pharmacy, Physiotherapy, Nutrition, Dentistry, Psychology and Social Work, being two doing initial training in nursing, two in dentistry and one from each of the other areas mentioned) - as well as with ten health workers from the Pernambuco State Health Department. Each worker was asked to choose an intervention in which he/she worked and to evaluate the intervention according to each of the instrument’s indicators, using a scale from 1 (definitely not in keeping with the indicator) to 4 (definitely in keeping with the indicator), and to make comments, criticisms, and suggestions about the instrument’s clearness, in particular. After individual evaluation by the workers, a conversation circle was coordinated in which more suggestions were presented, it being possible to verify the understanding of the target group as to the central idea of the instrument and thus confirm its usefulness in the evaluation of health interventions. Based on the suggestions made, changes were made to the initial version.

Stage 3

Content validation - analysis of the instrument by specialists in Health Promotion regarding the adequacy and clearness of the principles and respective indicators

First of all, ten specialists in Health Promotion were invited, all university teachers with PhDs and production in the area (three with initial training in dentistry, two in physical education, two in nursing, two in medicine and one in psychology), to whom the instrument was sent by email linked to a googleforms@ spreadsheet. Some of these experts were chosen by convenience; others were selected by searching for authors in Public Health journals, using the descriptor ‘Health Promotion’.

In the introductory text sent to the specialists, we sought to make it clear that the instrument’s target group are health workers who work in interventions that seek to promote health and that, when answering the instrument, they should select a specific intervention (for example: shared care provision, physical activity/bodily practices group, nutrition education group, smoking group, home visit, etc.) to be evaluated in relation to its level of closeness to the instrument’s indicators. The specialists were then asked to evaluate:

For items ‘a’ and ‘b’, the score ranged from 1 (very inadequate) to 4 (very adequate), and for item ‘c’, from 1 (very insufficient) to 4 (very sufficient). There was also a space on the instrument for specialists to write suggestions, comments and criticisms about each principle and its respective indicator(s).

The results of the quantitative part, as well as the comments made by the specialists, guided two main decisions: keeping only the principles that are contained in the National Health Promotion Policy;¹⁰10. Brasil. Portaria n. 2446, de 11 de novembro de 2014. Redefine a Política Nacional de Promoção da saúde (PNPS). Brasília, DF: Diário Oficial da União. 13 nov. 2014 [acesso 18 jun. 2019]: Seção 1, p. 68. Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2014/prt2446_11_11_2014.html .
http://bvsms.saude.gov.br/bvs/saudelegis... and having only one indicator for each principle. These changes aimed to make the instrument simpler and more direct, so that it could be better understood by the target group and have greater applicability. Thus, a new version of the instrument was developed.

Considering the changes made to the instrument, we decided to hold a new consultation with the specialists. All ten specialists who participated in the previous stage were invited again. As only eight responded, we decided to invite four more specialists, totaling 12 (three with initial training in dentistry, five in physical education, two in nursing, and two in medicine) who evaluated the new version. These specialists worked in eight different Brazilian states, Amazonas, Paraíba, Paraná, Pernambuco, Rio de Janeiro, Rio Grande do Sul, Santa Catarina, and São Paulo, as well as in the Federal District.

Stage 4

Face validity - administration of the final version of the instrument with the target group (health intervention workers)

The final version of the instrument was administered with 19 health residents, specifically from the multiprofessional residencies in Public Health and Family Health of the Recife Health Department. The residents had different backgrounds: biology (n=9), physical education (n=2), nursing (n=5), pharmacy (n=7), physiotherapy (n=1), speech therapy (n=1), veterinary medicine (n=10), nutrition (n=6), dentistry (n=8), social work (n=3) and occupational therapy (n=3).

In this stage, each participant was initially asked to individually evaluate an intervention in which they were participating. In a second step, the participants were divided into groups of three to five people and asked to choose an intervention (all group members should being working in this intervention) and discuss the intervention’s level of closeness to each indicator. As the indicators are presented in an affirmative manner, the residents/workers were asked to rate each one on a scale of 1 (strongly disagree) to 4 (strongly agree).

Each group was also asked to present the reasons for each answer and discuss possible actions/strategies to be taken to improve the indicators that received the most negative evaluations. In the final part of this stage, a conversation circle was held based on a script with two questions:

Was the instrument useful for identifying the points that were “far from” and “close” to the principles of Health Promotion in the intervention you evaluated?

Was the instrument useful for promoting the definition of strategies to improve the intervention and bring it closer to the principles of Health Promotion?

Everything that was said was recorded and later transcribed for analysis.

Stage 5

Reliability - analysis of internal consistency and temporal stability (test-retest) when administering the final version of the instrument with the target group (workers who work in health interventions)

The reliability of the instrument was investigated by analyzing internal consistency (Cronbach’s alpha [α]) and temporal stability (agreement between the first and second time the instrument was administered - test-retest). To this end, 31 health professionals working in the following areas in Florianopolis, Londrina and Recife participated: physical education (n=10), nursing (n=6), pharmacy (n=1), physiotherapy (n=1), speech therapy (n=1), veterinary medicine (n=2), nutrition (n=1), dentistry (n=3), social work (n=3), occupational therapy (n=1) and psychology (n=2). In this step, each participant was asked to individually evaluate an intervention in which they had participated in the past 12 months. This evaluation was done twice, no more than two weeks apart. It was emphasized both times that the object of this evaluation was the action/intervention.

The data collected were entered onto a Microsoft Excel^© spreadsheet and analyzed with absolute and relative frequency distribution. Content validity and face validity used the content validity index (CVI) estimate with regard to adequacy and clearness of the instrument’s indicators. The CVI measures the proportion of evaluators who agree on the indicators comprising the instrument. Reliability was tested by analyzing internal consistency and temporal stability. Internal consistency was verified based on the Cronbach alpha value,²³23. Souza AC, Alexandre NMC, Guirardello EB. Psychometric properties in instruments evaluation of reliability and validity. Epidemiol Serv Saude. 2017;26(3):647-57. doi: http://doi.org/10.5123/S1679-49742017000300022.
http://doi.org/10.5123/S1679-49742017000... which estimates the contribution of each indicator to the composition of the final score. Temporal stability was verified by the test-retest method (interval between seven and 13 days), and tested by means of the intraclass correlation coefficient (ICC) and relative agreement. The sum of the total scores of the instrument was used to estimate the ICC. Relative agreement shows the agreement between the answers given the first and the second time the instrument was tested. The following were considered adequate validity and reliability values: CVI≥70.0%; Cronbach’s alpha ≥0.70; ICC≥0.70; and relative agreement ≥70.0%. Analyses were performed using the Statistical Package for the Social Sciences (SPSS) version 21.0 and a 5% significance level.

The study project was approved by the State University of Londrina Human Research Ethics Committee: Opinion No. 2.855.332, issued on August 30, 2018; Certificate of Submission for Ethical Appraisal No. 95467618.6.0000.5231.

Results

Table 1 shows the results for content validity, regarding the adequacy of the indicators in relation to the respective principles, and the clarity of the indicators in the final version of the instrument. The experts’ analysis showed that the indicators were adequate to the principles and were clear in their wording. Regarding the adequacy of the indicators in relation to the principles, the average score of the nine indicators was 89.8%, with all indicators presenting agreement scores higher than 80.0% (five of them were higher than 90.0%). As for the clearness of the indicators, the overall average value was 82.5%, ranging from 66.7% to 100.0%; only the score given by the experts for the ‘empowerment’ indicator was below 70.0%. The internal consistency analysis (Cronbach’s alpha) showed a significant value (α=0.80) for the instrument’s indicators; and indicators that showed equivalent and satisfactory importance for achieving the score on the scale were kept in the final structure. The indicators showed high agreement, between 77.4% and 93.6%. The intraclass correlation coefficient - ICC - for the instrument was 0.93 (95%CI 0.85; 0.96).

Regarding face validity, all workers considered the instrument (Figure 2) useful for identifying the points of the intervention they evaluated with regard to how “far from” or “close” they were to the principles of Health Promotion, as well as being useful for encouraging discussion on possible strategies on how to improve the interventions to bring them closer to the principles of Health Promotion.

Discussion

The results obtained indicate that the instrument proposed showed content validity, face validity and reliability, all of these qualities being acceptable with regard to the instrument’s structure. It can be used by health professional working in interventions aimed at promoting health and, specifically, for them to evaluate the adequacy of these interventions in relation to the principles of the National Health Promotion Policy. Although initially designed for Primary Health Care workers, and assuming the probability of these workers using it more often, we found that the instrument could be used by health professionals from other sectors, as highlighted by the experts who were consulted.

However, some limitations need to be considered. The inclusion of other indicators would certainly increase the variability and complexity already observed in many principles of the instrument - although this increase could reduce its applicability. However, even with just one indicator for each principle, the analysis of validity and reliability indicated adequate results, with acceptable psychometric values. It is worth noting that it is unlikely that an intervention will achieve perfection in all indicators. Perhaps a perfect evaluation, or one close to perfect, is more likely to reflect a lack of self-criticism on the part of the evaluator rather than the quality of the intervention itself. Moreover, it is possible that one or more indicators do not apply for certain interventions. It is also important to highlight that a negative evaluation of a certain indicator does not necessarily represent a “failure” in the performance of the worker who does the evaluation. For example, in the pilot application of the initial version of the instrument, the indicator related to intersectoriality was not well evaluated in one of the interventions. In the discussion, it became evident that the responsibility was not only that of those involved in the intervention under evaluation. However, the need was noted to seek better articulation with other sectors.

The instrument can help to fill a gap, related to the scarcity of other forms of evaluation of actions focused on Health Promotion. It can help workers to devise adjustments in some aspects of their interventions, or to emphasize aspects not considered before. For example: those responsible for a certain intervention may perceive that it has not awakened social participation. Certainly, a relatively simple and feasible action would be to publicize the dates of meetings of the Municipal Health Councils, including the neighborhood councils, with due emphasis on the importance of these bodies for effective Social Participation, this being one of the principles of the Unified Health System (SUS).

With regard to operationalization of Health Promotion in the daily routine of health services, it is worth highlighting the rather diverse way the concept is translated into concrete actions,²⁴24. Medina MG, Aquino R, Vilasbôas ALQ, Mota E, Pinto Júnior EP, Luz LA, et al. Promoção da saúde e prevenção de doenças crônicas: o que fazem as equipes de Saúde da Família? Saude Debate . 2014;38(n.spe):69-82. doi: http://doi.org/10.5935/0103-1104.2014S006.
http://doi.org/10.5935/0103-1104.2014S00... which is why the importance of the health action evaluation process must be reinforced; and, considering Health Promotion in particular, its theoretical and praxis approach, as a component of integral care. Therefore, there is a growing need to define tools that allow these practices to be evaluated and that are useful for guiding this process, at both the micro and macro policy levels. In this scenario, use of this instrument can lead to the will to build/plan and evaluate Health Promotion actions.

An important point to be highlighted, observed in the stages in which the workers presented their perceptions about the possible usefulness of the instrument (stages 2 and 4), concerns its possible contribution to overcoming the notion of a health intervention that is merely “curative” or “health-promoting”. The definition of an action as “health-promoting” should not be “aprioristic”, which often happens when certain themes and actions are classified as health-promoting and others are not, without proper analysis of how they are carried out.²⁵25. Carvalho FFB, Cohen SC, Akerman M. Promoção da saúde na atenção básica: problematizar o instituído para aproximá-la do cuidado. Porto Alegre (RS): Artmed Panamericana; 2018. p. 9-37. (Sistema de Educação Continuada a Distância, vol. 3). Along this same line of reasoning, Health Promotion lies not only in the end, but fundamentally in the means, including analysis of how the actions are planned and executed.²⁶

The instrument showed content validity and face validity, acceptable reliability analysis (internal consistency and temporal stability) and has the potential to be used by workers involved in interventions the objective of which is to promote health. In particular, Health Promotion interventions should be evaluated as to their closeness to the principles of the National Health Promotion Policy. Naturally, the specificity of each context and intervention must be considered, since greater closeness depends on several factors, including working conditions themselves and the willingness of workers to undertake a frank evaluation of their interventions.

References

Publication Dates

History