Abstract

The article presents the methodology used in building a panel of indicators for monitoring and evaluating the implementation of the National Policy for the Integrative Health of the Black Population (PNSIPN). The methodology was developed in four stages: scenario identification, implementation context, PNSIPN indicators and validation of the indicators. The proponents of the policy, street-level bureaucrats, technical advisors of the collegiate representing the managers, representatives of social movements, of associations and forums of pathologies participated in every stage. Those actors identified and agreed on the panel’s indicators, which have been categorized into indicators of coping with racism; indicators of sociodemographic conditions according to gender, age and race/color; and indicators of morbidity and mortality according to gender, age and race/color. The indicator panel for monitoring and evaluating the PNSIPN implementation is feasible and can be used at municipal, state and federal levels, possibly subsidizing the implementation process and enabling the improvement of management. The methodology contributes to the identification of indicators for public policies aimed at guaranteeing human rights, rights surveillance and advocacy.

Keywords:
Public Health Policies; Black Population Health; Health Evaluation; Health Indicators; Management Indicators

The evaluation of the public policy implementation process involves judging the materialization, failure or success of its theoretical formulation and the actions taken to reach the propositions for this phase of the policy (Felisberto et al., 2008FELISBERTO, E. et al. Contribuindo com a institucionalização da avaliação em saúde: uma proposta de autoavaliação. Cadernos de Saúde Pública, Rio de Janeiro, v. 24, n. 9, p. 2091-2102, 2008.; Hartz, 1999HARTZ, Z. M. A. Avaliação dos programas de saúde: perspectivas teórico-metodológicas e políticas institucionais. Ciência e Saúde Coletiva, Rio de Janeiro, v. 4, n. 2, p. 341-353, 1999.; McConnell, 2015MCCONNELL, A. What is policy failure? A primer to help navigate the maze. Public Policy and Administration, Londres, v. 30, n. 3-4, p. 221-242, 2015.). For Tamaki et al. (2012TAMAKI, E. M. et al. Metodologia de construção de um painel de indicadores para o monitoramento e a avaliação da gestão do SUS. Ciência e Saúde Coletiva, Rio de Janeiro, v. 17, n. 4, p. 839-849, 2012.), before proceeding to this assessment, it is necessary to monitor the policy, which involves creating a process for collecting strategic data and observing, analyzing and evaluating how public policy actions, strategies and programs are being implemented: monitoring proposes the “routine monitoring of relevant [health] information […] to support decision making, identification, solution forwarding and problem reduction, as well as the correction of directions” (Tamaki et al., 2012TAMAKI, E. M. et al. Metodologia de construção de um painel de indicadores para o monitoramento e a avaliação da gestão do SUS. Ciência e Saúde Coletiva, Rio de Janeiro, v. 17, n. 4, p. 839-849, 2012., p. 841).

According to Novaes (2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-559, 2000.), it is important to understand that the evaluation is a punctual process that must occur in an established period and involves judgment of value or merit, while monitoring is a continuous process that can even provide information so that an evaluation can be carried out.

Arrethe (2001ARRETHE, M. T. S. Uma contribuição para fazermos avaliações menos ingênuas. In: BARREIRA, M. C. R. N.; CARVALHO, M. C. B. (Org.). Tendências e perspectivas na avaliação de políticas e programas sociais. São Paulo: IEE-PUCSP, 2001. p. 44-55.) states that monitoring the implementation of a policy is complex and continuous, influenced both by its content and by the context, as well as by the relationship between the social, economic and political environment and the government itself. These factors can interfere with the implementation, changing the directions initially planned and may impact plans and even objectives to achieve the proposed effect.

One of the ways to monitor the policy implementation process is to define indicators, which, according to Jannuzzi (2016JANNUZZI, P. M. Monitoramento e avaliação de programas sociais: uma introdução aos conceitos e técnicas. Campinas: Alínea, 2016.), are technical instruments that make it possible, in addition to understanding the ideological markers of policy formulation, to monitor its implementation process and evaluate its success/advance or failure/setback. For this, it is essential to build a panel of indicators that is consistent and easy to understand, that is feasible and useful for managers and civil society, and that “allows the deepening of academic research on social change and on the determinants of different social phenomena” (Jannuzzi, 2016JANNUZZI, P. M. Monitoramento e avaliação de programas sociais: uma introdução aos conceitos e técnicas. Campinas: Alínea, 2016., p. 22).

The construction of panels of monitoring and evaluation indicators can support less naive analyzes and can be useful to document policy development, document and inform its implementation, provide data to account for invested resources and perform more efficient public administration (Arrethe, 2001ARRETHE, M. T. S. Uma contribuição para fazermos avaliações menos ingênuas. In: BARREIRA, M. C. R. N.; CARVALHO, M. C. B. (Org.). Tendências e perspectivas na avaliação de políticas e programas sociais. São Paulo: IEE-PUCSP, 2001. p. 44-55.; Brousselle et al., 2011BROUSSELLE, A. et al. (Org.) Avaliação: conceitos e métodos. Rio de Janeiro: Editora Fiocruz, 2011.; CDC, 2015CDC - CENTERS FOR DISEASE CONTROL AND PREVENTION. Step by step: evaluating violence and injury prevention policies: brief 1: overview of policy evaluation. Atlanta, 2015. Disponível em: <Disponível em: https://bit.ly/3ht5YX7 >. Acesso em: 15 jun. 2020.
https://bit.ly/3ht5YX7... ; Faria, 2012FARIA, C. A. P. (Org.). Implementação de políticas públicas: teoria e prática. Belo Horizonte: Editora PUC Minas, 2012.; Silva, 2005SILVA, L. M. V. Conceitos, abordagens e estratégias para a avaliação em saúde. In: HARTZ, Z. M. A.; SILVA L. M. V. (Org.). Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Rio de Janeiro: Editora Fiocruz , 2005. p. 15-39.).

The set of indicators referring to a certain aspect of the social reality or to the area of intervention of a given phenomenon is called “system” or “panel of indicators.” Success in its construction is crucial for monitoring the implementation of the policy to which it refers. This monitoring, supported by the indicators, allows the identification and evidence of successes and failures, forwarding solutions and adjustments, correcting and improving actions, services or programs (Cunha, 2018CUNHA, C. G. S. Avaliação de políticas públicas e programas governamentais: tendências recentes e experiências no Brasil. Estudos de Planejamento, Porto Alegre, n. 12, p. 27-57, 2018. Disponível em: <Disponível em: https://bit.ly/2ULPfV7 >. Acesso em: 10 maio 2019.
https://bit.ly/2ULPfV7... ; McConnell, 2015MCCONNELL, A. What is policy failure? A primer to help navigate the maze. Public Policy and Administration, Londres, v. 30, n. 3-4, p. 221-242, 2015.).

Affirmative action policies and the National Policy for Integrative Health of the Black Population

The 1987/1988 National Constituent Assembly (ANC) represents a historic moment in Brazil, in which different popular movements focused on the state’s legal frameworks, demanding rights. According to Santos (2015SANTOS, N. N. S. A voz e a palavra do movimento negro na Assembleia Nacional Constituinte (1987/1988): um estudo das demandas por direitos. 2015. Dissertação (Mestrado em Direito) - Fundação Getúlio Vargas, São Paulo, 2015., p. 40),

In the ANC process, the health movement, which demands health for all, and the black movement, which demands public rights and policies capable of fighting racism, stand out in the ANC process. As for the first, Article 196 of the 1988 Federal Constitution is the ultimate expression of its achievement: “Health is the right of all and the duty of the State, guaranteed through social and economic policies aimed at reducing the risk of disease and other health problems and universal and equal access to actions and services for their promotion, protection and recovery” (Brasil, 1988BRASIL. Constituição da República Federativa do Brasil de 1988. Diário Oficial da União, Brasília, DF, 5 out. 1988. Disponível em: <Disponível em: https://bit.ly/2y3wry1 >. Acesso em: 16 jun. 2020.
https://bit.ly/2y3wry1... ). As for the second, for Santos (2015SANTOS, N. N. S. A voz e a palavra do movimento negro na Assembleia Nacional Constituinte (1987/1988): um estudo das demandas por direitos. 2015. Dissertação (Mestrado em Direito) - Fundação Getúlio Vargas, São Paulo, 2015.) and Lima (2010LIMA, M. Desigualdades raciais e políticas públicas: ações afirmativas no governo Lula. Novos Estudos Cebrap, São Paulo, n. 87, p. 77-95, 2010.) the ANC is in fact the moment when the theme of racism is inserted in the Brazilian governmental agenda, then turning the Black a political subject. The inclusions in the constitutional text referring to the issue gave rise to subsequent struggles for regulation and effective implementation of policies and laws.

Among the milestones in the history of the conquests of the black movement in the field of health, in 1995, there was the presentation to the then President of the Republic, Fernando Henrique Cardoso, of a series of demands for public policies and actions to combat racism, involving different portfolios. To meet these different demands, an inter-ministerial working group (GTI) was created to value the black population, with health being one of the sub-themes. One of the products of this GTI-Health was the inclusion of the item “color” in the Mortality Information System (SIM) and in the Live Birth Information System (Sinasc), which is essential for the production of data that denounces racial inequities and enables the discussion on the impact of racism on health (Batista et al., 2017BATISTA, L. E. et al. Avaliação do processo de implementação da Política Nacional de Saúde Integral da População Negra: indicadores de monitoramento e avaliação. São Paulo: Instituto de Saúde, 2017. Relatório de Pesquisa. Mimeo., p. 5).

In 2003, the Secretariat for Policies for the Promotion of Racial Equality (Seppir) was created, linked to the President of the Republic’s office. Its mission was to propose policies that promote racial equality, with an emphasis on black, indigenous and Romani populations. Seppir’s Affirmative Actions Secretariat was in dialogue with the Ministry of Health and, in this context, in 2004 the First National Seminar on Health of the Black Population was held, and the Technical Committee on Health of the Black Population was created. This committee had the challenge of preparing the initial version of the health policy aimed at this population and assisting the Ministry in the elaboration of the plan of actions and activities (Werneck, 2010WERNECK, J. Saúde da população negra: passo a passo: defesa, monitoramento e avaliação de políticas públicas. Rio de Janeiro: Criola, 2010. Disponível em: <Disponível em: https://bit.ly/2AsZUNP >. Acesso em: 18 maio 2020.
https://bit.ly/2AsZUNP... , p. 15).

In November 2006, the text of the National Policy for Integrative Health of the Black Population (PNSIPN) was unanimously approved at a meeting of the National Health Council, in 2008 it was agreed upon by the Tripartite Intergovernmental Commission and on May 14, 2009 it was published in the Official Gazette as Ordinance No. 992 (Brasil, 2009aBRASIL. Ministério da Saúde. Portaria nº 992, de 13 de maio de 2009. Institui a Política Nacional de Saúde Integral da População Negra. Diário Oficial da União , Brasília, DF, 14 maio 2009a.). In 2010, with the approval of Law No. 12,288 (Brasil, 2010BRASIL. Lei nº 12.288, de 20 de julho de 2010. Institui o Estatuto da Igualdade Racial; altera as Leis nº 7.716, de 5 de janeiro de 1989, 9.029, de 13 de abril de 1995, 7.347, de 24 de julho de 1985, e 10.778, de 24 de novembro de 2003. Diário Oficial da União , Brasília, DF, 21 jul. 2010.), PNSIPN became legally binding. The operationalization of strategies and actions for the implementation of the PNSIPN was agreed in the operational plans and included in the National Health Plan and the Pluriannual Plan, with budget coverage listed in the Annual Budget Law (Batista; Barros, 2017BATISTA, L. E.; BARROS, S. Enfrentando o racismo nos serviços de saúde. Cadernos de Saúde Pública, Rio de Janeiro, v. 33, e00090516, 2017. Suplemento 1., p. 2).

It is in this context that public policies for “promoting equity” aimed at the black and vulnerable populations were included in the Brazilian National Health System (SUS). National policies for Integrative Health for the Black Population (Brasil, 2009aBRASIL. Ministério da Saúde. Portaria nº 992, de 13 de maio de 2009. Institui a Política Nacional de Saúde Integral da População Negra. Diário Oficial da União , Brasília, DF, 14 maio 2009a.), Integrative Health for Lesbians, Gays, Bisexuals, Transvestites and Transsexuals (LGBT) (Brasil, 2011aBRASIL. Ministério da Saúde. Portaria nº 2.836, de 1º de dezembro de 2011. Institui, no âmbito do Sistema Único de Saúde (SUS), a Política Nacional de Saúde Integral de Lésbicas, Gays, Bissexuais, Travestis e Transexuais (Política Nacional de Saúde Integral LGBT). Diário Oficial da União , Brasília, DF, 2 dez. 2011a.), Integrative Health for Rural and Forest Populations (Brasil, 2011bBRASIL. Ministério da Saúde. Portaria nº 2.866, de 2 de dezembro de 2011. Institui, no âmbito do Sistema Único de Saúde (SUS), a Política Nacional de Saúde Integral das Populações do Campo e da Floresta (PNSIPCF). Diário Oficial da União , Brasília, DF, 5 dez. 2011b.), for the Homeless Population (Brasil, 2009bBRASIL. Decreto nº 7.053, de 23 de dezembro de 2009. Institui a Política Nacional para a População em Situação de Rua e seu Comitê Intersetorial de Acompanhamento e Monitoramento, e dá outras providências. Diário Oficial da União , Brasília, DF, 24 dez. 2009b.) and for Integrative Healthcare for the Gypsy/Romani People (Brasil, 2018BRASIL. Ministério da Saúde. Portaria nº 4.384, de 28 de dezembro de 2018. Altera a Portaria de Consolidação nº 2/GM/MS, de 28 de setembro de 2017, para instituir, no âmbito do Sistema Único de Saúde - SUS, a Política Nacional de Atenção Integral à Saúde do Povo Cigano/Romani. Diário Oficial da União , Brasília, DF, 31 dez. 2018.).

PNSIPN is marked by the recognition of racism, racial inequalities and institutional racism as social determinants of health. It aims to promote the integrative health of the black population, prioritizing the reduction of ethnic-racial inequalities and the fight against racism and discrimination in health institutions and services. Among its management strategies, the “development of specific actions to reduce disparities […], especially in maternal and child morbidity and mortality and in that caused by: violent causes; sickle cell disease; STD/HIV/AIDS; tuberculosis; leprosy; cervical and breast cancer; mental disorders” (Brasil, 2009aBRASIL. Ministério da Saúde. Portaria nº 992, de 13 de maio de 2009. Institui a Política Nacional de Saúde Integral da População Negra. Diário Oficial da União , Brasília, DF, 14 maio 2009a.).

The main advances and setbacks in the health of the black population are shown in Figure 1.

The timeline shown in Figure 1 shows the process of building a healthcare policy for the black population, which took place through seminars and conferences, especially in the 2000s. As of 2009, after the PNSIPN was published, efforts seem to be concentrated on its implementation, and the advances obtained are related to the agreement of three operational plans in the tripartite inter-management commissions. The contingency of PNSIPN resources and the change in the locus of the policy coordination group within the Ministry of Health demarcate the reality of its management.

In view of the presented framework, what indicators can be used to monitor and evaluate the implementation of a policy that aims to fight racism? In this article, we present the methodology of building a panel of indicators for this purpose.

How to build a panel of indicators?

To build the proposed panel, a partnership was established with the collegiate of representation of managers - National Council of Municipal Health Secretariats (Conasems), National Council of Health Secretaries (Conass) -, with the SUS Monitoring and Evaluation Department (Demas) of the Ministry of Health and the School of Nursing of Universidade de São Paulo.

Initially, it was identified the need to understand the scenario of the implementation of PNSIPN in the states and municipalities of the federation, to identify its context - how it happens, what strategies are adopted, what are the facilities and difficulties encountered -, to think about several possible indicators and agree on these indicators with managers and civil society. To meet these objectives, primary data were collected and technical meetings, focus groups, interviews and deliberative dialogue were held with strategic subjects, namely: the coordinators and technicians responsible for implementing the Policy, here called “street-level bureaucrats” (Lotta, 2012LOTTA, G. S. O papel das burocracias do nível da rua na implementação de políticas públicas: entre o controle e a discricionariedade. In: FARIA, C. A. P. (Org.). Implementação de políticas públicas: teoria e prática. Belo Horizonte: Editora PUC Minas , 2012. p. 20-49.), Policy makers, representatives of social movements, representatives of pathology forums, guest researchers (external researchers), researchers from the research group (internal researchers), technicians from different departments of the Ministry of Health, and representatives from Conass and Conasems. The methodology was then established, consisting of four stages, each with a specific objective and technical indication for data collection, strategic subjects and the product obtained (Chart 1).

Stage 1: scenario identification

The first stage tried to identify the state and municipal secretariats that implemented the PNSIPN.

Objective: to identify who is implementing the PNSIPN.

Tools and techniques used: questionnaire applied electronically to identify the places where the PNSIPN is implemented and what has been produced and carried out in the process.

Strategic subjects: policy makers, collegiate of representation of managers, national coordination of PNSIPN of the Ministry of Health, state and municipal coordinators of PNSIPN (street-level bureaucrats), representatives of social movements, of the National Federation of Associations of People with Sickle Cell Disease and the Forum of Pathologies the São Paulo State Health Council.

Product: mapping the status of the implementation of PNSIPN - places where PNSIPN was implemented, at the national level.

It should be noted that Conass and Conasems, here called “management representation collegiate,” forwarded the questionnaire link to those responsible for health portfolios in the municipal and state departments. At the end of the first stage, it was possible to identify municipal and state departments that had implemented the PNSIPN. The results of this stage of the study were published in the article “Facing racism in health services” (Batista; Barros, 2017BATISTA, L. E.; BARROS, S. Enfrentando o racismo nos serviços de saúde. Cadernos de Saúde Pública, Rio de Janeiro, v. 33, e00090516, 2017. Suplemento 1.).

Stage 2: implementation context

Representatives of state and municipal secretariats with experience in implementing the PNSIPN were invited to participate in a technical meeting and focus group that enabled to exchange experiences regarding the structure and expected functioning of the Policy in the municipalities and states. Also, at that time it was possible to identify the facilities and difficulties found in the implementation process. Policy makers, leaders of the black movement, representatives of the pathology forum, social movements and researchers also participated in the meeting and focus group.

During the technical meeting, external researchers were invited to teach classes on health indicators, policy evaluation and monitoring and evaluation indicators, and presented the Ministry of Health’s Strategic Management Support Room to participants.²2Available at: <http://sage.saude.gov.br/>. Access on: June 16. 2020.

Objective: to identify facilities and difficulties in implementing the PNSIPN.

Tools and techniques used: technical meetings with the participation of facilitators who taught classes on health assessment, assessment models, health indicators and health indicators panel (Furtado et al., 2013FURTADO, J. P. et al. A elaboração participativa de indicadores para a avaliação em saúde mental. Cadernos de Saúde Pública, Rio de Janeiro, v. 29, n. 1, p. 102-110, 2013.). Interviews were also conducted with participants to identify the strategies used in the implementation of PNSIPN and the focus group to discuss their path.

Strategic subjects: Policy makers, representatives of the national coordination of the PNSIPN of the Ministry of Health and the collegiate of representation of managers, street-level bureaucrats, representatives of the National Federation of Associations of People with Sickle Cell Disease, of the State Council’s Forum of Pathologies of São Paulo Health and social movements, and external specialists (from higher education institutions) who acted as facilitators of the technical meeting and focus group.

Products: logical model of PNSIPN.

In this stage, individual interviews were carried out, seeking to identify the facilities and difficulties found, the strategies used and the appropriate structure within the secretariats necessary for the implementation of the PNSIPN.

When asked about the facilities and difficulties of implementing the PNSIPN guidelines, the interviewees mention that the discussion about racism was included in the training process and studies were encouraged with ease, but it was difficult to expand the participation of the black movement in the municipal/state councils and implement monitoring and evaluation processes.

As for the implementation bottlenecks, managers point out that the main obstacles are personal and institutional racism, the lack of support from political power, lack of human and financial resources and lack of data disaggregated by race/color.

When questioned, respondents indicated that there is no policy to encourage the implementation of PNSIPN. However, it was subsequently found that the action strategy for Healthcare of Quilombola Populations was a government policy that lasted from 2003 to 2016, and that in that period a set of government incentives was incorporated into the PNSIPN implementation strategy.

On the other hand, in October 2018, R$ 6 million destined to support PNSIPN implementation projects were contingent by Mrs. Gerlane Baccarin, then Secretary of Strategic and Participative Management at the Ministry of Health (Ministério…, 2018MINISTÉRIO da Saúde: recursos da Política Nacional de Saúde Integral da População Negra. Avaaz.org: petições da comunidade, [s.l.], 24 out. 2018. Disponível em: <Disponível em: https://bit.ly/2Bhr0HO >. Acesso em: 16 jun. 2020.
https://bit.ly/2Bhr0HO... ).

The third activity of the contextualization stage was to collectively build the logical model after the class on evaluation models and health indicators and to discuss the facilities and difficulties of implementing the Policy. During the class, the teacher/external researcher presented some models of evaluation to the participants and highlighted some logical models of policies.

Realizing the group’s interest in logical models, the internal researchers pasted three cards in sequence and inserted the following questions: what are the objectives of PNSIPN? What are the PNSIPN guidelines? What strategies are used to implement the Policy? What are the activities carried out? How to monitor the activities carried out? What indicators do I use to judge the action taken?

After the participants answered the questions, the research team systematized the information by building the “logical model of intervention” (Figure 2), defined as “the set of postulates about the way in which a program is related to the benefits it is supposed to produce and the strategy and tactics that were adopted to achieve its goals and objectives” (Champagne et al., 2011CHAMPAGNE, F. et al. A avaliação no campo da saúde: conceitos e métodos. In: BROUSSELLE, A. et al. (Org.) Avaliação: conceitos e métodos. Rio de Janeiro: Editora Fiocruz , 2011. p. 41-60., p. 50).

The PNSIPN logical model (Figure 2) describes the objectives, the actions taken to achieve them, the expected benefits/results and the indicators used to evaluate the action taken. The analysis of the PNSIPN logical model suggests that: (1) the objective of developing actions to reduce problems prevalent in the black population has demanded several types of action in the routine work of street-level bureaucrats; (2) not all PNSIPN guidelines reach the monitoring and evaluation phase; and (3) the six PNSIPN guidelines are contained in the management strategies, but not all management strategies are mirrored in guidelines - for example, the healthcare of rural and forest peoples, including quilombola populations, is in the strategies and specific objectives of the Policy text, but is not explained in the guidelines; the Policy text does not provide a necessary organizational context or accuse the existence of institutional provisions that regulate the process of monitoring and evaluating implementation, which suggests the need for reformulation.

The discussion on facilities and difficulties in the implementation of the policy, together with the elaboration of the intervention’s logical model, suggested ways to identify possible indicators to compose a monitoring and evaluation panel.

Stage 3: indicators of the National Policy for Integrative Health of the Black Population

At this stage, street-level bureaucrats, technical advisors to the collegiate of managers, technical advisers representing the Ministry of Health and leaders of social movements participated in a focus group and two technical meetings in which they discussed what are sociodemographic indicators and of fighting racism.

Objective: to propose a set of indicators that can be used to monitor and evaluate the PNSIPN implementation process.

Tools and techniques: technical meetings were held with the participation of external researchers who acted as facilitators and taught classes on social indicators, demographic indicators existing in the databases of the Brazilian Institute of Geography and Statistics (IBGE) and indicators to confront institutional racism, also participating in the focus group that listed 45 indicators as strategic to compose the panel.

The strategy used included promoting a training process during technical meetings (Furtado et al., 2013FURTADO, J. P. et al. A elaboração participativa de indicadores para a avaliação em saúde mental. Cadernos de Saúde Pública, Rio de Janeiro, v. 29, n. 1, p. 102-110, 2013.) and conducting a focus group among street-level bureaucrats, technical advisors from the collegiate bodies representing managers and technicians from different areas of the Ministry of Health. Representatives of civil society were also invited, who in the collection of primary data were mentioned by managers as supporters of the implementation of the Policy. External and internal researchers participated in the focus group that defined the possible indicators for monitoring and evaluating the Policy implementation.

Strategic subjects: representatives of the national coordination of PNSIPN of the Ministry of Health and of the collegiate of representation of managers, Policy makers, street-level bureaucrats, representatives of the National Federation of Associations of People with Sickle Cell Disease, of the State Council’s Forum of Pathologies of São Paulo Health and social movements, external researchers (from higher education institutions, research institutes and non-governmental organizations) and internal researchers (Chart 2).

Product: list of 45 indicators for monitoring and analyzing the implementation of the PNSIPN to be agreed in stage 4.

Stage 4: validation of indicators

In the last stage, a deliberative dialogue was carried out to recommend or not the indicators proposed in stage 3.

Objective: to identify indicators to monitor and evaluate the implementation of PNSIPN.

Strategic subjects: policy makers, street-level bureaucrats, technicians from different areas of the Ministry of Health, representatives of the collegiate body representing managers, civil society, associations and the pathology forum, external researchers (from higher education institutions, research institutes and non-governmental organizations) and internal.

Deliberative dialogue was carried out as a resource to share knowledge among researchers, policy makers, street-level bureaucrats, civil society and other stakeholders, in addition to clarifying areas of uncertainty and developing a common understanding of the pros and cons of each indicator (Acosta; Oelke; Lima, 2017ACOSTA, A. M.; OELKE, N. D.; LIMA, M. A. D. S. Considerações teóricas do diálogo deliberativo: contribuições para prática, política e pesquisa em enfermagem. Texto e Contexto Enfermagem, Florianópolis, v. 26, n. 4, art. e0520017, 2017. Disponível em: <Disponível em: https://bit.ly/2C4MU13 >. Acesso em: 13 jun. 2019.
https://bit.ly/2C4MU13... ; Lavis et al., 2009LAVIS, J. N. et al. Support tools for evidence-informed health policymaking (STP) 14: organising and using policy dialogues to support evidence-informed policymaking. Health Research Policy and Systems, Londres, v. 7, art. S14, 2009. Suplemento 1. Disponível em: <Disponível em: https://bit.ly/3fpaTXh >. Acesso em: 25 jul. 2019.
https://bit.ly/3fpaTXh... ).

Before carrying out the deliberative dialogue, participants were sent a list of 45 indicators defined in stage 3, accompanied by a form and a scoring scale - good, regular, excellent. The Interagency Health Information Network Indicator Qualification Form (Ripsa) was also sent, which contained the following items: validity (ability to measure what is intended); reliability (data quality); sensitivity (it is sensitive to capture the attribute); disaggregation (possible territorial level); and periodicity (time interval in which the indicator is updated). The return of the form and the Form was a condition for participating in the face-to-face meeting. The answers were consolidated by the research team and presented during the meeting.

On the first day of the deliberative dialogue, the indicators with the highest score were presented and, of the 45 indicators initially proposed, only 22 were highlighted to be discussed on the second day of work.

During the second day of the deliberative dialogue, the participants discussed each of the 22 indicators based on a matrix prepared by the Demas team. The matrix contained the following items: conceptualization of the indicator, calculation method, use, potential, limitations and representativeness as an indicator of the Policy. In the discussion round, 19 indicators were defined for monitoring and analyzing the implementation of the Policy (Chart 3).

After the deliberative dialogue, technicians from different areas of the Ministry of Health tested the indicators in each database, and the Demas technical team took charge of filling out the Indicators Qualification Form, composed of the following items: name of the indicator; national average goal or standard; baseline/year; calculation method; data source; measurement unit; lower level of disaggregation; limitations; administrative unit responsible for the data; and technical responsible for the indicator.

Product: Panel of indicators for monitoring and evaluating the implementation of PNSIPN (Chart 3).

Indicator panel for monitoring and analyzing implementation

A process was established in which it was proposed to listen to Policy makers, street-level bureaucrats, representatives of social movements, pathology forums, guest researchers and researchers from the research group, technicians from the Ministry of Health, technical advisers from Conass and Conasems in all stages of the proposal - identification of the problem, registration of strategies, facilities and difficulties in implementing the Policy and definition and agreement of its indicators.

According to Lotta (2012LOTTA, G. S. O papel das burocracias do nível da rua na implementação de políticas públicas: entre o controle e a discricionariedade. In: FARIA, C. A. P. (Org.). Implementação de políticas públicas: teoria e prática. Belo Horizonte: Editora PUC Minas , 2012. p. 20-49.), in the health area there is little literature devoted to analyzing the importance of implementing bureaucrats (street-level bureaucrats). For the author, directing a look at them is essential to understand the actions and factors that influence the changes in the implementation process, in the results and in the evaluation of public policies. When proposing the identification of policy indicators, it is necessary to use a methodology that involves its formulators, street-level bureaucrats, managers, social movements and internal and external researchers (Novaes, 2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-559, 2000.; Tamaki et al., 2012TAMAKI, E. M. et al. Metodologia de construção de um painel de indicadores para o monitoramento e a avaliação da gestão do SUS. Ciência e Saúde Coletiva, Rio de Janeiro, v. 17, n. 4, p. 839-849, 2012.). Therefore, it was important to include street-level bureaucrats for the panel construction methodology.

The methodology for constructing the indicator panel was preceded by a long and intense work of political articulation with strategic actors, namely:

Certain factors were considered strategic:

Finally, the methodological strategies allowed to build environments in which the participants felt at ease to contribute, establishing a climate of complicity, collaboration and learning between formulators, civil society, technicians, researchers and street-level bureaucrats.

In turn, the panel is categorized by the themes presented by external researchers invited to contribute to the group formation: indicators of confrontation with racism; indicators of sociodemographic conditions of the population according to gender, age group and race/color; and indicators of the profile of morbidity and mortality according to gender, age group and race/color. According to Tamaki et al. (2012TAMAKI, E. M. et al. Metodologia de construção de um painel de indicadores para o monitoramento e a avaliação da gestão do SUS. Ciência e Saúde Coletiva, Rio de Janeiro, v. 17, n. 4, p. 839-849, 2012., p. 844),

As some Demas technicians represented the Ministry of Health at Ripsa meetings, this prior knowledge made it easier to point out indicators that would be difficult to capture in the SUS information systems and those that did not exist on any national basis, those for fighting racism.

These indicators - inclusion of PNSIPN in municipal and state health plans; existence of a body responsible for conducting, coordinating and monitoring health actions for the black population; inclusion in the Annual Budget Law and Sargsus; and specific training and training of health professionals on racism and the health of the black population - they signal whether the racial theme is contained in the management instruments and in the sensitivity of health professionals.

As stated, this information did not exist in the national databases, but the performance of the IBGE technicians transformed a difficulty in conquest, because, when realizing this weakness for the construction of a panel, the representative of the Institute in the technical meetings and in the deliberative dialogue internally referred to IBGE a proposal to include some issues in the health axis of Munic and Estadic, in the collegiate item of regional management, in the municipal health plan and in PNSIPN (IBGE, 2015IBGE - INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Perfil dos estados e municípios brasileiros: 2014. Rio de Janeiro, 2015.). The proposal was accepted, and a survey was carried out in the 5,570 municipalities in the 26 states of the federation, plus the Federal District. This database is available, and the result of this work can be accessed online.³3Available at: <https://bit.ly/3dazdKT>. Access on: June 16. 2020.

According to Tamaki et al. (2012TAMAKI, E. M. et al. Metodologia de construção de um painel de indicadores para o monitoramento e a avaliação da gestão do SUS. Ciência e Saúde Coletiva, Rio de Janeiro, v. 17, n. 4, p. 839-849, 2012.), the panel must be able to inform about the aspects that constitute the theoretical model, incorporate practicality into a monitoring instrument with evaluative potential, have available databases and have a small number of

Therefore, among the indicators to combat racism, the presence of a specific instance to lead, coordinate and monitor health actions for the black population is the key indicator.

The second group of indicators concerns sociodemographic data, of which population according to gender, age and race/color is fundamental, but there is a limitation: population data disaggregated by gender, age group and race/color are updated only in the Census Demographic, every ten years.

The indicators categorized as morbidity and mortality according to gender, age group and race/color are those available in SIM, Sinasc and Hospital Information System, and its reading allows to monitor and evaluate the scope of the health system’s action in the healthcare of the black population: premature mortality rate (between 30 and 69 years) due to chronic non-communicable diseases; homicide mortality rate; reason of maternal death; child mortality rate; tuberculosis mortality rate; percentage distribution of syphilis cases in pregnant women; proportion of people living with HIV/AIDS; AIDS incidence rate; proportional mortality rate due to ill-defined causes; number of live births diagnosed with sickle cell disease; and hospitalization rate for mental disorder. It is difficult to point out among the indicators of morbidity and mortality which is the main one, but it is easy to point out its limitation: the poor completion of race/color information in information systems (Batista; Barros, 2017BATISTA, L. E.; BARROS, S. Enfrentando o racismo nos serviços de saúde. Cadernos de Saúde Pública, Rio de Janeiro, v. 33, e00090516, 2017. Suplemento 1.; Braz et al., 2013BRAZ, R. M. et al. Avaliação da completude da variável raça/cor nos sistemas nacionais de informação em saúde para aferição da equidade étnico-racial em indicadores usados pelo Índice de Desempenho do Sistema Único de Saúde. Saúde em Debate, Rio de Janeiro, v. 37, n. 99, p. 554-562, 2013. Disponível em: <Disponível em: https://bit.ly/30Jazys >. Acesso em: 15 jun. 2020.
https://bit.ly/30Jazys... ).

The development of a panel of indicators may prove to be one of the strategies for evaluating policies that aim to fight racism. The methodology for building this panel for monitoring and analyzing the implementation of the PNSIPN made it possible to suggest a viable indicator base, consistent with the Policy guidelines, from the perspective of rights surveillance and advocacy.

The fact that the IBGE included the indicators to combat racism in the profile surveys of Brazilian states and municipalities can be understood as a gain, a product of the participatory process of building the panel.

Final considerations

The participatory methodology was key to build the PNSIPN indicator panel. This methodology can be used to create panels of indicators to monitor and evaluate the implementation of health policies for the LGBT population, the rural and forest population, the Romani population and the homeless population.

Panels of indicators for monitoring and analyzing the implementation of affirmative policies can be instrumental in verifying whether the theoretical formulation of the policy dialogues with the actions undertaken so that in future studies it is possible to judge the success or failure of its implementation process.

The panel is expected to contribute to policy management and adoption of actions to reduce racial inequities in health, and that this work will contribute to the improvement of SUS actions, with academics and social movements interested in the implementation of PNSIPN.

Acknowledgements

To Professor Hillegonda Maria Dutilh Novaes, for the suggestions given in the theoretical construction of the work methodology; to the National Council of Municipal Health Secretariats and the National Council of Health Secretaries, for making the questionnaire available to municipal and state managers; to Maria Mercedes Loureiro Escuder, for careful reading; and to Maritsa Bertoli, for collaboration in the construction of the online instrument.

References

Publication Dates

History