Abstract

Objective:

to build a model to evaluate use of the National Program for Improving Primary Health Care Access and Quality (PMAQ-AB) in Brazil.

Methods:

document analysis and literature review followed by a consensus workshop with specialists, 19 specialists were invited who were selected because of their teaching or research experience, professional activities or publications related to the subject; validation took place on an online platform where participants were asked to analyze the evaluation proposal and to give their opinion based on the categories ‘I totally agree’, ‘partially agree’, and ‘disagree’.

Results:

15 specialists confirmed their participation; at the end of three rounds, the specialists emphasized the clarity of the proposal and the clarity of the object of evaluation presented in both the Logic Model and the Theoretical Logic Model; the specialists fully agreed with the Evaluation Matrix and confirmed its theoretical consistency.

Conclusion

this model will be applied to evaluate the use of PMAQ-AB in Primary Health Care management and health team actions.

Keywords:
Primary Health Care; Program Evaluation; Health Evaluation; Health Care Surveys

Introduction

Public administration modernization, management by results, the introduction of regulatory mechanisms and evaluation-based funding are important aspects for the emphasis on health evaluation which occurred with effect from 1990 in Brazil.¹1. Figueiró AC, Felisberto E, Dubeux LS, Samico I. Avaliando as avaliações: meta-avaliação, uma introdução ao tema. In: Samico I, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010. p. 161-70. Of similar importance were the legal and administrative changes in Brazilian National Health System (SUS) management, decentralization of responsibilities, actions and resources within the system, as well as greater complexity of the epidemiological profile requiring new approaches and incorporation of technology.²2. Figueiró AC, Frias PG, Navarro LM. Avaliação em saúde: conceitos básicos para a prática nas instituições. In: Samico I, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook ; 2010:p. 1-13.

A national research and initiatives agenda to improve and consolidate SUS began in 2003, such as the Project for Expanding and Consolidating the Brazilian Family Health Program, aiming at strengthening technical evaluation capacity with the objective of driving forward systematization, coordination and integration of management activities in the Brazilian municipalities and states.⁴4. Natal S, Santos E, Alves C, Felisberto E, Santos E, Sousa A. A implementação do projeto estratégico de fortalecimento da capacidade técnica em monitoramento e avaliação das secretarias de saúde: avaliação formativa e auto-avaliação. In: Hartz ZMA, Felisberto E, Vieira da Silva LM, organizadores. Meta-avaliação da atenção básica em saúde: teoria e prática. Rio de Janeiro: Editora Fiocruz; 2008. p.341-67. Standing out with effect from 2011 is the National Primary Health Care Access and Quality Improvement Program (PMAQ-AB) to motivate managers and teams in order to improve health service quality and access through agreed commitments and indicators.⁵5. Brasil. Ministério da Saúde. Portaria MS/GM no 1645, de 2 de outubro de 2015. Dispõe sobre o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ-AB) [Internet]. Diário Oficial União, Brasília, 2015 out 2 [citado 2019fev 4]. Disponível em: Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt1645_01_10_2015.html
http://bvsms.saude.gov.br/bvs/saudelegis...

PMAQ-AB proposes the introduction of an evaluation culture in everyday Primary Health Care, whereby receipt of funding is conditioned to performance achieved as identified through evaluation. After three evaluation cycles, it is important to verify whether PMAQ-AB has been able to assimilate the evaluation conclusions and recommendations, their use and impact on the changes made to the program.

In order to assist investigations capable of answering such questions, the purpose of this study was to build a model to evaluate the use of PMAQ-AB in municipal Primary Health Care management and health team actions.

Methods

This evaluability study consists of the first stage of the project entitled Evaluation of the use of the National Program for Improving Primary Health Care Access and Quality (PMAQ-AB) in municipalities of the Brazilian state of Santa Catarina carried out at the Postgraduate Public Health Program, Federal University of Santa Catarina (PPGSC/UFSC). This stage involved building the evaluation model and took into consideration (i) program delimitation regarding guidelines, development phases and expected results, (ii) building the Theoretical Logic Model based on the program components and inter-relations of the evaluation use construct, and (iii) validation of the model proposed.

Our documentary research included the following documents: The 1988 Brazilian Federal Constitution; Public Health Organic Laws No. 8080 and No. 8142; National Primary Health Care Policy and Ordinance No. 2488/2011; specific PMAQ-AB ordinances - 1654/2011; 1089/2012; 1645/2015 -; instruction manuals and program external evaluation tools. With regard to the literature, we conducted a review in electronic databases with the purpose of identifying studies applied to use, usage and influence of health evaluation. Finally, we sought to relate program description, objectives, targets and development phases to the construct of the use of the evaluation process and results, proposing a theoretical logic model for PMAQ-AB use.

Validation of the model took place in a Consensus Workshop using the Delphi Method on the Survey Monkey® platform. Specialists were selected from among researchers, technicians and lecturers with publications and actions developed on the topic, experience in health evaluation and in Primary Care, as well as experience in the institutionalization of this evaluation by PMAQ-AB. We asked them to analyze the items initially proposed for the Theoretical Logic model and to give their judgment based on previously established categories (‘I totally agree’; ‘partially agree’; and ‘disagree’), followed by optional comments and suggestions. We carried out three workshop rounds between November 2017 and June 2018, with a summary of convergent and divergent topics found in each one of them, including further development and suggested and accepted reformulations.

The research project complied with Resolution No. 422/2016 of the National Human Research Ethics Council and was approved by the Human Research Ethics Committee of the Federal University of Santa Catarina: Report CEPSH/UFSC No. 2.229.055, dated 20 August 2017.

Results

The results of this construction are the fruit of document analysis, literature review and consensus agreements. Nineteen specialists were invited to take part in the consensus: 15 confirmed their participation; two of them did not accept for personal reasons; and two of them did not reply to the invitation.

In the first round, ten specialists gave feedback on the proposal. In relation to the Theoretical Model, four of them partially agreed and six of them totally agreed. Their contributions, presented in the form of suggestions, were to detail the program and establish a direct relation between PMAQ-AB intermediary results, effects and expected results. With regard to the Logic Model, one specialist partially disagreed, one partially agreed, four totally agreed and one did not answer. The divergences were attributed to understanding of the evaluation focus, which was restricted to the use of PMAQ-AB, and not to the program itself.

In the second round, feedback came from seven specialists. Regarding the Theoretical Model, four totally agreed and three partially agreed. Their suggestions were: justify the building of the model and concatenate the expected PMAQ-AB results. Five specialists totally agreed with the Logic Model and two partially agreed, requiring convergence between the results of the model and the framework indicators.

In the third round, seven specialists gave feedback on the proposal. They totally agreed with the Theoretical Model adjustments, emphasizing the clarity of the proposal and of the evaluated object. Regarding the Logic Model, five specialists totally agreed and two partially agreed, suggesting that the expected results of the program and of PMAQ-AB use should be kept in the model.

Figure 1 presents this construction in the form of a diagram.

Discussion

The model assumes that SUS, as a state policy provided for in the Brazilian Federal Constitution, is based on the constitutional definition of health as a “right of all people and the duty of the State”, anchored in the principles of universality, equity, comprehensiveness, decentralization, regionalization, hierarquization and social participation.⁷7. Brasil. Constituição da República Federativa do Brasil [Internet]. Brasília: Senado Federal; 1988 [citado 2019 jun 21]. Disponível em: Disponível em: http://www.planalto.gov.br/ccivil_03/constituicao/constituicao.htm
http://www.planalto.gov.br/ccivil_03/con...

Primary Health Care, within the organizational and operational context of Public Health, is oriented toward accessibility, linkage, care continuity, care comprehensiveness, accountability and humanization. Primary Health Care is provided with a high degree of decentralization and capillarity, under the direct responsibility of municipal managers. The work process of Primary Health Care teams seeks to expand actions that provide solutions, helping to deal with the most frequent and relevant health needs in Brazil and impacting on the population’s health situation.⁸8. Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Política nacional de atenção básica [Internet]. Brasília: Ministério da Saúde; 2012 [citado 2019 jun 21]. 110 p. Disponível em: Disponível em: http://189.28.128.100/dab/docs/publicacoes/geral/pnab.pdf
http://189.28.128.100/dab/docs/publicaco...

PMAQ-AB was implemented by the Ministry of Health with the aim of inducing increased access and improving the quality of this level of care. Taking part in the program is voluntary and its success depends on the motivation and the proactiveness of the actors involved. Change in management culture and enhancing Primary Care requires engagement and mobilization of managers, teams and service users in order to develop a culture of planning, negotiation and formal agreement on the provision of funding according to commitment and results presented. The program guidelines point to a continuous process of enhancing standards and indicators, transparency in actions and definition of a quality standard in order for teams to have greater solution provision capacity.⁵5. Brasil. Ministério da Saúde. Portaria MS/GM no 1645, de 2 de outubro de 2015. Dispõe sobre o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ-AB) [Internet]. Diário Oficial União, Brasília, 2015 out 2 [citado 2019fev 4]. Disponível em: Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt1645_01_10_2015.html
http://bvsms.saude.gov.br/bvs/saudelegis...

There are three consecutive phases and one cross-cutting axis: the “Adherence and Agreement Setting” phase establishes commitments between health teams, municipal health managers and the Ministry of Health; the “Certification” phases verifies team self-evaluation and the indicators agreed to, as well as external evaluation of health teams and management; and the “Agreement Resetting” phase is based on the results of certification and resets standards and indicators.⁵5. Brasil. Ministério da Saúde. Portaria MS/GM no 1645, de 2 de outubro de 2015. Dispõe sobre o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ-AB) [Internet]. Diário Oficial União, Brasília, 2015 out 2 [citado 2019fev 4]. Disponível em: Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt1645_01_10_2015.html
http://bvsms.saude.gov.br/bvs/saudelegis...

The Strategic Cross-cutting Axis includes self-evaluation, monitoring, continuing education, institutional support and horizontal cooperation. It aims to ensure the development of actions to improve access and quality in all stages of Primary Care.⁵5. Brasil. Ministério da Saúde. Portaria MS/GM no 1645, de 2 de outubro de 2015. Dispõe sobre o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ-AB) [Internet]. Diário Oficial União, Brasília, 2015 out 2 [citado 2019fev 4]. Disponível em: Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt1645_01_10_2015.html
http://bvsms.saude.gov.br/bvs/saudelegis...

PMAQ-AB expected results include: changes in the work process, in management capacity and in the population’s health situation; trained workers; services orientated to the needs of their users; and strengthened civil society participation.⁵5. Brasil. Ministério da Saúde. Portaria MS/GM no 1645, de 2 de outubro de 2015. Dispõe sobre o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ-AB) [Internet]. Diário Oficial União, Brasília, 2015 out 2 [citado 2019fev 4]. Disponível em: Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt1645_01_10_2015.html
http://bvsms.saude.gov.br/bvs/saudelegis...

By articulating the program with the theory of evaluation, PMAQ-AB adopts a performance evaluation strategy, conditioning financial incentives for municipalities to agreed commitments to expanding access and improving quality in SUS Primary Health Care.⁵5. Brasil. Ministério da Saúde. Portaria MS/GM no 1645, de 2 de outubro de 2015. Dispõe sobre o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ-AB) [Internet]. Diário Oficial União, Brasília, 2015 out 2 [citado 2019fev 4]. Disponível em: Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt1645_01_10_2015.html
http://bvsms.saude.gov.br/bvs/saudelegis... The practice of health evaluation qualifies public service management decision-making when it is integrated with an evaluation culture combined with the incorporation of results.²2. Figueiró AC, Frias PG, Navarro LM. Avaliação em saúde: conceitos básicos para a prática nas instituições. In: Samico I, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook ; 2010:p. 1-13.

As such, use typologies were included in the model, categorized in the literature as instrumental, conceptual and symbolic use. Instrumental use is directly reflected in helping decision-making or promoting changes in the program;⁹9. Hartz ZMA, Santos E, Matida A. Promovendo e analisando o uso e a influência das pesquisas avaliativas: desafios e oportunidades ao se institucionalizar a avaliação em saúde. In: Hartz ZMA, Felisberto E, Vieira-da-Silva LM, organizadores. Meta-avaliação da atenção básica em saúde: teoria e prática. Rio de Janeiro: Editora Fiocruz ; 2008. p. 325-40. conceptual use is considered to be a medium or long-term effect on knowledge production, deepening discussions on the intervention and indicating a change in the way of thinking;⁹9. Hartz ZMA, Santos E, Matida A. Promovendo e analisando o uso e a influência das pesquisas avaliativas: desafios e oportunidades ao se institucionalizar a avaliação em saúde. In: Hartz ZMA, Felisberto E, Vieira-da-Silva LM, organizadores. Meta-avaliação da atenção básica em saúde: teoria e prática. Rio de Janeiro: Editora Fiocruz ; 2008. p. 325-40. and symbolic use is applied to legitimize pre-defined decisions or to support argumentation about a specific action.¹⁰10. Sandison P. The utilisation of evaluations. In: Slim H, Bonwick A. Protection: an ALNAP guide for humanitarian agencies. London: Oxfam; 2006 [cited 2019 Jun 21]. Chapter 3. p. 89-144. Available from: Available from: https://www.alnap.org/help-library/protection-an-alnap-guide-for-humanitarian-agencies
https://www.alnap.org/help-library/prote... Considering that the entire evaluation process may be useful to the program, procedural use is linked to behavior changes in individuals or in the organization, as a result of evaluation, e.g. sharing experiences and commitment of those involved.¹¹11. Patton MQ. Utilization-focused evaluation. Thousand Oaks: Sage publications; 2008.^,1212. Mark MM. Toward better research on-and thinking about-evaluation influence, especially in multisite evaluations. N Direct Eval [Internet]. 2011 Mar [cited 2019 Jun 21];2011(129):107-19. Available from: Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.359 . doi: 10.1002/ev.359
https://onlinelibrary.wiley.com/doi/abs/...

By using PMAQ-AB the following is expected: decision-making substantiation; work process enhancement; improved communication; learning; knowledge management support; accountability; certification of best practices; strategic negotiation; empowerment of those involved; receptiveness to evaluation; and changes in the way of thinking and in organizational culture.

An important contribution made by the program has been the increase in resources allocated to Primary Care, as well as capacity-building in evaluation based on the use of self-evaluation and external evaluation.¹³13. Pinto HA, Sousa ANA, Ferla AA. O programa nacional de melhoria do acesso e da qualidade da atenção básica: várias faces de uma política inovadora. Saúde Debate [Internet]. 2014 out [citado 2019 jun 21];38(esp):358-72. Disponível em: Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext_pr&pid=S0103-11042014000600358&tlng=pt . doi: 10.5935/0103-1104.2014S027
http://www.scielo.br/scielo.php?script=s... However, no studies on this topic have been published yet. There is a gap in this knowledge and, consequently, the need exists to identify the use of a program with these dimensions. To this end, a multiple case study will be conducted, by means of semi-structured interviews in municipalities of Santa Catarina state that have more than 100,000 inhabitants and where health teams have been certified as being “above average” and “well above average” according to cycles I and II of the PMAQ-AB in order to further develop program use.

References

Publication Dates

History