ABSTRACT

OBJECTIVE

To analyze scopes and limits of the use of participatory methodology of evaluation with municipal health managers and administrators.

METHODS

Qualitative research with health policymakers and managers of the Comissão Intergestores Regional (CIR – Regional Interagency Commission) of a health region of the state of Sao Paulo in Brazil. Representatives from seven member cities participated in seven workshops facilitated by the researchers, with the aim of assessing a specific problem of the care line, which would be used as a tracer of the system integrality. The analysis of the collected empirical material was based on the hermeneutic-dialectic methodology and aimed at the evaluation of the applied participatory methodology, according to its capacity of promoting a process of assessment capable to be used as a support for municipal management.

RESULTS

With the participatory approach of evaluation, we were able to promote in-depth discussions with the group, especially related to the construction of integral care and to the inclusion of the user’s perspective in decision-making, linked to the search for solution to concrete problems of managers. By joint exploration, the possibility of using data from electronic information systems was opened, as well as information coming directly from the users of the services, to enhance discussions and negotiations between partners. The participants were disbelievers of the replication potential of this type of evaluation without the direct monitoring of the academy, given the difficulty of organizing the process in everyday life, already taken by emergency and political issues.

CONCLUSIONS

Evaluations of programs and services carried out within the Regional Interagency Commission, starting from the local interest and facilitating the involvement of its members by the use of participatory methodologies, can contribute to the construction of integral care. To the extent that the act of evaluating stay invested with greater significance to the local actors, its involvement with the evaluations at the federal level can also be stimulated.

Health Manager; Decision-making; Program Evaluation; Health Services Evaluation; Participatory Evaluation; Brazilian Unified Health System; Qualitative Research

INTRODUCTION

The evaluation in health remains quite studied and discussed in the world scenario¹⁰10. Fernandes FMB, Ribeiro JM, Moreira MR. Reflexões sobre avaliação de políticas de saúde no Brasil. Cad Saude Publica. 2011;27(9):1667-77. DOI:10.1590/S0102-311X2011000900002
https://doi.org/10.1590/S0102-311X201100... . The crisis in the health sector calls for reflection on the results achieved to improve the performance of the system⁵5. Contandriopoulos AP. Avaliando a institucionalização da avaliação. Cienc Saude Coletiva. 2006;11(3):705-11. DOI:10.1590/S1413-81232006000300017
https://doi.org/10.1590/S1413-8123200600... , and the demand for transparency in the management of public resources remains a priority on the agenda.

In Brazil, the publications approach many aspects of this topic with considerable depth. There are studies on the constitution of the field of evaluation¹⁴14. Furtado JP, Vieira-da-Silva LM. A avaliação de programas e serviços de saúde no Brasil enquanto espaço de saberes e práticas. Cad Saude Publica. 2014;30(12):2643-55. DOI:10.1590/0102-311X00187113
https://doi.org/10.1590/0102-311X0018711... , on the process of institutionalization of the ongoing evaluation in the Country⁸8. Felisberto E. Da teoria à formulação de uma Política Nacional de Avaliação em Saúde: reabrindo o debate. Cienc Saude Coletiva. 2006;11(3):553-63. DOI:10.1590/S1413-81232006000300002
https://doi.org/10.1590/S1413-8123200600... , and reports and robust analyses of experiences carried out³3. Brito-Silva K, Bezerra AFB, Chaves LDP, Tanaka OY. Integralidade no cuidado ao câncer do colo do útero: avaliação do acesso. Rev Saude Publica. 2014;48(2):240-8. DOI:10.1590/S0034-8910.2014048004852
https://doi.org/10.1590/S0034-8910.20140... ,¹²12. Figueiró AC, Hartz Z, Samico I, Cesse EAP. Usos e influência da avaliação em saúde em dois estudos sobre o Programa Nacional de Controle da Dengue. Cad Saude Publica. 2012;28(11):2095-105. DOI:10.1590/S0102-311X2012001100009
https://doi.org/10.1590/S0102-311X201200... .

Within the Brazilian Ministry of Health (MS), a very fruitful period for the field took place between 2003 and 2008, represented by the Programa de Expansão e Consolidação da Saúde da Família (PROESF – Program of Expansion and Consolidation of Family Health). Together with the establishment of the National Policy for Evaluation of Primary Health-care, which sought to solidify the process of institutionalization of the evaluation under the perspective of decentralization, a great investment was carried out, which, among other actions, has enhanced the creation of collaborating centers in evaluation in different institutions of higher education in the Country and encouraged a large reflective process on the topic⁹9. Felisberto E, Freese E, Bezerra LCA, Alves CKA, Samico I. Análise da sustentabilidade de uma política de avaliação: o caso da atenção básica no Brasil Cad Saude Publica. 2010;26(6):1079-95. DOI:10.1590/S0102-311X2010000600003
https://doi.org/10.1590/S0102-311X201000... ,¹⁵15. Hartz ZMA, Felisberto E, Vieira-da-Silva L, organizadores. Meta-avaliação da atenção básica à saúde: teoria e prática. Rio de Janeiro (RJ): Editora Fiocruz; 2008..

Currently, the Department of Monitoring and Evaluation of SUS (DEMAS)^aa Ministério da Saúde, Secretaria-Executiva, Subsecretaria de Assuntos Administrativos, Coordenação-Geral de Gestão de Pessoas. Por dentro do Ministério da Saúde: orientações aos novos servidores. Brasília (DF); 2013. Available from: http://bvsms.saude.gov.br/bvs/publicacoes/por_dentro_ministerio_saude_orientacoes.pdf , created in 2011, set the establishment of a “Evaluation System for the Qualification of SUS”²⁰20. Minayo MCS. O desafio do conhecimento: pesquisa qualitativa em saúde. São Paulo: Hucitec; 2004., composed of evaluation instruments such as the Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ – National Program for Improving Access and Quality of Primary Health-care)^bb Ministério da Saúde, Secretaria de Atenção à Saúde, Departamento de Atenção Básica. Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ): manual instrutivo. Brasília (DF); 2012. (Série A. Normas e Manuais Técnicos). Available from: http://189.28.128.100/dab/docs/publicacoes/geral/manual_instrutivo_pmaq_site.pdf and the Índice de Desempenho do Sistema Único de Saúde (IDSUS – Index of Performance of the Unified Health System)^cc Ministério da Saúde. Índice de Desempenho do SUS – IDSUS. Brasília (DF). Available from: http://idsus.saude.gov.br , with the aim of “producing, by evaluations, a set of necessary and strategic information to the development and qualification of SUS”²²22. Oliveira PTR, Sellera PEG, Reis AT. O monitoramento e a avaliação na gestão do Ministério da Saúde. Rev Bras Monit Aval. 2013;5:114-29..

We placed such initiatives as part of “monitoring and evaluation macrosystems”¹⁶16. Jannuzzi PM. Sistemas de monitoramento e avaliação de programas sociais: revisitando mitos e recolocando premissas para sua maior efetividade na gestão. Rev Bras Monit Aval. 2013;5:4-27., aimed at producing information of broad thematic scope, with enough depth to the management monitoring of the programs, allowing us to analyze their performance and set the necessary adjustments to the policies. Aimed at health-care at the federal level, they offer the same evaluation format for all regions of Brazil, using the same judging criteria, thus complicating the decentralized decision-making and, therefore, the adjustment of the programs to the different contexts in which they develop.

We can observe, in the field, a gap in the study of evaluations carried out at the local level, aimed at achieving a deeper level of understanding of the organization of services, including relationships between certain contexts and formats.

The Brazilian Unified Health System (SUS) idealizes the strengthening of the local management²⁴24. Silva SF, Souza NM, Barreto JOM. Fronteiras da autonomia da gestão local de saúde: inovação, criatividade e tomada de decisão informada por evidências. Cienc Saude Coletiva. 2014;19(11):4427-38. DOI:10.1590/1413-812320141911.16612013
https://doi.org/10.1590/1413-81232014191... , and processes to stimulate the development of territorial governance are in progress. The study of local evaluations would add tools in this same direction.

It is important, moreover, to point out the special difficulty in finding studies on methodologies and structuring of evaluations that move away from the standardized application of normative and quantitative indicators and that take as a basis different labels for different sociocultural contexts²2. Bomfim MI, Goulart VMP, Oliveira LZ. Formação docente na área da saúde: avaliação, questões e tensões. Interface (Botucatu). 2014;18(51):749-58. DOI:10.1590/1807-57622013.0974
https://doi.org/10.1590/1807-57622013.09... .

The evaluation, especially of participatory approach, can become an instrument of learning for local actors, to the extent that it addresses their questions and contribute with relevant information to decision-making, allowing to expand the potential for changes in municipal practices of work and management.

Many definitions of “participation” and classifications of “participatory approaches” exist. However, one aspect that seems common to all of them, and is one of the features of the process that will be analyzed in this article, is the emphasis on the horizontal partnership established between evaluation specialists and people interested on the object evaluated, aiming at the joint production of evaluative knowledge.

Since we worked with a group of managers, we decided to use the Pragmatic Participatory Evaluation referential⁶6. Cousins JB, Whitmore E. Framing participatory evaluation. N Dir Eval. 1998;(80):5-23. DOI 10.1002/ev.1114
https://doi.org/10.1002/ev.1114... , centered on facilitating the use of the evaluation process in the instrumental (support to decision-making), conceptual (focused on the actors’ learning), and political dimensions (facilitating negotiations between partners).

Routinely, the attention of municipal health managers is monopolized by emergency situations in the technical and political context, as well as by information demands of the state and federal levels, rather than by reflexive actions of planning and evaluation. Often, the lack of professional training, especially in small cities²¹21. Nickel DA, Natal S, Hartz ZMA, Calvo MCM. O uso de uma avaliação por gestores da atenção primária em saúde: um estudo de caso no Sul do Brasil. Cad Saude Publica. 2014;30(12):2619-30. DOI:10.1590/0102-311X00022314
https://doi.org/10.1590/0102-311X0002231... ,²⁵25. Spedo SM, Tanaka OY, Pinto NRS. O desafio da descentralização do Sistema Único de Saúde em município de grande porte: o caso de São Paulo, Brasil. Cad Saude Publica. 2009;25(8):1781-90. DOI:10.1590/S0102-311X2009000800014
https://doi.org/10.1590/S0102-311X200900... , is also associated with the restricted use of these two management tools by them.

This study analyzed the application of a methodology for evaluation at the local level and in participatory basis, focused on understanding the meaning that the organization of health services acquires for the regional actors. We assume that the complementarity of this model with the initiatives at the federal level will allow a potentiation of both.

METHODS

We used the methodology of single case study²⁶26. Yin RK. Estudo de caso: planejamento e métodos. 5.ed. Porto Alegre: Bookman; 2015., which allows enhancing the discussion on evaluation strategies at the local level, as well as identifying significant topics for the organization of health services that are common to the diversity of existing contexts.

The studied health region belongs to the state of Sao Paulo and was chosen because of potentially favorable factors for the feasibility of the research and that are present in several real spaces of SUS implementation: (i) policymakers and managers with an active participation in the Regional Interagency Commission (CIR) and in process of group strengthening, potentially increasing their sensitivity as to the utility of evaluative processes; (ii) reasonable installed capacity in the health area – taking all the cities –, constituting a variety of elements that would tend to enrich the process of formulation of evaluative questions.

A group with managers who participate of CIR was formed, from seven^dd Participated in the research: the hub city of the region, class VI of population size (from 100,001 to 500,000 inhabitants), a city of class V (from 50,001 to 100,000 inhabitants), two cities of class IV (20,001 to 50,000 inhabitants), two cities of class III (10,001 to 20,000 inhabitants) and one city of class I (up to 5,000 inhabitants), according to the classification used by IBGE/2011. of the 18 cities that constitute it. The proposed goal was to evaluate a specific problem in a line of care provided by the health region, which would be chosen by them later.

The proposal was presented in one of the meetings of this body and the accession of managers was voluntary. There was a direct invitation to specific cities, made by participants that were first interested by the proposal, to form a diverse group in terms of population size and presence or absence of carceral and indigenous population.

As already mentioned, the process was developed based on a participative methodology, understood here according to the Cousins and Whitmore referential⁶6. Cousins JB, Whitmore E. Framing participatory evaluation. N Dir Eval. 1998;(80):5-23. DOI 10.1002/ev.1114
https://doi.org/10.1002/ev.1114... . This choice led all the decisions of the process and aimed to ensure the relevance of the process to the participants, creating a space for learning and facilitating the use of evaluative findings.

For triggering activities, some activities with defined goals have been proposed to the group (Table 1). From this, the steps that followed were based on the participants’ demands.

The topic initially chosen by the group for the evaluation was the flow of users between the primary health-care and the medium and high complexity health-care. Because of the coverage and diversity of the specialties involved, the elective general surgery has been identified as tracer of the process (according to Kessner’s et al.¹⁷17. Kessner DM, KalK CE, Singer J. Assessing health quality: the case for tracers. N Engl J Med. 1973;288(4):189-94. DOI:10.1056/NEJM197301252880406
https://doi.org/10.1056/NEJM197301252880... definition). Then, the group focused in cholecystectomy surgeries, according to the greater number of cases and long waiting time, aiming at a non-exhaustive data collection, but that would bring information about the main aspects to be evaluated and with potential in the decision-making of local managers.

Seven workshops were held and the steps of construction of the evaluation listed in Table 2 were covered. All workshops were recorded and the audios, transcribed.

The empirical material was analyzed by hermeneutic-dialectic methodology¹⁹19. Minayo MCS. Hermenêutica-dialética como caminho do pensamento social. In: Minayo MCS, Deslandes SF, organizadores. Caminhos do pensamento: epistemologia e método. Rio de Janeiro (RJ): Editora Fiocruz; 2002. p.83-108.,²⁰20. Minayo MCS. O desafio do conhecimento: pesquisa qualitativa em saúde. São Paulo: Hucitec; 2004.. This approach preserves the richness of the collected qualitative material and values the context of the health region where the evaluation took place.

This study was approved by the Ethics Committee from the Faculdade de Saúde Pública of Universidade de São Paulo – FSP/USP (Opinion 1,006,380). All participants signed the informed consent form.

RESULTS AND DISCUSSION

The use of pragmatic participatory methodology⁶6. Cousins JB, Whitmore E. Framing participatory evaluation. N Dir Eval. 1998;(80):5-23. DOI 10.1002/ev.1114
https://doi.org/10.1002/ev.1114... allowed the workshops to settle in a democratic way and to be directed by the direct interests of policymakers and managers. The gathered empirical material showed three results that portray the managers’ appropriation of the process and that are configured as axes of analysis:

• Deepening the understanding regarding the organization of adult health-care in the defined region;

• Appropriation of tools for survey and analysis of quantitative and qualitative data;

• Possibilities for action.

Below, the item related to the replication of the evaluation process is presented, which also emerged from the participants’ suggestions, and arises here as one of the main limitations to the use of the methodology.

Deepening the Understanding Regarding the Organization of Adult Health-care in the Defined Region

Facilitating a highly reflective space for managers and keeping proximity to everyday problems, the process undertaken increased the participants’ view about the dynamics of SUS’s current operation, whose discussion usually do not have space in the predominantly administrative CIR meetings²¹21. Nickel DA, Natal S, Hartz ZMA, Calvo MCM. O uso de uma avaliação por gestores da atenção primária em saúde: um estudo de caso no Sul do Brasil. Cad Saude Publica. 2014;30(12):2619-30. DOI:10.1590/0102-311X00022314
https://doi.org/10.1590/0102-311X0002231... .

The choice of elective surgeries portrayed the concern with the construction of integral care, directly linked to the possibility of intergovernmental negotiation. The group discussed the autonomy of municipal managers facing decisions and inductions of policies coming from the state or federal management¹⁸18. Machado CV, Lima LD, Viana ALD, Oliveira RG, Iozzi FL, Albuquerque MV, et al. Federalismo e política de saúde: comissões intergovernamentais no Brasil. Rev Saude Publica. 2014;48(4):642-50. DOI:10.1590/S0034-8910.2014048005200
https://doi.org/10.1590/S0034-8910.20140... and the division between the different realms of management and the different rationales present in the field – funding logic, understanding of the role of the federated entities in the organization of the system¹¹11. Feuerwerker L. Modelos tecnoassistenciais, gestão e organização do trabalho em saúde: nada é indiferente no processo de luta para a consolidação do SUS. Interface (Botucatu). 2005;9(18):489-506. DOI:10.1590/S1414-32832005000300003
https://doi.org/10.1590/S1414-3283200500... . They also approached the practice of decentralization that occur in the Country without the necessary planning for prior expansion of the administrative and institutional capacity of the cities²³23. Santos AM, Giovanella L. Governança regional: estratégias e disputas para gestão em saúde. Rev Saude Publica. 2013;48(4):622-31. DOI:10.1590/S0034-8910.2014048005045
https://doi.org/10.1590/S0034-8910.20140... .

Our survey has many punctual demands, such as “urgency and emergency” and “difficulty in the access to medium complexity”. If we have to start by topics that are under the governance of cities, I would say that none of them is. It’s no use just the cities. The state has to work for the whole system work... It’s a gear, a machine... (Municipal Manager 1)

We here, facing a hospital bed crisis... You should have a shared system, of partners that are involved in the same problem, and what’s the answer that we have from the state government? “It’s a ’municipal fault, which doesn’t invest in hospital care”. So, how can we deal with this? (Municipal Manager 2)

But I doubt if this is because we have no structure... Who conducts the surgeries? It’s mainly the State Hospital for city X and for the others. With what logic does the city works, and, to some extent, as consequence, the AME [Outpatient Department of Specialties] itself works? With the logic of the population needs. With what logic does the State Hospital works? With the logic of terms they made. They’re going to conduct 500 surgeries, “oh, but I have 3,000 patients”. Then he thinks: “If I do this here, we’re going to spend such amount of money”. They have an economic system there... (Municipal Manager 3)

At the time of analysis of the data collected among the users, they discussed the need for systematic collection and use of this type of information in the management. They emphasized that the acquaintance of the experiences lived by the main interested ones in the service¹1. Almeida CAL, Tanaka OY. Perspectiva das mulheres na avaliação do Programa de Humanização do Pré-Natal e Nascimento. Rev Saude Publica. 2009;43(1):98-104. DOI:10.1590/S0034-89102009000100013
https://doi.org/10.1590/S0034-8910200900... makes the discussions more significant and allows the deepening of the analyses.

They unfolded the discussion to another facet of integral care: the consideration to the integral being of the user and to the health needs brought by him⁴4. Cecílio LCO, Merhy EE. A integralidade do cuidado como eixo da gestão hospitalar. In: Pinheiro R, Mattos RA, organizadores. Construção da integralidade: cotidiano, saberes e práticas em saúde. 4.ed. Rio de Janeiro: IMS/UERJ; CEPESC - ABRASCO; 2007. p.197-210, including the management of directions and the provision of information about the operation of the services.

The doctor, when we pass, says “there’s a bomb inside you”. And then you go to the line... Who sleeps like this? (User 1)

I had a crisis that I thought I was going to die, I was taking saline at the prompt service... That’s when I got in touch with the doctor of my city, because I knew he operated for another city, and I went after him. I paid the consultation to see him and he operated on me in the other city, and by the SUS! If I had waited for AME, I would be dead today. (User 2)

The user, in addition to being in the hands of two managers (state and municipal), one does not know what the other is doing. Neither the user or us know where he is on the waiting list, which is the prediction of his attendance. (Municipal Manager 1)

Appropriation of Tools for Survey and Data Analysis

Managers and administrators often stated that the information available to the city regarding the user’s path always stopped when the patient was referred to the equipment under state management. An initial joint study of the reports provided by the CROSS system (Coordination Center of Offer of Health Services) showed the usefulness of an effort of appropriation of the information contained there, often devalued, or even unknown, by the managers. Similarly, the review of internal processes of the city also stood out. Information that could be collected in the municipal regulatory services were routinely lost.

Then she just didn’t go see a surgeon in our municipal network, but simply went to AME. Then I managed to get into the AME system. I can’t figure out why she stayed only in the AME. (Municipal Manager 2)

They used the Tabwin program, provided by the Ministry of Health to consult the data from the Hospital Information System, and the majority of managers used it for the first time. They valued the gains and argumentations with partners, which arised from the generated information.

The group observed that both the work done by AME and the reference State Hospital took place autonomously regarding the actions of primary health-care. Each federated entity preserves its autonomy, focusing on the role formally granted to it, and not on the networking.

To do this reasoning, one had to add the urgency that you make in the city to the urgency that the State makes. If the offer of the elective was higher, the urgency would be smaller. (Municipal Manager 2)

You will come to the conclusion that what is agreed within the management contract with the provider is not answering the need... If neither of our Collegiate Body, let alone the whole network in which the state is a reference... (Municipal Manager 1)

Possibilities for action

In the workshops, the group noted that any proposal for a change in the system would involve a review of all services available under the different levels of management in the territory – including their own. The review of the primary and emergency units while referral sources, of the regulation and use of the CROSS system, as well as of the management of charitable hospitals, proved to be fruitful.

In the CROSS system, 48 hours before we can no longer change the name of the patient. If he gives up because of a misadventure, if a urgent last minute case appears, we cannot insert anymore... With that, he ends up missing the consultation. It was time to review that. (Municipal Manager 1)

Manager 4: In my city there is a Charity Hospital, but it doesn’t have the service... it’s basically a matter of providing. When we had a team of participatory providers, with credibility, which collected funds, it was one reality. After that, nobody else campaigned and basically the city hall maintains the Charity Hospital, then there’s an office that chooses someone and put him there, it’s all political office, nobody understands anything and it stays as it is.

Manager 2: You have to press the provider, as I am doing there, to hire technicians, people hired for career here and teach everything to continue, because otherwise every time a new administrator enters, we’ll have always to say the same things we say.

From reports of moments of political negotiation with partners, we realized that the group started to rely more on technical arguments and in-depth reflections, reflecting a maturing to make possible new action strategies. These are only initial strategies, but that reflect the expansion of proactivity to the establishment of a link between the different governmental actors and private partners.

CIR’s technical chamber has been strengthened as a space of exchange and negotiation with private providers, to the extent that it builds a more balanced power among the participants. Such balance results from the appropriation of knowledge of municipal managers about flows, degree of autonomy of each service, and mechanisms that are at stake in the agreement, among others. Obviously, important limitations remained, e.g., regarding decisions about the organization and terms of services of medium and high complexity health-care that occur centrally in the Health Secretariat of the State of Sao Paulo, which virtually precludes the involvement of municipal managers.

A possible repositioning of the role of existing hospitals in the cities, which would assume part of the demand for general surgery, was considered for new designs of regionalization of the care.

When we talked in the monitoring group of providers about increasing the number of elective surgeries, the hospital director said “we would need to study it, we don’t have more capacity... unless we open the third shift of surgery”, and then suddenly we enter in these possibilities, if we start pushing it somewhat... (Municipal Manager 1)

With the COAP [Organizational Contract of Public Action], we can covenant that it’s going to be the municipal hospital that will give vent to the region, that will attend, make this type of procedure... a negotiation between entities, both the municipal ones... and see which of them can give support. (Municipal Manager 1)

Replication of the Evaluative Process

The respect to the pace of the group did not prevent that all structuring steps of an evaluation were completed in the process, generating appropriation of the “evaluative reasoning”. It is known that the very questioning of the “why”, “when” and “how” evaluate allows a critical analysis of the evaluated object in its complexity.

However, as expected, given the novelty of the process, the participants did not really believe that the process could be replicated regardless of the presence of the facilitators and of the contract that was made. They reiterated that the daily management is taken over by emergency issues and that there would be no room for the construction of this type of process, if it was not for the commitment made with the academy.

I think that before the diversity of the work that we have, I would hardly stop to analyze all the items, I don’t have time for that. I stopped because it was a directed work, of making a commitment... I’m saying that this thinking is not part of the routine. (Municipal Manager 5)

CONCLUSIONS

It is essential to contextualize this study as the use of a methodology that stimulated the involvement of administrators and local managers in the search for answers to questions of direct interest. It was possible to start a process of recognition of the necessity, approximation and appropriation of information, supported by simple measures of data collection, analysis and reflection, which tend to strengthen the fragile situation of the evaluation in the decision-making process.

The participatory process brought consistency to the identification of variables that influence the service evaluated. The adjustment of the offer was identified as an alternative to improve the flow of assistance between primary health-care and medium and high complexity health-care, but the notion that this is not the only possible action was presented. As a result, new fronts of communication could open. The proactivity of policymakers and managers has increased in the search for negotiation with the other levels of management. Even during the process, minor changes, tending to facilitate the flow management, could be combined in person, in local meetings. Obviously, such changes have taken a local range: actors of different levels of management that establish more proximity and leverage changes.

We stress the procedural aspect of this initiative, which, as in any learning process, does not predict or require the reach of large and visible results in a short time.

We aim to contribute to the construction of a culture of evaluation in the local levels of the system, strengthening a proactive attitude of policymakers and managers in the search to use the information for the enrichment of the political processes. Even if supported by the constant presence of the academy, at least in its early stages, the proposed evaluation model enhances evaluations carried out on the macrosystemic level. To discover, for the evaluation, a meaning more connected to improvements in the concrete practice, making it less prescriptive and bureaucratic, can facilitate its transposition into the comprehension of broader spheres.

Important limitations arise from the lack of institutionalization of the process, which had limited organizational insertion. However, experience shows that we hardly begin from ideal conditions: they are built from bottom-up processes.

In that sense, in democratic processes, the matter of participation is exemplary regarding the division between theory and practice. There is a consensus in methodologies that advocate the involvement of different interested entities in the evaluation: that the multiplicity of points of view enriches the construction and analysis of the evaluated process, and tends to leverage the use of its results⁷7. Cousins JB, Whitmore E, Shulha L. Arguments for a common set of principles for collaborative Inquiry. Am J Eval. 2012;34(1):7-22. DOI:10.1177/1098214012464037
https://doi.org/10.1177/1098214012464037... ,¹³13. Furtado JP. A avaliação participativa. In: Otero MR, organizador. Contexto e prática da avaliação de iniciativas sociais no Brasil: temas atuais. São Paulo: Fundação Peirópolis; 2012. p.21-41.. The process in focus has allowed us to identify that the conditions for such participation to occur are not yet in the public service. It takes effort to enhance its construction, in the sense of negotiation in action priorities, building of credibility, and gradual maturation of the process.

REFERENCES

Publication Dates

History