Abstract

Objective:

to evaluate the implementation of the Results Control, Monitoring and Evaluation System (e-Car) at the Health Surveillance Secretariat (SVS), Brazilian Ministry of Health, in the period 2012-2015.

Methods:

this was an evaluation study using mixed methods for collection of primary and secondary data, followed by definition of the system’s degree of implementation and analysis of the influence of the political-organizational context.

Results:

the e-Car System was considered to have been implemented (75.4%) at SVS; the worst scores for its structure dimension related to the computer and operating system maintenance service, as well as little knowledge of the system manual; as for the system’s process dimension, the existence of collegiate bodies was considered beneficial for monitoring.

Conclusion:

the e-Car System had been implemented at SVS; however, its sustainability was fragile, and SVS was recommended to institutionalize and strengthen its monitoring and evaluation practices.

Keywords:
Health Evaluation; Information Systems; Public Policy; Health Planning; Public Health Surveillance

Introduction

In the 1990s in Western democracies in general and in Latin America in particular, concerted efforts were made to strengthen the role of ‘evaluation’ in government management. In several countries, including Brazil, this movement led to systems with different scopes being built in order to evaluate public policies. This was justified by the need to ‘modernize’ public sector management in a context of efforts to dynamize and legitimize State reform.¹1. Faria CAP. A política da avaliação de políticas públicas. Rev Bras Ci Soc [Internet]. 2005 out [citado 2019 abr 30];20(59):97-109. Disponível em: Disponível em: http://www.scielo.br/pdf/rbcsoc/v20n59/a07v2059.pdf . doi: 10.1590/S0102-69092005000300007
http://www.scielo.br/pdf/rbcsoc/v20n59/a... These initiatives reaffirmed the need for an evaluation policy to inform planning and management systems and strengthen monitoring by civil society.²2. Arretche MTS. Tendências no estudo sobre avaliação. In: Rico EM (org). Avaliação de políticas sociais: uma questão em debate. São Paulo: Cortez; 1998. p. 29-39.^,33. Paim JS, Teixeira CF. Política, planejamento e gestão em saúde: balanço do estado da arte. Rev Saúde Pública [Internet]. 2006 ago [citado 2019 abr 30];40(esp):73-8. Disponível em: Disponível em: http://www.scielo.br/pdf/rsp/v40nspe/30625.pdf . doi: 10.1590/S0034-89102006000400011
http://www.scielo.br/pdf/rsp/v40nspe/306...

Given this scenario, systems for monitoring and evaluating public sector management of health systems have been prioritized in government initiatives committed to outcome-based management and evaluation models.⁴4. Hartz ZMA. Avaliação dos programas de saúde: perspectivas teórico metodológicas e políticas institucionais. Ciênc Saúde Coletiva [Internet]. 1999 [citado 2019 abr 30];4(2):341-53. Disponível em: Disponível em: http://www.scielo.br/pdf/csc/v4n2/7117.pdf . doi: 10.1590/S1413-81231999000200009
http://www.scielo.br/pdf/csc/v4n2/7117.p... In Brazil, “outcome-based contracting” was proposed in 1995 by the Ministry of Federal Administration and State Reform. More than an isolated tool, it was seen as a lever for improving public sector management using a results-based approach.⁵5. Pacheco RS. Brasil: avanços da contratualização de resultados no setor público. In: Anais do XI Congreso Internacional del CLAD sobre la Reforma del Estado y de la Administración Pública [Internet]. 2006 nov 7-10; Ciudad de Guatemala: Centro Larinoamericano de Administración para el Desarrollo; 2006 [citado 2019 abr 30]. Disponível em: Disponível em: https://www.researchgate.net/publication/267198557_Brasil_avancos_da_contratualizacao_de_resultados_no_setor_publico/download
https://www.researchgate.net/publication...

This complex institutional movement, which is questioning in view of its reflexive potentiality, implies renegotiation and transfer of responsibilities and attributions. Albeit somewhat late in relation to the implementation of its technical and financial management decentralization, today the relevance of building a monitoring and evaluation (M&E) policy for the Brazilian National Health System is under discussion.⁶6. Campelo LM, Santos EM, Oliveira PTR. Análise estratégica do processo de formulação da PM&A-SUS: lições aprendidas e desafios. Saúde Debate [Internet]. 2017 mar [citado 2019 abr 30]; 41(esp):34-49. Disponível em: Disponível em: http://www.scielo.br/scielo.php?pid=S0103-11042017000500034&script=sci_abstract&tlng=pt. doi: 10.1590/0103-11042017s04
http://www.scielo.br/scielo.php?pid=S010... The adoption of monitoring practices in Brazilian public administration became more institutionally manifest with effect from the 2000-2003 Multiannual Plan (PPA). Since then, monitoring has been made feasible through information systems, which provide government managers with information to strengthen planning and management of public policies.⁷7. Branco MAF. Informação e saúde: uma ciência e suas políticas em uma nova era. Rio de Janeiro: Editora Fiocruz; 2006.

8. Moraes IHS. Política, tecnologia e informação em saúde: a utopia da emancipação. Salvador: Ed. Casa da Qualidade; 2003.^-99. Mota E, Carvalho DM. Sistemas de informação em saúde. In: Rouquayrol MZ, Almeida Filho N, editores. Epidemiologia e saúde. Rio de Janeiro: Medsi Editora Médica e Científica; 2003. p. 605-28.

In 2011, the Brazilian Ministry of Health reached an agreement on a results-based agenda, thus aligning the Multiannual Plan with the Brazilian National Health Plan.⁶6. Campelo LM, Santos EM, Oliveira PTR. Análise estratégica do processo de formulação da PM&A-SUS: lições aprendidas e desafios. Saúde Debate [Internet]. 2017 mar [citado 2019 abr 30]; 41(esp):34-49. Disponível em: Disponível em: http://www.scielo.br/scielo.php?pid=S0103-11042017000500034&script=sci_abstract&tlng=pt. doi: 10.1590/0103-11042017s04
http://www.scielo.br/scielo.php?pid=S010... In 2012 it adopted the Results Control, Monitoring and Evaluation System (e-Car)¹⁰10. Ministério da Saúde (BR). Secretaria Executiva. Departamento de Monitoramento e Avaliação do Sistema Único de Saúde. Planejamento estratégico do Ministério da Saúde 2011-2015: resultados e perspectivas [Internet]. Brasília: Ministério da Saúde; 2013 [citado 2019 abr 30]. Disponível em: Disponível em: http://bvsms.saude.gov.br/bvs/publicacoes/planejamento_estrategico_ministerio_saude_2011_2015_5ed.pdf
http://bvsms.saude.gov.br/bvs/publicacoe... ^,1111. Figueiró AC, Santos MA, Cruz MM, Ubarana JA, Hartz Z. Implementação de um sistema de monitoramento e avaliação de âmbito federal: o caso do e-Car no Departamento de Monitoramento e Avaliação do Sistema Único de Saúde (SUS). An Inst Hig Med Trop [Internet]. 2017 [citado 2019 abr 30]; 16 Suppl 2: S65-S74. Disponível em: Disponível em: https://anaisihmt.com/index.php/ihmt/article/view/59/47 . doi: 10.25761/anaisihmt.59
https://anaisihmt.com/index.php/ihmt/art... as a government monitoring tool, whereby the Ministry’s Secretariats became responsible for updating the system monthly. The e-Car system was developed using easily customized open-source software to meet government needs. Reports filled in on the system were mainly qualitative, containing information about the evolution of targets and results agreed at the planning stage, highlighting the current scenario, critical points and recommendations.

At the Health Ministry’s Health Surveillance Secretariat (SVS), the results were discussed weekly by a collegiate body of managers, and helped them with planning and monitoring results and the achievement of the Secretariat’s mission, namely the collection, analysis and dissemination of health data. This form of monitoring at SVS was articulated under the leadership of the Planning and Budget General Coordination (CGPLAN) division.

Despite progress with aligning management instruments and progress with result monitoring, health information requires a thorough technical, scientific and social approach.⁷7. Branco MAF. Informação e saúde: uma ciência e suas políticas em uma nova era. Rio de Janeiro: Editora Fiocruz; 2006.

8. Moraes IHS. Política, tecnologia e informação em saúde: a utopia da emancipação. Salvador: Ed. Casa da Qualidade; 2003.

9. Mota E, Carvalho DM. Sistemas de informação em saúde. In: Rouquayrol MZ, Almeida Filho N, editores. Epidemiologia e saúde. Rio de Janeiro: Medsi Editora Médica e Científica; 2003. p. 605-28.

10. Ministério da Saúde (BR). Secretaria Executiva. Departamento de Monitoramento e Avaliação do Sistema Único de Saúde. Planejamento estratégico do Ministério da Saúde 2011-2015: resultados e perspectivas [Internet]. Brasília: Ministério da Saúde; 2013 [citado 2019 abr 30]. Disponível em: Disponível em: http://bvsms.saude.gov.br/bvs/publicacoes/planejamento_estrategico_ministerio_saude_2011_2015_5ed.pdf
http://bvsms.saude.gov.br/bvs/publicacoe...

11. Figueiró AC, Santos MA, Cruz MM, Ubarana JA, Hartz Z. Implementação de um sistema de monitoramento e avaliação de âmbito federal: o caso do e-Car no Departamento de Monitoramento e Avaliação do Sistema Único de Saúde (SUS). An Inst Hig Med Trop [Internet]. 2017 [citado 2019 abr 30]; 16 Suppl 2: S65-S74. Disponível em: Disponível em: https://anaisihmt.com/index.php/ihmt/article/view/59/47 . doi: 10.25761/anaisihmt.59
https://anaisihmt.com/index.php/ihmt/art... ^-1212. Lins MJCA. Necessidades e demandas de informações: uma abordagem a partir da vigilância sanitária [dissertação]. Rio de Janeiro: Escola Nacional de Saúde Pública, Fundação Instituto Oswaldo Cruz; 2001. Disponível em: https://www.arca.fiocruz.br/handle/icict/5083
https://www.arca.fiocruz.br/handle/icict... The low use of health information is related mainly to inadequate databases, difficulties with internet connection, lack of personnel training and lack of investment in an information technology culture.⁵5. Pacheco RS. Brasil: avanços da contratualização de resultados no setor público. In: Anais do XI Congreso Internacional del CLAD sobre la Reforma del Estado y de la Administración Pública [Internet]. 2006 nov 7-10; Ciudad de Guatemala: Centro Larinoamericano de Administración para el Desarrollo; 2006 [citado 2019 abr 30]. Disponível em: Disponível em: https://www.researchgate.net/publication/267198557_Brasil_avancos_da_contratualizacao_de_resultados_no_setor_publico/download
https://www.researchgate.net/publication... Difficulties can also be observed in the quality and use of the PPA information monitoring system.¹²12. Lins MJCA. Necessidades e demandas de informações: uma abordagem a partir da vigilância sanitária [dissertação]. Rio de Janeiro: Escola Nacional de Saúde Pública, Fundação Instituto Oswaldo Cruz; 2001. Disponível em: https://www.arca.fiocruz.br/handle/icict/5083
https://www.arca.fiocruz.br/handle/icict... ^,1313. Santos AR. Monitoramento e avaliação de programas no setor público: a experiência do PPA do Governo Federal no período 2000-2011 [monografia]. Brasília: Instituto Serzedello Corrêa (ISC); 2012.

Considering that e-Car system is important for decision-making and considering that there are difficulties in information system implementation processes,¹¹11. Figueiró AC, Santos MA, Cruz MM, Ubarana JA, Hartz Z. Implementação de um sistema de monitoramento e avaliação de âmbito federal: o caso do e-Car no Departamento de Monitoramento e Avaliação do Sistema Único de Saúde (SUS). An Inst Hig Med Trop [Internet]. 2017 [citado 2019 abr 30]; 16 Suppl 2: S65-S74. Disponível em: Disponível em: https://anaisihmt.com/index.php/ihmt/article/view/59/47 . doi: 10.25761/anaisihmt.59
https://anaisihmt.com/index.php/ihmt/art... the objective of this study was to evaluate the implementation of the e-Car system at the Health Surveillance Secretariat (SVS) of the Brazilian Ministry of Health, during the period in which the 2012-2015 PPA was in force.

Methods

This was an implementation evaluation case study, using a combination of methods to estimate the level of e-Car system implementation at SVS and to verify the influence of the political-organizational context on this process during the period in which the 2012-2015 PPA was in force.

Managers and technicians responsible for monitoring SVS health surveillance actions were the participants of this case study. The actions in question included the following topics: communicable disease surveillance, prevention and control; surveillance of risk factors for the development of chronic non-communicable diseases; environmental and occupational health surveillance; and analysis of the health situation of the Brazilian population.

Initially, 36 people were invited to participate in the study, these being 100% of the stakeholders involved with the e-Car system at SVS: technicians who had accompanied the system for longer; and managers, responsible for the results agreed when planning health surveillance actions.

Because of political transition and the dismissal of some staff, there was a 30% loss of respondents, with 25 people remaining, including technicians, coordinators, directors and other informants, distributed over five SVS departments, identified here as Departments A, B, C, D and E.

The evaluation design considered the internal and external contexts. For the internal context, we used the Intervention Logic Model (ILM) on the basis of the structural components defined by Donabedian.¹⁴14. Donabedian A. Basic approaches to assessment: structure, process and outcome. In: Donabedian A. Explorations in quality assessment and monitoring. Ann Arbor, Michigan: Health Administration Press; 1980. p. 77-125 This allowed the organization of the e-Car system to be modeled according to its structure, process and outcome dimensions (Figure 1).¹⁵15. Guimarães EAA. Avaliação da implantação do Sistema de Informação sobre nascidos vivos em municípios de Minas Gerais [tese]. Belo Horizonte (MG): Fundação Instituto Oswaldo Cruz; 2011. Disponível em: https://www.arca.fiocruz.br/handle/icict/4286
https://www.arca.fiocruz.br/handle/icict...

The ILM guided the definition of an analysis and judgment matrix (Figure 2), with indicators and parameters that were used as an ideal standard with which to compare the study findings. Although ILM design covers the structure, process and outcome dimensions, the decision was taken to evaluate only the structure and process dimensions. Regarding the external context, we considered two of the three vertices of the Matus Government Triangle:¹⁶16. Matus C. Política, planificacion y gobierno. Washington, D.C: Borrador; 1987. government capacity; and governability. These dimensions and vertices, rather than all of them, were proposed for evaluation because of the limited amount of time available to perform the analysis.

The ILM and the analysis and judgment matrix were validated by stakeholders involved with the e-Car system at SVS. This measure was decisive for the preparation of the questionnaires and interview scripts. The primary data were collected by electronic mail, by means of semi-structured questionnaires administered to 20 SVS interviewees (five coordinators and 15 technicians), in order to define the level of implementation.

In addition, scripts were developed to guide the on-site collection of information from the other five key informants comprised of directors and technicians. This enabled the analysis of the political-organizational context. The interviews were recorded using digital equipment and later transcribed. Secondary data collection was done by consulting reports retrieved from the e-Car system, institutional materials and published scientific articles. The entire validation and data collection process occurred from July to September 2016.

Analysis of the data in order to define the level of implementation was done by adding together the scores for the answers related to the structure and process components, dividing the total by the maximum score set by the analysis and judgment matrix, and multiplying by 100. The e-Car system was classified as follows: implemented (>75%); partially implemented (50 to 75%); incipient (25 to 49%); and not implemented (<25%).

Analysis of the political-organizational context was based on documents and reports of SVS directors and other informants. This information was examined and categorized using content analysis. The techniques used to analyze and describe the messages found in the contents were based on the presence or frequency of sentences, words and outlines with common characteristics among the interviewees (Figure 3).¹⁷17. Bardin L. Análise de conteúdo. São Paulo: Edições 70; 2011. E-Car weaknesses and strengths were identified through the semi-structured interviews and were subsequently linked to aspects regarding the context reported by managers and other informants, as well as through document observation and analysis.

The study project was approved by the Research Ethics Committee of the Sergio Arouca/Oswaldo Cruz Foundation National School of Public Health (ENSP/Fiocruz): Certification of Submission for Ethical Appraisal (CAAE) No. 1,542,745, dated May 13^th, 2016. The participants signed a Free and Informed Consent Form. The confidentiality of the stakeholders and management bodies included in the evaluation was ensured.

Results

The e-Car system was found to have been implemented at a level of 75.4% at SVS in the period analyzed, proportionately distributed between 72.5% implementation for the structure dimension and 78.3% implementation for the process dimension (Table 1). The worst structure performances were related to the computer and operating system maintenance service, reported by 11 of the 20 respondents, and knowledge of the existence of the e-Car system instruction manual, which only eight participants knew about. In the process dimension, eight out of 20 respondents said they received regular training, 1 to 2 times a year, and six of them knew about sporadic training, with no annual regularity. Eleven participants reported that positive change had occurred in the preparation of reports and monitoring mechanisms between 2012 and 2015.

The level of implementation was influenced by the political-organizational context of SVS in the period 2012-2015, considering governability/sustainability and government ability. The five people interviewed following the interview script stated that the system had strong visibility within SVS. However, more concrete incorporation of the system so as to ensure sustainability would depend on a profound change in institutional culture:

According to the interviewees, the system’s sustainability is linked to the manager’s political will. Even though the main activities foreseen on an agenda arise from a political decision, individually, each manager has their own views and wants to leave the mark of their term of office in government, which ends up condemning planning, monitoring and in particular evaluation policies to a short lifespan.

In four years, the Brazilian Ministry of Health had four ministers and SVS had two secretaries. This influenced how the e-Car system was monitored and implemented. For informant 2, this instability in the planning and monitoring process can create distrust as to the performance of the area and, consequently, make the system more vulnerable in relation to its sustainability.

Regarding the government capacity vertex, human resources were seen as insufficient. For the interviewees, the large number of consultants working at SVS contributed to service instability: they reported high staff turnover, as a result of civil servants constantly changing their working area or consultants being dismissed. At times, new staff designated to system monitoring did not know the system or had not been trained to operate it.

In relation to the peculiarities between the different SVS departments, the status of Department A stood out. It had its own technological network and up to date computers, having a structure level different from the rest of the Brazilian Ministry of Health. This difference was also perceived in the organization of Monitoring and Evaluation (M&E) processes: Department A had a specific committee for monitoring e-Car system results.

The other SVS departments were located far from the Ministry’s main building, supported by a precarious structure, with constant internet connection problems. In the case of Department B, there was a further aggravating factor: its director did not work in the same building as the sections under his responsibility. The report given by one of the informants showed the shortcomings of this physical separation within the same structure:

Solving the structural part appeared to be more complex and challenging, since, with the exception of Department A, the other departments had little autonomy in this field. Other structural difficulties in relation to the organization of the e-Car system were the gaps found in standardization, since there was no Ministerial Ordinance to define the system as a monitoring tool, as well as little knowledge about the existence of the system manual.

Improved quality of system monitoring and use was guaranteed by the provision of training courses and efforts to qualify and standardize SVS reports. When we compared reports issued between 2012 and 2015, it was possible to observe an evolution in the quality of analysis carried out by the SVS departments and coordination bodies.

The existence of collegiate management bodies was also seen as beneficial to the monitoring process, since it made possible a series of mobilizations, right from the technical level to the level of the Secretary:

For the interviewees, putting into practice solutions for the problems found during the collegiate body meetings made reports more objective. Another stimulus for this enhancement happened when external control organs started to follow the evolution of the results posted on the e-Car system.

It is important to highlight that e-Car identifies the person responsible for the targets monitored. The service manager is held accountable at collegiate meetings and by organs responsible for monitoring targets, such as the Brazilian Ministry of Planning, Development and Management, the Brazilian Office of the Comptroller General (CGU) and the Brazilian Federal Court of Auditors (TCU).

The system’s qualitative content has contributed to the evolution of targets and to the clarification and resolution of problems. The system design logic and the process of monthly information input and monitoring, which are the subject of weekly discussions at the collegiate body, were found to have supported the decision-making process in a continuous manner.

All the enhancement of the monitoring process carried out at that time allowed e-Car results to be used in planning and monitoring, contributing to the departments’ management and to the preparation of the next PPA for the period 2016-2019.

The influence of the SVS political-organizational context on the level of implementation of the e-Car system found was characterized by means of strengths and weaknesses, as shown in Figure 4. Among the strengths in structure was the recognition of the people responsible for the results, while weaknesses related to problems with system access and settings. In terms of processes, information for decision-making was perceived as a strength, while risks of the system becoming a bureaucratic tool and the lack of a policy to establish M&E were identified as weaknesses.

Discussion

The same SVS Secretary remained in office from 2012 to 2014, positively influencing the improvement of the monitoring of results agreed in the 2012-2015 Strategic Planning⁶6. Campelo LM, Santos EM, Oliveira PTR. Análise estratégica do processo de formulação da PM&A-SUS: lições aprendidas e desafios. Saúde Debate [Internet]. 2017 mar [citado 2019 abr 30]; 41(esp):34-49. Disponível em: Disponível em: http://www.scielo.br/scielo.php?pid=S0103-11042017000500034&script=sci_abstract&tlng=pt. doi: 10.1590/0103-11042017s04
http://www.scielo.br/scielo.php?pid=S010... using e-Car. It is known that institutionalized monitoring is essential to correct directions and achieve targets and expected outcomes for government plans.¹¹11. Figueiró AC, Santos MA, Cruz MM, Ubarana JA, Hartz Z. Implementação de um sistema de monitoramento e avaliação de âmbito federal: o caso do e-Car no Departamento de Monitoramento e Avaliação do Sistema Único de Saúde (SUS). An Inst Hig Med Trop [Internet]. 2017 [citado 2019 abr 30]; 16 Suppl 2: S65-S74. Disponível em: Disponível em: https://anaisihmt.com/index.php/ihmt/article/view/59/47 . doi: 10.25761/anaisihmt.59
https://anaisihmt.com/index.php/ihmt/art... ^,1818. Maia LVA, Silva MS, Salviano VM. O monitoramento das prioridades de governos no Ministério do Planejamento. Rev Bras Monit Aval [Internet]. 2013 jan-jun [citado 2019 abr 30];5:28-47. Disponível em: Disponível em: https://aplicacoes.mds.gov.br/sagirmps/ferramentas/docs/RBMAs/RBMA_5.pdf
https://aplicacoes.mds.gov.br/sagirmps/f... ^,1919. Oliveira PTR, Sellera PEG, Reis AT. O monitoramento e a Avaliação na Gestão do Ministério da Saúde. Rev Bras Monit Aval [Internet]. 2013 jan-jun;(5):114-29. Disponível em: http://idsus.saude.gov.br/documentos/Artigo_M&A_RBMA_5.pdf
http://idsus.saude.gov.br/documentos/Art... However, its sustainability depends on many contextual factors.

Because of more profound changes in the political context of the Brazilian Ministry of Health in 2015, including within SVS, monitoring began to be carried out on a less regular basis. This decline was felt more strongly in 2016, when the e-Car process was disrupted at the Ministry of Health and, therefore, also at SVS, given the impact of the change in the President of the Republic, which had a direct cascade effect on changes in government management both at ministerial level and within the Ministry’s secretariats.

Although this sudden change of political direction occurred after this study was conducted, it must be mentioned as a driving force for political-organizational changes within the Ministry of Health, involving the dismissal of managers and the weakening of SUS monitoring and evaluation services. In 2016, it took almost six months to update the new e-Car system database of results to be monitored in the period 2016 to 2019. This resulted in discontinuity of data input and, consequently, discontinuity of monitoring by SVS.

This situation shows how institutionalization of M&E is still fragile and remains at the mercy of the political will of the moment. Leeuw and Furubo²⁰20. Leeuw FL, Furubo JE. Evaluation systems: what are they and why study them? Evaluation [Internet]. 2008 Apr [cited 2008 Apr 1);14(2):157-69. Available from: Available from: http://evi.sagepub.com/cgi/content/abstract/14/2/157 . doi: 10.1177/1356389007087537
http://evi.sagepub.com/cgi/content/abstr... ^,2121. Serpa SMHC. Uma leitura dos usos da avaliação na administração pública no Brasil a partir da caracterização dos sistemas de avaliação de programas governamentais (tese). Brasília (DF): Universidade de Brasília; 2014. Disponível em: http://repositorio.unb.br/handle/10482/16562
http://repositorio.unb.br/handle/10482/1... highlighted the difficulty in defining an evaluation system the sustainability of which depends on a more mature evaluative culture, built by the triad “time, investment and permanent arrangements”.²¹21. Serpa SMHC. Uma leitura dos usos da avaliação na administração pública no Brasil a partir da caracterização dos sistemas de avaliação de programas governamentais (tese). Brasília (DF): Universidade de Brasília; 2014. Disponível em: http://repositorio.unb.br/handle/10482/16562
http://repositorio.unb.br/handle/10482/1... ^,2222. Furubo JE, Rist RC, Sandahl R. International atlas of evaluation. New Brunswick, London: Transaction Publishers; 2002. It would appear there was a great deal of investment in the system, technician training and the SVS monitoring process. However, our research identified a weakness in the system’s ability to institutionalize and standardize such practices. Organizational culture needs time and investment to absorb changes more consistently and gain stability. Four years appear to have been a short space of time for the public administration to internalize a system such as e-Car.

The institutionalization of evaluation is perceptible, as if its integration into the organizational system influenced action, connecting analytical activities to intervention management activities.²³23. Hartz ZMA, Vieira LMS. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador: Ed UFBA; 2014.^,2424. Carvalho ALB, Souza MF, Shinizu HE, Senra IMVB, Oliveira KC. A gestão do SUS e as práticas de monitoramento e avaliação: possibilidades e desafios para a construção de uma agenda estratégica. Ciênc Saúde Coletiva [Internet]. 2012 abr [citado 2019 abr 30];17(4):901-11. Disponível em: Disponível em: http://www.scielo.br/pdf/csc/v17n4/v17n4a12.pdf . doi: 10.1590/S1413-81232012000400012
http://www.scielo.br/pdf/csc/v17n4/v17n4... The culture of an organization reflects the traditions, values and assumptions shared by its members, and establishes its rules of conduct.²⁵25. Bourgeois I, Cousins JB. Understanding dimensions of organizational evaluation capacity. Am J Eval [Internet]. 2013 May [cited 2019 Apr 30];34(3):299-319. Available from: Available from: https://journals.sagepub.com/doi/abs/10.1177/1098214013477235 . doi: 10.1177/1098214013477235
https://journals.sagepub.com/doi/abs/10.... Organizational leadership is also a premise underlying the construction of the M&E process, being considered as management support for implementing and sustaining evaluative capacity in organizations. Thus, if organizational leadership changes and the monitoring system and process are not institutionalized, their continuity is more fragile and threatened.²⁵25. Bourgeois I, Cousins JB. Understanding dimensions of organizational evaluation capacity. Am J Eval [Internet]. 2013 May [cited 2019 Apr 30];34(3):299-319. Available from: Available from: https://journals.sagepub.com/doi/abs/10.1177/1098214013477235 . doi: 10.1177/1098214013477235
https://journals.sagepub.com/doi/abs/10.... A Brazilian Ministry of Health publication released in 2005 signaled health service evaluation as being incipient and little incorporated into practices, too bureaucratic and distant from the institutional culture. Fragmented guidelines were mentioned as one of the limits to monitoring and evaluation activities, thus hindering the achievement of coordinated actions.²⁶26. Ministério da Saúde (BR). Secretaria Executiva. Departamento de Apoio à Descentralização. Nota técnica: considerações sobre a instituição de um processo de monitoramento e avaliação do SUS. Brasília: Ministério da Saúde; 2005.

It should be said that when the political situation was favorable, between 2012 and 2015, implementation of the system at SVS was successful, whereby its process dimension was considered to be fully implemented, while its structure dimension was partially implemented. Nevertheless, there are frequent structural difficulties for the implementation of computerized systems.

In 2014, the Brazilian Federal Court of Auditors released the results of a survey providing a maturity rating of the evaluation systems of Federal Direct Administration programs.²⁷27. Tribunal de Contas da União (BR). Diagnóstico do perfil e da maturidade dos sistemas de avaliação dos programas governamentais [Internet]. Brasília: Tribunal de Contas da União; 2014 [citado 2019 abr 30]. Disponível em: Disponível em: https://portal.tcu.gov.br/lumis/portal/file/fileDownload.jsp?inline=1&fileId=8A8182A14D92792C014D92800F763762
https://portal.tcu.gov.br/lumis/portal/f... This survey involved 2,062 managers from 27 ministries, but only 750 answered it (36.4%). Although the Ministry of Health’s rating was 73.1 - the highest of all the ministries -, the survey highlighted the need to adjust budgetary and financial resources, improve Information Technology infrastructure and reorganize the number of staff available at the ministry’s various offices in order to ensure adequate production of information.²⁷27. Tribunal de Contas da União (BR). Diagnóstico do perfil e da maturidade dos sistemas de avaliação dos programas governamentais [Internet]. Brasília: Tribunal de Contas da União; 2014 [citado 2019 abr 30]. Disponível em: Disponível em: https://portal.tcu.gov.br/lumis/portal/file/fileDownload.jsp?inline=1&fileId=8A8182A14D92792C014D92800F763762
https://portal.tcu.gov.br/lumis/portal/f...

According to another study, this time about the perception of Family Health teams, the Primary Health Care Information System (SIAB) was used only sporadically by teams owing to staff turnover. This structural problem in the area of Human Resources hindered the filling-in of forms, creating difficulties for analyzing, monitoring and evaluating information held on the system.²⁸28. Freitas FP, Pinto IC. Percepção da equipe de saúde da família sobre a utilização do sistema de informação da atenção básica-SIAB. Rev Latino-Am Enfermagem [Internet]. 2005 jul-ago [citado 2019 abr 30];13(4):547-54. Disponível em: Disponível em: http://www.scielo.br/pdf/rlae/v13n4/v13n4a13.pdf . doi: 10.1590/S0104-11692005000400013
http://www.scielo.br/pdf/rlae/v13n4/v13n...

The influence of structure on the implementation of information systems also appears in a study of the implementation of the Brazilian Live Birth Information System (SINASC) in Pernambuco state (PE) in 2010: SINASC/PE implementation was considered advanced at the central level, while at the level of inner state regional offices it was evaluated as insufficient. It is possible that these variations in implementation are related to insufficient structure of regional health offices and little knowledge of rules in force by system operators in the state.²⁹29. Pereira CCB, Vidal SA, Carvalho PI, Frias PG. Avaliação da implantação do Sistema de informações sobre nascidos vivos em Pernambuco. Rev Bras Saúde Mater Infant [Internet]. 2013 jan-mar [citado 2019 abr 30];13(1):39-49. Disponível em: Disponível em: http://www.scielo.br/pdf/rbsmi/v13n1/a05v13n1.pdf . doi: 10.1590/S1519-38292013000100005
http://www.scielo.br/pdf/rbsmi/v13n1/a05...

In yet another study worth mentioning, low use of the Public Health Budgets Information System (SIOPS) was attributed to two factors: (i) the precarious situation of connection to the internet; and (ii) lack of knowledge about how to use the system, evaluated based on data input regularity and the use of the system as a management tool.³⁰30. Silva KSB, Bezerra AFB, Sousa IMC, Gonçalves RF. Conhecimento e uso do Sistema de Informações sobre Orçamentos Públicos em Saúde (SIOPS) pelos gestores municipais, Pernambuco, Brasil. Cad Saúde Pública [Internet]. 2010 fev [citado 2019 abr 30];26(2):373-82. Disponível em: Disponível em: http://www.scielo.br/pdf/csp/v26n2/16.pdf . doi: 10.1590/S0102-311X2010000200016
http://www.scielo.br/pdf/csp/v26n2/16.pd...

Unlike the findings of these studies, in the case of the e-Car system, partially implemented structure did not hinder the process dimension, which was considered to be fully implemented, possibly because of efforts made by management.

Apart from monitoring PPA targets, the format used to create the e-Car system enabled the incorporation of complementary targets important for SVS as part of the monitoring process. The concern with building consistent and measurable outcome indicators, cited as a challenge by Santos,¹³13. Santos AR. Monitoramento e avaliação de programas no setor público: a experiência do PPA do Governo Federal no período 2000-2011 [monografia]. Brasília: Instituto Serzedello Corrêa (ISC); 2012. was considered in the programming for the period. All targets and strategic results monitored by e-Car were part of Brazilian Ministry of Health strategic planning and were related to the objectives agreed between health service managers.

Standing out among the limitations of this study is the management transition at the Brazilian Ministry of Health stemming from the President’s Office during the period analyzed. The impact of this fact was the reduction of the final number of participants, from 36 to 25 interviewees. Although not all SVS managers and technicians participated in this study, it allowed an understanding of the status of e-Car implementation at SVS based on knowledge of its weaknesses and strengths.

In view of the results found, we concluded that the e-Car system was implemented at the Health Surveillance Secretariat (SVS) in the period when 2012-2015 PPA was in force. Nonetheless, the system’s sustainability was considered fragile, requiring time, investment and the adoption of more mature arrangements capable of promoting the institutional culture change needed for the incorporation of monitoring and evaluation practices.

To improve both the Results Control, Monitoring and Evaluation System (e-Car) and the process of monitoring government outcomes, we recommend the following initiatives: (i) institutionalization of e-Car and the outcome monitoring process; (ii) continuing personnel training in planning, monitoring and evaluation; (iii) maintenance of the monitoring process in order to strengthen decision-making; (iv) improvement of the system’s structure, with a faster and more stable internet connection; and (v) retention of staff trained in e-Car development and technical support.

Acknowledgments

To the managers, coordinators and technicians of the Health Surveillance Secretariat, Ministry of Health (SVS/MS), who contributed to this study. To the Professional Master’s Program in Public Health, Sergio Arouca/Oswaldo Cruz Foundation National School of Public Health. To SVS/MS and the SUS Monitoring and Evaluation Department for the support provided.

References

Publication Dates

History