ABSTRACT

The multidimensionality of poverty was the central premise of the Brazilian Without Misery Plan (2011-2014), which aimed at eradicating misery in the country. The Oswaldo Cruz Foundation coordinated a health research induction program that promoted research aimed at producing knowledge to mitigate problems related to misery. The evaluability study was guided by the theoretical and methodological instrument of the Theory of Change, which allowed us to trace the path from the initial scenario to an expected scenario, based on the induction results, represented in the logical model of the program, in which the structure, processes, and results were also described. Through document analysis and interviews with stakeholders, it was possible to identify the objectives, expected results of the program, as well as the public benefited by it and by the evaluation of the program’s results. This study guided the planning of the assessment that followed, as well as the definition of the Canadian model, to measure the return on health research investment as a methodological strategy. The limitations were recognized, as well as the issues that favored the implementation and development of the induction program. It is further hoped that this study may contribute to evaluations and/or pre-evaluations of other similarly targeted programs.

KEYWORDS:
Poverty areas; Poverty; Brazil; Program evaluation

Introduction

Brazil is known as a challenging and complex country in terms of mitigating poverty. Together with the Middle East and Sub-Saharan Africa, Brazil remains among the three most unequal regions in the world, according to the 2018 World Inequality Report, which states that “income inequality has remained relatively stable, at extremely high levels”¹1 Alvaredo F, Chancel L, Piketty T, et al., coordenadores. World inequality report: english version. [Paris]: World Inequality Lab; 2018.⁽⁶⁾.

Despite the increase in economic development in the last five decades, when Brazil entered the group of the 10 largest economies in the world in absolute Gross Domestic Product (GDP), the country never distributed this wealth in order to face historical inequality. It is notable that the Gini index, a globally accepted indicator of inequality, decreased continuously between 2001 and 2014, and, after changes in economic policies, as the political crisis in Brazil worsened, it rose again²2 Neri MC. What was the impact of the crisis on poverty and income distribution? Rio de Janeiro: FGV; 2018..

The best period of inequality reduction in Brazil is explained by a set of macroeconomic measures based on policies to open jobs, continually increasing the net value of the minimum wage and income transfer, through the Bolsa Família Program (PBF). During this period, the issue of tackling extreme poverty was highlighted on the Brazilian national agenda. In fact, the PBF was the largest conditional cash transfer program in the world and allowed millions of Brazilians to get out of extreme poverty. However, the diagnosis made in 2010³3 Instituto Brasileiro de Geografia e Estatística. Universo preliminar do censo demográfico 2010. Rio de Janeiro: IBGE; [2010]. found that poverty remained significant, which meant about 16 million people living below the poverty threshold, in places where the state did not even reach.

To face this scenario, the Brazilian Without Misery Plan⁴4 Brasil. Decreto nº 7.492, de 02 de junho de 2011. Institui o Plano Brasil Sem Miséria. Diário Oficial da União. 3 Jun 2011. (PBSM) was launched in 2011. Through this plan, the federal government has set the goal of eradicating extreme poverty in 4 years, from 2011 to 2014. Under the coordination of the Ministério do Desenvolvimento Social e Combate à Fome, MDS (Ministry of Social Development and Fight against Hunger), as it was named and structured at that time, the promotion of rights was at the center of the narrative of the plan’s official policy⁵5 Paes-Sousa R. Plano Brasil Sem Miséria: incremento e mudança na política de proteção e promoção social no Brasil. Work. pap. 2013; Ago 113.. The PBSM supported its guidelines on the articulation of actions to guarantee income in addition to actions aimed at improving the living conditions of the extremely poor population, considering the multidimensionality of the situation of poverty⁶6 Paes-Sousa R, Vaitsman J. The zero hunger and Brazil without extreme poverty programs: a step forward in brazilian social protection policy. Ciênc. Saúde Colet. 2014; 19(11)4351-4360.^,77 Campello T, Mello J. O processo de formulação e os desafios do plano Brasil sem miséria: por um país rico e com oportunidades para todos. In: Ministério do Desenvolvimento Social e Combate à Fome (Brasil). O Brasil sem miséria. Brasília, DF: Ministério do Desenvolvimento Social; 2014..

Based on the integration and articulation of social policies, programs, and actions, the purpose of PBSM was to strengthen existing actions and link them to new programs, with the objective of including the poorest population in the opportunities generated by the Brazilian economic growth of that period. According to the MDS report⁸8 Brasil. Ministério do Desenvolvimento Social. Plano Brasil sem miséria: caderno de resultados (junho/2011 a dezembro/2014). Brasília, DF: Ministério do Desenvolvimento Social; 2015. , from the perspective of income and according to the standards of the period, the end of extreme poverty was reached still in 2013, with the inclusion of about 20 million Brazilians who lived below the poverty threshold and with the achievement of all goals foreseen in the PBSM.

In the field of health, the theme ‘neglected diseases, poverty eradication, and the Brazil Without Misery Plan’ stood out, linked to the axis of access to public services. This inclusion was the result of the articulation of the MDS and the Oswaldo Cruz Foundation (Fiocruz), with the final decision of the Presidency of the Republic. As Fiocruz is the largest non-university academic institution in the country and responsible for more than 25 doctoral courses in different areas of health, one of the main developments of this articulation was the implementation of a program to induce health research, from 2013 to 2018, through the cooperation among Fiocruz, MDS and the Coordination for the Improvement of Higher Education Personnel (Capes), an official body that coordinates and finances higher education at the doctoral level.

This induction program, called the Fiocruz-Capes-Brazil Without Misery Program, was one of Capes’ first induction programs, an unprecedented experience at Fiocruz and a desirable approximation between the academy and social policies. The main purpose of the program was to generate health knowledge to mitigate problems related to extreme poverty, through the implementation of scholarships for doctoral students and postdoctoral researchers. With the participation of 16 doctoral courses, after the development period, the products were presented in the form of theses, scientific articles, technical and educational productions.

This article presents the first results of the evaluation research of this induction program, also called an intervention, through the study of evaluability⁹9 Leviton LC, Collins CB, Laird BL, et al. Teaching evaluation using evaluability assessment. London: Evaluation; 1998. , which brought together a set of actions to plan the evaluation, considering the parties interested in its results, in order to maximize its usefulness and answer if the intervention was eligible for evaluation.

The results showed that it was possible to identify the purposes, structure, activities, and expected products that involved the intervention and whether these elements responded to the problem situation that was intended to be solved, as well as to represent them graphically in the logical model of the intervention. The political context in which the intervention was designed and implemented and the evaluation questions, which should be prioritized in the subsequent stages of the evaluation, are also the outcomes of this study.

Material and methods

This evaluability study adopted a descriptive-qualitative approach with the following steps: i) analysis of official documents to build the first version of the logical model and identification of stakeholders; ii) validation of the program’s logical model with stakeholders, through semi-structured interviews; and iii) formulation and selection of evaluation questions that supported the evaluation¹⁰10 Thurston WE, Ramaliu A. Evaluability assessment of survivors of a torture program: lessons learned. Can. J. Program Eval. 2005; (20):1-25.^,1111 Mendes MF. Avaliabilidade ou pré-avaliação de um programa. In: Samico I, Felisberto E, Figueiró AC, et al., organizadores. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010.. There is also the theoretical support of the ‘Theory of Change’¹²12 Anderson AA. The community builder's approach to theory of change: a practical guide to theory development. New York: Aspen Institute Roundtable on Community Change; 2006.^,1313 Arnold M. Theories of change in internacional development: communications, learning or ccountability? London: Justice and Security Research Programme; 2014. as a process by which ‘change pathways’ can be traced, that is, what was expected from the results and effects of the program when it was conceived.

The document analysis¹⁴14 Cutolo LR, Madeira KH. O trabalho em equipe na estratégia saúde da família: uma análise documental. Arq. Catarin. Med. 2010; 39(3):79-84.^,1515 Prates JC, Prates FC. Problematizando o uso da técnica de análise documental no serviço social e no direito. Soc. Debate. 2009; 15(2):111-125. gathered six official documents from the induction program. Namely: i) Technical note 1/2011/IOC-Fiocruz; ii) Decree No. 7,492, of 6/2/2011, which instituted the Brazilian Without Misery Plan; iii) Document approved by the Technical Teaching Chamber of Fiocruz - September/2011; iv) Cooperation Agreement No. 123/2011 - Fiocruz/Capes; v) Cooperation Agreement No. 128/2011 - Fiocruz/MDS; and vi) Strategic planning by the Ministry of Health (2011-2015). Approved by the Human Research Ethics Committee, CAEE: 62571016.5.0000.5248, opinion No. 1.950.248.

This analysis guided the first version of the logical model of the intervention, as it was a source of data collection considered adequate for this stage since it was aimed at subsidizing the description of the program to represent it through graphics¹⁶16 Bezerra LCA, Cazarin G, Alves CKA. Modelagem de programas: da teoria à operacionalização. In: Samico I, Felisberto E, Figueiró AC, et al., organizadores. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010.^,1717 Cassiolato MM, Gueresi S. Como elaborar modelo lógico: roteiro para formular programas e organizar avaliação. Brasília, DF: Ipea; 2010..

This initial version was presented at the end of the six interviews with stakeholders from the three institutions involved, to obtain contributions and validation. The interviews took place from October 2017 to February 2018, followed a semi-structured script, and allowed for the characterization of the political context in which the program was designed, as well as the purposes, the expected results, the public benefited by the program and by the evaluation, and the questions that should be answered in the assessment. All participants signed the Free and Informed Consent Form and agreed to the audio recording, which was transcribed and analyzed through an interpretative analysis¹⁸18 Vianna IO. Metodologia do trabalho científico: um enfoque didático da produção científica. São Paulo: EPU; 2001..

Stakeholders (SH) were selected due to the key positions they occupied during the program’s creation period, in a representative situation, with autonomy and decision-making power in the three institutions involved.

From MDS, the strategic formulator of the PBSM, three major stakeholders were invited and agreed to participate in the study (SH1,2,3); the director of scholarship programs at that time at Capes, the agency that made the intervention possible through grants (SH4); the representative of the postgraduate general coordination and the then institutional development coordinator were interviewed from Fiocruz (research institution) (SH5,6).

Results and discussion

The first result of the evaluability study was the elaboration and validation of the logical model of the intervention by the stakeholders, shown in figure 1, in which it was possible to identify and describe the following issues: i) problematic situation that generated the need to create the program; ii) the objective of the program; iii) the target audience of the program; iv) the program structure and activities, and v) the expected results.

It is possible to observe in the logical model that, although the intervention involved three institutions, the responsibility for the structures and processes was centralized at Fiocruz, which corroborates the centrality of the institution identified in the number of official documents on the intervention, objects of document analysis.

The second result of this study was the characterization of the political context external and internal to Fiocruz and the description, from the perspective of the stakeholders, of the objectives of the program and the expected results, which made it possible to trace the path of the change intended with this intervention.

Characterization of the political context

The political context, in general, is not described in official program documents, and this was the case with the intervention under analysis. However, to plan the evaluation, it is desirable to know it since the success or not of an intervention can be (co-)determined by the context in which it is inserted.

In this sense, the interview participants were asked to describe the political context in which the Fiocruz-Capes-Brazil Without Misery induction program was designed.

It was possible to identify convergence and complementarity in the reports of the key actors, in the sense of a very favorable context in the country for the implementation of PBSM and its actions. The reports also revealed the managers’ attention/concern regarding the limitation of the social programs in force until 2010 and the need for action focused on the extremely poor. According to the main stakeholder of the MDS:

In addition to the contribution of Fiocruz, the renowned academic institution, partners were sought to give legitimacy to PBSM. Thus, two aspects were considered by the MDS managers: a managerial one, the ability to present results and produce health knowledge, with a focus on neglected diseases; and another political one, the symbolic potential of the institution.

Although the external context was favorable and the proposal was converged with Fiocruz’s agenda and with its own mission in the face of social inequalities, there was resistance on behalf of the academy, as it was guided by the political agenda, as highlighted by the then coordinator of management and institutional development:

This finding revealed that there was no consensus at Fiocruz on cooperation and guided an aspect in the next stages of the evaluation, which was to identify the reason for the non-adherence and/or low adherence of graduate students to the induction program, as well as the concentration of research in a single graduate course: Tropical Medicine, which had 40% of the induction scholarships.

It is worth mentioning that the favorable internal context in the design of the program, even if not consensual, was not maintained during its implementation, since the foundation went through a moment of internal division, which had direct effects on the evaluated intervention. Among the effects, the restructuring of the management committee of the induction program stands out, with the departure of key actors since its conception. In the light of the theory of change and the path that was sought to be represented in the logical model, this finding was essential for the definition of management indicators in the evaluation of this intervention.

From the perspective of Capes, which was experiencing a moment of expansion, with an increase in the budget forecast to finance research, through graduate scholarships, the director of the scholarship program at the time started promoting thematic inductions, which came to meet Fiocruz’s proposal to induce health research to tackle the diseases of poverty:

Although Capes was key to this intervention, be it for the resources allocated to scholarships or for the enthusiasm to promote health research focused on the public of the Brazil Without Misery Plan, the institution did not follow the process of implementation and results of the program. One of the reasons to be highlighted about this non-involvement is the historical problem of discontinuity of Brazilian public actions that reflect the characteristics and dynamics of the decision-makers of the moment, with a sequence of interruptions and ruptures that directly interfere in the achievement of program results¹⁹19 Brousselle A, Champagne F, Contandriopoulos A, et al., organizadores. Avaliação: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011..

Objectives of the induction program

When asked about what was expected in 2011 to induce health research and what results were projected, the interviewees highlighted the applicability of knowledge, that is, that in some way the knowledge generated would return directly or indirectly to the poorest and to society in general.

According to representatives of Fiocruz, the greatest contribution of the induction program to PBSM was systematized knowledge, and one of the main means was the Technical Notes, result of research, to guide managers and decision makers in the formulation of public policies.

Stakeholders 1 and 4, from MDS and Capes, respectively, shared the idea that the academy had to be concerned with the country’s problems:

The articulation of public policies to improve of public policies themselves, with a consequent improvement in life for the population, was also an expected result of the intervention.

Regarding the audacious objectives of the stakeholders, it is important to highlight that the induction program was experimental, and, in the scope of training researchers, the objectives were, to some extent, much more a ‘will’ than something possibly accomplished. This finding was particularly important when tracing the path of change since the translation of knowledge and the improvement of public policies with direct effects on the population are historical challenges to the country.

The public benefited by the induction program

The public benefited by the induction program, according to the interviewees, is remarkably diverse, including researchers, political actors, managers, and decision makers, based on the improvement and/or implementation of public policies for the general population, through the application of the knowledge generated.

The breadth and diversity of the public benefited by the induction program, identified during the construction of the path of change, reflect the immense gaps and needs of one of the most unequal countries in the world. They also reflect the false dichotomy between academic knowledge and the idea of ‘utility’, revealed during the interviews.

The public benefited by the evaluation of the induction program

The public benefited by the program evaluation, according to the interviewees, are especially the institutions involved, as well as the public policymakers and decision-makers since the results can reorganize the formulation of programs with a similar scope.

The benefits of evaluating the program, from the perspective of the stakeholders, converge to the learning perspective of the authors when they were committed to evaluate and learn from this experience, to verify whether this path was promising or not. The expectation is that, based on evaluation research, which begins with this evaluability study, knowledge will be produced for new actions with similar characteristics.

Evaluation questions

This study started and was conducted under the premise that the ongoing evaluation would strongly consider the perspective of those interested in the program. Thus, the evaluation questions raised by the representatives of the three institutions were strategic for this study of evaluability and were decisive for the decision of the methodological strategy of the evaluation.

The questions are organized by institution, on purpose, to allow that different institutional perspectives are viewed.

Identifying the evaluation questions of the stakeholders was an important step in planning the evaluation. This survey allowed us to understand what was important to be answered for those who designed the program and also allowed us to identify that the moment of the interview was, for some, the first moment when they thought about this evaluation, a fact that corroborates the challenge of institutionalizing the evaluation in public institutions in the country²⁰20 Carvalho AL, Shimizu HE. A institucionalização das práticas de monitoramento e avaliação: desafios e perspectivas na visão dos gestores estaduais do Sistema Único de Saúde (SUS). Interface. 2017; 21(60):23-33..

The last step was to select which of these questions could be answered, considering issues such as time to carry out the research, feasibility of collecting and analyzing the data, and methodological possibilities. The selected questions were those involving academic and technical products, including qualitative analysis. Questions that had a broader proposal on institutional changes and the effects or consequences of the knowledge produced did not fall within the scope of the assessment, although both merit and legitimacy are recognized.

Final considerations

The purposes of the evaluability study were achieved based on the following results: i) the logical model of the program developed and validated by the interested parties; ii) the political context described as favorable to the implementation and development of the program; and iii) the evaluation questions, which were mostly identified and recognized as possible to be answered.

This study was also crucial to choose the methodological strategy of the evaluation, which was the model developed by the Canadian Academy of Health Sciences, available in the document ‘Making an Impact: A preferred Framework and Indicators to Measure Returns on Investment in Health Research’²¹21 Canadian Academy of Health Sciences. Making an impact: a preferred framework and indicators to measure returns on investment in health research. Canadian Academy of Health Sciences: Canada; 2009. since the great guidance for the evaluation was to answer whether the public and/or the society benefited from the public investment in the production of knowledge.

It is worth recognizing that the discontinuity of public policies in Brazil is the main limitation of this study and a posteriori evaluation, especially from the perspective of learning adopted for the evaluation that followed this study, that is, that based on the results and recommendations of this evaluation, new programs to induce health research could be promoted and improved.

Also, considering that an important part of the results of the induction program was suitable, based on the knowledge produced, to subsidize the (re)organization of current public policies and/or to guide the formulation of new policies and that this depended on political articulations with the government, which was interrupted in 2016, the linearity expected and presented in the logical model did not happen and it will hardly happen in the current government, which has a political project diametrically opposed to the one that conceived the induction program.

However, there were more than 15 million Brazilians living, in 2018, below the extreme poverty threshold, according to the Brazilian Institute of Geography and Statistics (IBGE)²²22 Instituto Brasileiro de Geografia e Estatística. Síntese de indicadores sociais: uma análise das condições de vida da população brasileira. Rio de Janeiro: IBGE; 2018.. If new studies conducted by the World Bank are considered, which take into account the multiple dimensions of well-being, we reach the number of 50 million Brazilians living below the poverty threshold. These data demonstrate the relevance of the object of the study and the recognition that there is much to be done to eradicate poverty in the country, and both academia and the health field play a leading role in the care of millions of Brazilians who remain in the poverty-disease-poverty cycle.

References

Publication Dates

History