ABSTRACT

This essay is based on the epistemological concerns of health evaluators in their search to understand the meanings of their professional practice and their theoretical formulations. Such concerns emerge from the strangeness accumulated over years of work in the field and a growing questioning, given the naturalization of the relationship between the concepts used in health evaluation and their implications for the maintenance of power relations. In this essay, we present an analytical proposal for understanding the concepts mobilized by health evaluation supported by Michel Pêcheux’s Discourse Analysis, Louis Althusser’s Materialist Philosophy, and Georges Canguilhem’s Historical Epistemology. We propose a materialist reading approach for the analysis of the standardized discourse about the development of evaluation, seeking to contribute to overcoming idealistic readings around approaches and standards that succeed one another and are stabilized as truths, with a view to advancing towards a historical understanding. Understanding the formation of the conceptual discourse around the social practice of health evaluation in Brazil-and, thus, understanding the ideological functioning of its enunciation-can aid us to deepen the analysis of the meanings that operate in the practice and formulation of this social space.

KEYWORDS:
Health evaluation; Public health; Epistemology; Discourse; History.

Introduction

This study is based on questions and concerns of the group of researchers and evaluators of the Laboratory of Studies and Evaluation in Health (AVAL-LAB) of the Federal University of São Paulo (UNIFESP), over years of work on the health evaluation subject. Throughout this trajectory, several efforts to analyze health evaluation itself as an area of formulations and practices have been employed from various perspectives and practices. We can describe the understanding of health evaluation as a tool for institutional change via the constructivist approach^{11 Furtado JP. Um método construtivista para a avaliação em saúde. Ciênc saúde coletiva. 2001;6(1):165-181. DOI: https://doi.org/10.1590/S1413-81232001000100014
https://doi.org/10.1590/S1413-8123200100...} , the sociological approach to social and power dynamics involved in the emergence of health evaluation in Brazil^{22 Furtado JP, Vieira-da-Silva LM. A avaliação de programas e serviços de saúde no Brasil enquanto espaço de saberes e práticas. Cad Saúde Pública. 2014;30(12):2643-2655. DOI: https://doi.org/10.1590/0102-311X00187113
https://doi.org/10.1590/0102-311X0018711... ,} and the analysis of the philosophical and epistemological foundations that underpin evaluation practices^{33 Gasparini MFV. Bases filosóficas e epistemológicas da avaliação: caminhos a serem trilhados. Rev Aval [Internet]. 2020 [acesso em 2024 jan 10];3(17):12-31. Disponível em: http://periodicos.ufc.br/aval/article/view/60287/161879
http://periodicos.ufc.br/aval/article/vi...} . It is also an area in which we are involved as practitioners, conducting evaluations of programs and services within the Brazilian Unified Health System (SUS)<sup>44 Gasparini MFV, Furtado JP. Longitudinalidade e integralidade no Programa Mais Médicos: um estudo avaliativo. Saúde debate. 2019;43(120):30-42. DOI: https://doi.org/10.1590/0103-1104201912002.
https://doi.org/10.1590/0103-11042019120...

5 Furtado JP. Avaliação da situação atual dos Serviços Residenciais Terapêuticos no SUS. Ciênc saúde coletiva. 2006;11(3):785-795. DOI: https://doi.org/10.1590/S1413-81232006000300026
https://doi.org/10.1590/S1413-8123200600...

6 Borysow IC, Furtado JP. Acesso, equidade e coesão social: avaliação de estratégias intersetoriais para a população em situação de rua. Rev Esc Enferm USP. 2014;48(6):1069-1076. DOI: https://doi.org/10.1590/S0080-623420140000700015
https://doi.org/10.1590/S0080-6234201400...

7 Iacabo P, Furtado JP. Núcleos de Apoio à Saúde da Família: análises estratégica e lógica. Saúde debate. 2020;44(126):666-677. DOI: https://doi.org/10.1590/0103-1104202012606
https://doi.org/10.1590/0103-11042020126... -88 Carvalho AP, Furtado JP. Moradia assistida para pessoas em situação de rua no contexto da política de drogas brasileira: avaliação de implantação. Physis. 2021;31(1):e310116. DOI: https://doi.org/10.1590/S0103-73312021310116
https://doi.org/10.1590/S0103-7331202131...</sup> .

Specifically, this study arises from a reflection over the hegemonic positions of health evaluation regarding its practices and mobilized concepts, whose meanings are considered transparent, presenting themselves as truths via statements that produce recognition with an effect of evidence and, therefore, ideological^{99 Althusser L. Sobre o trabalho teórico: dificuldades e recursos. In: Barison T, organizador. Teoria marxista e análise concreta: textos escolhidos de Louis Althusser e Étienne Balibar. São Paulo: Expressão Popular; 2009. p. 83-114.}. Such positions understand the development of evaluation as a succession of methodological approaches, preferred by evaluators, timelines of authors, or models that follow a linear logic of surpassing previous approaches, generating a profusion of classifications with theoretical and methodological claims for evaluation practices. Moreover, evaluation has been characterized by the imposition of standards and criteria that regulate what are considered more or less adequate evaluations, standardizing behaviors for both evaluations and meta-evaluations, or evaluations of evaluations.

This essay aims to introduce a materialist reading approach, that is, a form of analysis that considers knowledge as a historical process of production, proposing that the history and social function of evaluation be addressed via the dialectical and material relations established in its production processes, based on structural determinations. This approach is supported by Michel Pêcheux’s Discourse Analysis theory and Georges Canguilhem’s Historical Epistemology, along with the formulations of Materialist Philosophy proposed by the philosopher Louis Althusser. The combination of these theoretical approaches seems to offer pathways for analyzing the meanings of health evaluation toward a critical reflexivity that ultimately serves the purpose of questioning material relations whose meanings are ideologically constructed and produce power asymmetries, based on the reproduction of concepts that align with a particular hegemonic model of society.

The analysis employed in this essay is based on the association between the issue, the object, and the method used. This articulation has presented itself to us as a process of analytical experimentation that simultaneously seeks to apply a theoretical lens to a specific object and generates its questions throughout the investigation. From the perspective of Discourse Analysis, it is understood that research questions, or those of discourse, emerge in action during the analytical process. For this reason, Discourse Analysis is not considered a method since new paths, relationships, and contradictions emerge without necessarily having a specific starting point or a predetermined endpoint. This is equivalent to state that the understanding of the issue to be analyzed in this study arises precisely from the theoretical contribution applied to our object, seeking to escape the empiricist trap^{99 Althusser L. Sobre o trabalho teórico: dificuldades e recursos. In: Barison T, organizador. Teoria marxista e análise concreta: textos escolhidos de Louis Althusser e Étienne Balibar. São Paulo: Expressão Popular; 2009. p. 83-114.}. In other words, the issue of normative inventories for understanding the meanings of health evaluation emerges only as a result of the theoretical lens we apply to comprehend our own field of work and elaborations: Discourse Analysis, Historical Epistemology, and Materialist Philosophy. As a theoretical essay, this study aims to challenge and expand the analytical boundaries of our object via a reflective and interpretative approach based on an explicit theoretical framework, whose premises and analyses need to be coherent, avoiding subjectivity and reliance on anecdotal evidence^{1010 Meneghetti FK. O que é um Ensaio-Teórico? RAC. 2011;15(2):320-332. DOI: https://doi.org/10.1590/S1415-65552011000200010
https://doi.org/10.1590/S1415-6555201100...} .

Starting from the Discourse Analysis approach, we created an archive-understood as a set of meaning relations based on formal supports, such as textual types, rather than the content of the documents themselves^{1111 Barbosa Filho FR. Ler o arquivo em análise de discurso: observações sobre o alienismo brasileiro. Cad Est Ling. 2022;64:e022007. DOI: https://doi.org/10.20396/cel.v64i00.8664658
https://doi.org/10.20396/cel.v64i00.8664... ,1212 Pêcheux M. Ler o arquivo hoje. In: Orlandi EP, organizador. Gestos de leitura: da história no discurso. Tradução: Bethânia S. C. Mariani. Campinas: Editora da Unicamp; 1994. p. 55-66.}-comprising articles, books, theses, practical guides, and professional evaluation guidelines. This collection is employed as a reference in this essay, constituting the view that has been shown to be hegemonic to explain the development of program evaluation and the orientation of the practice considered adequate. From this, we established a relationship of tension between this discursivity and our theoretical lens of Discourse Analysis, Historical Epistemology, and the contributions of Materialist Philosophy, proposing a materialist reading approach to guide future interpretations of the conceptual discourse of evaluation, and more specifically, of health evaluation.

For a materialist reading approach to analyze evaluation concepts: articulation between Discourse Analysis and Historical Epistemology

As stated by Sérgio Arouca in ‘O dilema preventivista’ [The prevention dilemma]^{1313 Arouca S. O dilema preventivista: contribuição para a compreensão e crítica da Medicina Preventiva. São Paulo, Rio de Janeiro: Ed. Unesp, Ed. Fiocruz; 2003. DOI: https://doi.org/10.7476/9788575416105
https://doi.org/10.7476/9788575416105...} , a work in which the author is anchored in Canguilhem to produce one of the most important theoretical-conceptual bases in the field of Public Health, the contribution of a Historical Epistemology of the sciences operates as a device capable of delimiting the ideological in social practices, thus favoring a theoretical practice that can offer new analytical instruments for health practices^{1414 Ayres JRCM. Georges Canguilhem e a construção do campo da saúde coletiva brasileira. Intell Rev Hist Intelec. 2016;2(1):139-155. DOI: https://doi.org/10.11606/issn.2447-9020.intelligere.2016.115732
https://doi.org/10.11606/issn.2447-9020....} , as is the case with the practice of evaluation.

In this way, we understand, from Historical Epistemology, that the history of a given scientific discipline (and here we assume evaluation as a scientific discipline or a knowledge with scientific pretensions)^{1515 Narzetti C. Para uma história epistemológica do conceito de formação discursiva. Ling (Dis)curso. 2018;18(3):647-663. DOI: https://doi.org/10.1590/1982-4017-180311-12917
https://doi.org/10.1590/1982-4017-180311...} is given by understanding the conceptual context and the objectives around which its effects of meaning are composed. In other words, it does not seem enough to us, in a study that seeks to understand the meanings of the discourse of evaluation, to start from a broad and generic conceptualization, such as the one that understands evaluation as a systematic practice that uses scientific approaches to determine merit and value via the judgment of a social intervention. This widely accepted and replicated concept, found in various references on the subject, seems to raise more questions-or even cause erasures-than clarify assumptions.

To consider an object of investigation as discourse means understanding that the material reality of a given object, within a specific historical context, does not present itself as an explicitly visible phenomenon readily available to the investigator’s gaze^{1212 Pêcheux M. Ler o arquivo hoje. In: Orlandi EP, organizador. Gestos de leitura: da história no discurso. Tradução: Bethânia S. C. Mariani. Campinas: Editora da Unicamp; 1994. p. 55-66.}. To approach the object of investigation by its discursivity-in our case, its textual discursivity-means proposing a specific way of reading, seeing, and listening, and, more than that, a specific way of producing knowledge based on the statements that constitute meaning relations.

Therefore, it is not about collecting data presented in reality such as ripe fruit in a field (as the word ‘collection,’ taken as a metaphor, might suggest), which would imply an empiricist epistemological stance that separates the knowing subject from the object to be known^{1616 Althusser L. Iniciação à filosofia para os não-filósofos. São Paulo: WMF Martins Fontes; 2019.}. Our stance in this investigation-or our reading approach regarding the effects of meaning in the discourse we aim to investigate-assumes that, in addressing discourse, we are immersed in a broader production process ultimately determined by a specific productive structure within the class society and its conflicts^{1717 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.}, and overdetermined by the dialectical relations established within social class struggles^{1818 Althusser L. Por Marx. Campinas, SP: Unicamp, 2015.}.

The use of Historical Epistemology guides us to approach scientific statements based on the concepts that underpin their practices, understanding them as ‘cells of knowledge’-as Bachelard defined (cellules de savoir)^{1919 Peña-Guzmán DM. French epistemology: discourse, concepts, and the norms of rationality. Stud Hist Philos Sci. 2020;79:68-76. DOI: https://doi.org/10.1016/j.shpsa.2019.01.006
https://doi.org/10.1016/j.shpsa.2019.01....} -which form the link between universal statements (or theories) and historical materiality. This connection enables scientists to formulate, designate, and constitute the objects that make up reality. Through a philosophy of science that contrasts with Epistemology of Knowledge, Historical Epistemology treats scientific discourse-understood as concepts in action, governed by rules-as its object or, more appropriately, the history of this discourse and, consequently, its historicity^{2020 Narzetti C. Para uma história epistemológica do conceito de formação discursiva. Ling (Dis)curso. 2018;18(3):647-663. DOI: https://doi.org/10.1590/1982-4017-180311-1291
https://doi.org/10.1590/1982-4017-180311...} . In particular, we endorse the assumptions about the history of science proposed by Canguilhem^{2121 Canguilhem G. Estudos de história e filosofia das ciências: concernentes aos vivos e à vida. Rio de Janeiro: Forense Universitária; 2012.}, for whom there is no science without concepts, and concepts are historically determined. In other words, a concept does not exist forever in an unchanging form but rather emerges from specific conditions and is reformulated, corrected, refined, expanded, and/or abandoned. We might even say that concepts are historically contested. Moreover, concepts are embedded in relationships with other concepts that provide them with meaning, so the researcher needs to reconstruct the synthesis in which the concept is embedded^{2020 Narzetti C. Para uma história epistemológica do conceito de formação discursiva. Ling (Dis)curso. 2018;18(3):647-663. DOI: https://doi.org/10.1590/1982-4017-180311-1291
https://doi.org/10.1590/1982-4017-180311...} .

The concept must also be understood by its association with an issue, as defining a concept is also about defining an issue. It is in the relationship between issue and concept that the condition for seeking scientific validity is found. Therefore, the issues that a concept is meant to address are what best shape its meanings^{2020 Narzetti C. Para uma história epistemológica do conceito de formação discursiva. Ling (Dis)curso. 2018;18(3):647-663. DOI: https://doi.org/10.1590/1982-4017-180311-1291
https://doi.org/10.1590/1982-4017-180311...} .

Considering that concepts designate, constitute, formulate, and define crucial issues in a field, making some problems addressable by thought and others even unthinkable^{2222 Canguilhem G. O normal e o patológico. Rio de Janeiro: Forense Universitária; 2020.}, and considering the plurality characterizing practices in health evaluation, we recognize that addressing concepts and their privileged function in shaping a scientific area is crucial to enable the understanding of the discursive determinations that define the conceptual issues of health evaluation. Our approach to reading and analyzing the object allows for a critical understanding of the relationship between evaluation practice and its specific approaches, which we will detail here.

In the formulations of traditional program evaluation-a term used to refer to approaches, especially from the U.S. context throughout the 20th century, that laid the foundations for health evaluation within Brazilian public health^{22 Furtado JP, Vieira-da-Silva LM. A avaliação de programas e serviços de saúde no Brasil enquanto espaço de saberes e práticas. Cad Saúde Pública. 2014;30(12):2643-2655. DOI: https://doi.org/10.1590/0102-311X00187113
https://doi.org/10.1590/0102-311X0018711...} -the generic practice of evaluation acquires its meanings as various practical approaches and related issues emerge, being historically determined, and always establishing a dependence of meaning on what best defines the focus, scope, or object of the evaluation. Such a relationship between specific approaches and the underlying problems of their emergence does not appear explicitly in technical, scientific, and normative statements, being up to the analyst to reconstruct these contexts from the statements that constitute their meanings. In other words, the approaches proposed for evaluation and their normative aspects indicate the issues that the concepts associated with evaluation were mobilized to address, rather than the other way around. This functioning is not explicit but rather obscured in the discourse that provides linearity and objectivity to the development of evaluation around models and norms, rather than the functioning concepts.

What we propose in this essay is to adopt a reading position that questions the given relationship between subject and object^{1717 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.}, or between the so-called evaluation theorists^{2323 Alkin MC, editor. Evaluation roots: tracing theorists’ view and influences. Thousand Oaks: Sage; 2004. DOI: http://doi.org/10.1177/1098214006287988
http://doi.org/10.1177/1098214006287988... ,2424 Shadish WR, Cook TD, Leviton LC. Foundations of program evaluation: theories of practice. Newbury Park: Sage Publications; 1995.} and the systematized conceptions of practiced approaches and established norms, thus implying a materialist reading approach. By questioning the object of this discourse-evaluation approaches and norms-and their development from a linear and progressive accumulation of productions, we understand the association between discourse and object, rather than between subject and object^{1717 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.}.

At this point, it is important to recognize the intellectual challenge of adopting a materialist reading approach, as the subject who analyzes inherently maintains a relationship with their object. In this case, the discourse analyst is also a program evaluator-an aspect that applies to both the authors of the texts in our archive and the authors of this essay. This factor might have been insufficiently considered in discussions about the development of evaluation, which is understandable since the idealist investigative stance often operates spontaneously within the subject-form, in which the relationship between the subjects of knowledge and the object is assumed to be natural. Such functioning leads to the empiricist illusion that subjects are products of their own conceptions. Thus, evaluation approaches would be seen as conceptions originating from subjects, rather than being historically determined by specific discursive formations^{1717 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.}.

The readings on evaluation analyzed by us, which compose the assembly of our archive^{2626 Orlandi EP, organizador. Gestos de leitura: da história no discurso. Tradução: Bethânia S. C. Mariani. Campinas: Editora da Unicamp; 1994.}, seem to focus on the definition of objects already conceived in the framework of their own conceptions and explanatory theories, in an endogenous, idealistic movement. In this regard, it was in Pêcheux that we found an explanation for the functioning of ideology, particularly in the reproduction of the so-called ‘subject-effect,’ which aids in understanding the issue surrounding inventories and the idealist illusion of the subject as the source of knowledge. This is illustrated in the following excerpt from ‘Language, semantics, and ideology’^{2525 Pêcheux M. Semântica e discurso: uma crítica à afirmação do óbvio. Tradução Eni Pulcinelli Orlandi. 5. ed. Campinas, SP: Editora da Unicamp; 2014.(149-150)}:

However, it is necessary, at this point, to clarify what we are calling a materialist reading stance about the approaches to program evaluation, the central focus of this essay. In his introductory notes to the book ‘Reading Capital’^{1717 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.}, Althusser focused on explaining to the reader how he would engage with Marx’s magnum opus, that is, what kind of reading he intended to undertake. Althusser clarified that his approach to reading would be through the lens of Materialist Philosophy, employing the same ‘poison’ that Marx applied to his object in the mentioned work. Thus, unlike a reading from the perspective of Economics, for example, which would seek, in ‘Capital’, to analyze and associate internal elements of the discipline itself, or even from the perspective of History-whose analysis would lead to the establishment of historical relations and the dynamics of objects, understanding the internal logic of History as a field of knowledge-a reading from Philosophy challenges, from the outset, the supposedly transparent relationship between the subject and the object of analysis. This initial delimitation around the elucidation of the association between subject and object stems from a rejection of the naturalization that subject and object are intertwined, which would be equivalent to saying that subjects themselves are the original sources of their objects of knowledge, or that subjects produce their own objects of knowledge^{1717 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.}.

For Pêcheux, a reading gesture represents a type of approach to reading, the way in which a certain file will be interpreted (inserted, in our case, in the textual discourse). In the text ‘Ler o arquivo hoje’ [Reading the archive today]^{1212 Pêcheux M. Ler o arquivo hoje. In: Orlandi EP, organizador. Gestos de leitura: da história no discurso. Tradução: Bethânia S. C. Mariani. Campinas: Editora da Unicamp; 1994. p. 55-66.}, it is said that Pêcheux presents a particular type of archival reading, starting with the identification of two traditional cultures that define specific reading practices throughout history. The first type is concerned with a reading not involved in producing original knowledge, a classificatory and bureaucratic reading considered objective, exemplified by figures such as the scribe or the copyist. This reading approach is very familiar to us, as it represents how scientific discourse shows, in a hegemonic manner, standardized reading approaches. This is evident, for example, in the various structured techniques for delineating scientific reading, such as systematic review protocols, bibliometrics, and variations arising from content analysis (inscribed in the logical-mathematical space). Here, there is no room for a slip of the tongue, for its meanings, or for ambiguous relations, as the aim is precisely to control this overflow of meanings that escape the ‘objective’ tools defined for reading. Not by chance, this model has been seen as a limitation to the theoretical development of science, in a reasoning that prioritizes the descriptive emphasis on research objects, in which objectivism and empiricism overshadow the role of theory in the construction of original knowledge.

Pêcheux also presents a second reading gesture, the literary type, which starts from the author’s individuality and establishes a specific way of reading (‘singular and individual’)^{1212 Pêcheux M. Ler o arquivo hoje. In: Orlandi EP, organizador. Gestos de leitura: da história no discurso. Tradução: Bethânia S. C. Mariani. Campinas: Editora da Unicamp; 1994. p. 55-66.}. This reading tradition includes historians, philosophers, and literary scholars, whose debates and tensions do not exceed disciplinary boundaries, maintaining an implicit reading approach. Here, both reading approaches and even the tensions between them are based on the premise that the utterances have a given meaning, with transparent meanings. If the first approach to reading supported scientific discourse and its own interests, the second normatively defines the meanings of reading through the lens of each specialist, hermetic to contextual pressures and demands. For Pêcheux, a reading of discourse does not assume that meaning is self-evident in language, but rather that meanings are constructed via the various possible associations between discourses, varying according to the relationship each subject of the discourse establishes^{1212 Pêcheux M. Ler o arquivo hoje. In: Orlandi EP, organizador. Gestos de leitura: da história no discurso. Tradução: Bethânia S. C. Mariani. Campinas: Editora da Unicamp; 1994. p. 55-66.}.

The French author, whose theoretical framework supports this study, challenges and questions the isolated position of each type of reading and its subjects (literary scholars and scientists) in the face of a situation that threatens the very concept of memory and increasingly imposes the normalization of meanings^{1212 Pêcheux M. Ler o arquivo hoje. In: Orlandi EP, organizador. Gestos de leitura: da história no discurso. Tradução: Bethânia S. C. Mariani. Campinas: Editora da Unicamp; 1994. p. 55-66.}. It may serve to inform our own challenge to scientists and literary scholars in their evaluation of both traditional reading practices.

On one hand, reading practices increasingly seek to standardize and linearize the development of the field, both in its methodological approaches and in the standardization of terms and practices, in the pursuit of scientification and the professionalization of evaluation. However, as Patton points out, this association invokes the ‘fight against the darkness’ that threatens knowledge^{2828 Patton MQ. Evaluation Science. Am J Eval. 2018;39(2):183-200. DOI: https://doi.org/10.1177/1098214018763121
https://doi.org/10.1177/1098214018763121...} . Such a reading practice-and meaning production-seems to embed an attempt to contribute to the social practice of evaluation with ‘scientifically based’ frameworks, supported by theoretical contributions that define its procedures. However, it must be said that this normative objective often does not hold in practice, as the act of evaluation does not always follow established theoretical-methodological frameworks that would provide it with immediate meaning^{2929 Christie CA. What guides evaluation? A study of how evaluation practice maps onto evaluation theory. New Dir Eval [Internet]. 2003 [acesso em 2024 jan 10];(97):7-35. Disponível em: https://wmich.edu/sites/default/files/attachments/u58/2015/What_Guides_Evaluation.pdf
https://wmich.edu/sites/default/files/at...} .

The practice of program evaluation in Brazil is also a professional practice inserted in its own market of services, which has sought to establish normative parameters to regulate its performance by different interest groups. Thus, in addition to the normative discourse around the approaches in evaluation, whose interface we can affirm has taken place within the academic field, there are also disputes in the field of professional performance and corporate interests^{3030 Furtado JP. Por uma meta-avaliação sem metafísica. Rev Bras Aval. 2022;11(2):e112022. DOI: http://dx.doi.org/10.4322/rbaval202211020
http://dx.doi.org/10.4322/rbaval20221102...} .

Moreover, on the other hand, it would be appropriate to provoke a reading practice that confines the interpretation of the evaluation field to itself, via its intellectuals or theorists, isolating its object from the historical context that shapes and overdetermines the production of its meanings. In this type of reading, the historical context usually appears as a supporting role, such as a scenario in which furniture illustrates narratives of discoveries, insights, and individual trajectories endowed with special pioneering attributes. It is, as Pêcheux says, in the middle of these two cultures of reading, these two reading approaches and their specific ways of manifesting themselves within the field of evaluation, that we seek to position ourselves.

With this, we want to defend a reading position that denaturalizes the relationship between the subject and the object that produces a certain knowledge of its own field, understanding such relationship as determinations of a discursive process.

What does this stance regarding our (materialist) reading imply for how we will engage with our object of investigation, namely the evaluation and its concepts? In the first place, we are delimiting what Althusser^{1717 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.} calls a guilty reading, that is, a reading that admits the contestation of the statement and its form, given as knowledge. By questioning the statements that aim to construct a historicity of evaluation from an empiricist subject-object relationship, we assume that evaluation is a type of social practice. This implies understanding that, ultimately, it is the historically determined relations of production that confer meaning to evaluation, rather than the individual practitioners alone, as individuals are addressed by ideology as subjects in a materialist reading<sup>1616 Althusser L. Iniciação à filosofia para os não-filósofos. São Paulo: WMF Martins Fontes; 2019.

17 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.-1818 Althusser L. Por Marx. Campinas, SP: Unicamp, 2015.</sup>. Thus, more than a succession of approaches proposed and discoveries developed throughout history by ‘theorists’ and practitioners of evaluation, what seems more important to us is to understand the historical context and its determinations for the concepts of evaluation.

By questioning the reading of evaluation concepts as if they were transparent, as if the nature of the object were merely a given fact, we seek to escape the “mirror myth of knowledge as the vision of a given object or the reading of an established text, neither of which is ever anything but transparency itself,” as Althusser^{1717 Althusser L, Balibar E, Establet R. De o capital à filosofia de Marx. In: Althusser L, Balibar E, Establet R. Ler o Capital. Rio de Janeiro: Zahar editores, 1980. v. II. p. 11-74.(18)} teaches regarding the empiricist perspective. Knowledge is production since it can transform the object. What the theorists or ‘inventory specialists’ of evaluation overlook in their chronologies is that there is no pre-existing object, detached from history, developing over generations^{3131 Guba EG, Lincoln YS. Avaliação de quarta geração. Campinas: Unicamp; 2011.}, waves^{3232 Vedung E. Four Waves of Evaluation Diffusion. Evaluation. 2010;16(3):263-277. DOI: https://doi.org/10.1177/1356389010372452
https://doi.org/10.1177/1356389010372452...} , or branches of a tree^{2323 Alkin MC, editor. Evaluation roots: tracing theorists’ view and influences. Thousand Oaks: Sage; 2004. DOI: http://doi.org/10.1177/1098214006287988
http://doi.org/10.1177/1098214006287988...} , or even a set of quality standards for virtuous professional conduct. Instead, the object is generated by the very effort of producing knowledge within its own operation of knowledge.

We start from the assumption that such a context is worth re-reading and having its memory contested, aiming to understand meanings whose discursive transparency grants them the status of established memory as truth. This understanding seeks to go beyond the transparency of general discourses on the subject-already written, read, and spoken-which may contain prohibitions, unspoken elements, and historically determining conditions of meanings yet to be elucidated.

The issue of inventories of approaches and standards around evaluation: chronologies and disputes between professionals and regulatory associations of evaluation

Inventorying already systematized approaches, fields, and evaluation methodologies has been a very common way of organizing the different productions developed throughout the history of program evaluation, based on the fragmentations and definitions of specific focuses. Such inventories exist in abundance, and they have been used by authors and practitioners of evaluation, whose need for practical guidance-as a way of understanding different ways of evaluating certain objects, even producing a certain historicity of their development as a professional area-has been common.

This is how Guba and Lincoln^{3131 Guba EG, Lincoln YS. Avaliação de quarta geração. Campinas: Unicamp; 2011.} developed what we can call a generational methodological chronology, in which groups of evaluators exercise their practices according to the emphasis on certain objectives of the evaluations, in different chronological spaces. The authors categorize evaluations according to specific periods, identifying a period focused on measuring observed performance in programs by applying tests and normative comparisons with pre-established standards (first generation); a generation emphasizing detailed descriptions of how programs produce or fail to produce expected results (second generation); another focused on prescribing, based on judgment, the performance of programs according to the achievement of their results (third generation); and finally, a generation characterized by proposing evaluations as negotiation processes involving different parties engaged with the interventions and interpreting their perspectives on the evaluated object, aiming to create evaluations sensitive to the demands of beneficiaries and constructive in terms of information production and learning throughout the process (fourth generation). The latter, as we know, constitutes the authors’ own proposition^{3131 Guba EG, Lincoln YS. Avaliação de quarta geração. Campinas: Unicamp; 2011.}, and we can consider that it would be hierarchically situated at the top of the approaches, as the state of the art evaluation, given that the authors situate the other approaches as previous generations, tacitly suggesting an evolutionary scale.

We could cite other inventory proposals for evaluation approaches, such as the one formulated by Robert Stake^{3333 Stake RE. Evaluación compreensiva y evaluación basada en estándares. Barcelona: Graó; 2006.}, who presents his analysis based on the polarization of authors and favored methodological approaches, or that of Marvin C. Alkin^{2323 Alkin MC, editor. Evaluation roots: tracing theorists’ view and influences. Thousand Oaks: Sage; 2004. DOI: http://doi.org/10.1177/1098214006287988
http://doi.org/10.1177/1098214006287988...} , who, with his metaphorical evaluation tree, delineates a set of approaches according to the emphasis given by prominent evaluators to characteristics such as value, method, and use. There is also a strict link between the history of evaluation and the paradigms and ideas of ‘theoreticians’ in the area, which are equivalent to stages of stagnant theoretical development linked to specific names and personalities, disregarding the socio-historical context, except as an illustrative background, as we can see in Shadish et al.^{2424 Shadish WR, Cook TD, Leviton LC. Foundations of program evaluation: theories of practice. Newbury Park: Sage Publications; 1995.}.

Evaluation approaches have also been inventoried by Evert Vedung, using the metaphor of waves^{3232 Vedung E. Four Waves of Evaluation Diffusion. Evaluation. 2010;16(3):263-277. DOI: https://doi.org/10.1177/1356389010372452
https://doi.org/10.1177/1356389010372452...} , in which four waves overlap, leaving sediments that impact evaluations over time: the scientific wave, marking the beginning of program evaluations in the 1950s and 1960s; the dialogue-oriented evaluation wave, critiquing the previous model in the early 1970s; the neoliberal wave, emerging in the late 1970s and pressuring evaluations towards market logic; and the evidence-based evaluation wave, starting in the mid-1990s, which, according to Vedung, represents a revival of the tradition expressed in the first wave.

In an important book on evaluation translated into Brazilian Portuguese, Worthen, Sanders, and Fitzpatrick^{3434 Worthen BR, Sanders JR, Fitzpatrick JL. Avaliação de programas: concepções e práticas. São Paulo: Gente; 2004.} present an extensive range of approaches and practical procedures, including several examples of what we have discussed: goal-oriented evaluations; management-oriented evaluations; consumer-oriented evaluations; expert-oriented evaluations; adversary-oriented evaluations; and participant-oriented evaluations. Regarding health evaluation, we can cite inventories that define specific types of evaluations, such as the one organized by Brousselle et al.^{3535 Brousselle A, Champagne F, Contandriopoulos AP, et al., organizadores. Avaliação: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.}: normative evaluation; strategic analysis; logical analysis; production analysis; effects analysis; economic evaluation; and implementation analysis. Additionally, Vieira-da-Silva proposes evaluations within the context of demands and needs emerging in universal health systems, such as those in Brazil^{3636 Vieira-da-Silva LM. Avaliação de políticas e programas de Saúde. Rio de Janeiro: Editora Fiocruz; 2014. 110 p.}: coverage, access, equity, effectiveness, efficiency, quality, implementation, and user perception evaluations. More recently, a similar effort has been made by a group of health evaluation researchers, in the proposition of specific models and methods for the practice of evaluation, with emphasis on the approach centered on theory and modeling of interventions^{3737 Santos EM, Cardoso GCP, Oliveira EA. Aprendendo Avaliação: modelos e métodos aplicados. Rio de Janeiro: Cebes; 2023. DOI: https://doi.org/10.5935/978-65-87037-06-6.B001
https://doi.org/10.5935/978-65-87037-06-...} . We can also mention the proposal developed by Russel Glasgow^{3838 Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322-7. DOI: https://doi.org/10.2105/ajph.89.9.1322
https://doi.org/10.2105/ajph.89.9.1322...} for impact evaluation, the RE-AIM model (an acronym for its component attributes: reach, efficacy, adoption, implementation, and maintenance).

The definition of standards, criteria, attributes, and dimensions to characterize the evaluation has also served for its own standardization, by defining priority focuses of analysis that, ultimately, represent concepts whose effects of meanings are remarkable, but not explicit. For example, the six evaluation criteria defined by the Development Assistance Committee (DAC) network of the Organization for Economic Cooperation and Development (OECD) are: relevance, effectiveness, impact, coherence, efficiency, and sustainability^{4040 Organisation for Economic Cooperation and Development. Applying evaluation criteria thoughtfully. Paris: OECD Publishing; 2021. DOI: https://doi.org/10.1787/543e84ed-en
https://doi.org/10.1787/543e84ed-en...} . There are also established standards for meta-evaluation, or the evaluation of evaluations and their practitioners, which encompass dimensions such as the conduct and stance of evaluators and concepts to be applied. The Joint Committee on Standards for Educational Evaluation (JCSEE), an association of U.S. and Canadian professionals focused on evaluation quality, has adopted the following standards: utility, feasibility, propriety, accuracy, and accountability^{4141 Yarbrough DB, Shula LM, Hopson RK, et al. The Program Evaluation Standards: A guide for evaluators and evaluation users [Internet]. 3. ed. Thousand Oaks, CA: Corwin Press; 2010 [acesso em 2024 jan 10]. Disponível em: https://jcsee.org/program/
https://jcsee.org/program/...} .

It is noted that, alongside the definition of criteria for evaluation, the logic surrounding inventories further advances towards the standardization of professional practices related to evaluation itself, as outlined in the JCSEE’s proposals in other documents, which could be considered regulatory. The American Evaluation Association (AEA) defines a set of guiding principles for the professional practice of evaluators, centered around five dimensions: systematic inquiry, competence, integrity, respect for people, and common good and equity. The stated objectives are to “govern the behavior of evaluators in all stages of the evaluation”^{4242 American Evaluation Association. American Evaluation Association: Guiding Principles for Evaluators [Internet]. Washington, DC: AEA; 2011 [acesso em 2023 jan 25]. Disponível em: https://www.eval.org/Portals/0/Docs/AEA_289398-18_GuidingPrinciples_Brochure_2.pdf
https://www.eval.org/Portals/0/Docs/AEA_...} . This is the case for work produced in the context of Latin America and the Caribbean, initiated by the Latin American and Caribbean Network of Monitoring, Evaluation, and Systematization (ReLAC), which outlines four guidelines for evaluation practice in a document: methodological rigor, professional ethics, cultural understanding, and relevance^{4343 Bilella PR, Martinic SV, Soberón LA, et al. Diretrizes para Avaliação para a América Latina e o Caribe [Internet]. Ciudad Autónoma de Buenos Aires: Akian Grafica Editora S.A; 2016 [acesso em 2024 jan 10]. Disponível em: https://www.deval.org/fileadmin/Redaktion/Bilder/2016_FINAL_ESTANDARES_DIGITAL_PORTUGUES.pdf
https://www.deval.org/fileadmin/Redaktio...} . Guidelines for evaluation practice have also been developed by the Brazilian Network for Monitoring and Evaluation (RBMA), which aims to guide and standardize the professional conduct of evaluators in the country by focusing on the following standards: Learning and Utility, Rights and Integrity, Contextualization and Valuation, and Method and Feasibility^{4444 Silva RR, Joppert MP, Gasparini MFV, organizadores. Diretrizes para a prática de avaliação no Brasil [Internet]. Rio de Janeiro: Rede Brasileira de Monitoramento e Avaliação; 2020 [acesso em 2024 jan 10]. Disponível em: https://sinapse.gife.org.br/download/diretrizes-para-pratica-de-avaliacoes-no-brasil
https://sinapse.gife.org.br/download/dir...} .

In all these examples of work on typologies, approaches, criteria, standards, and norms for evaluations, there is a relationship of meaning that links evaluation to some other qualifying element, or to others-these indeed consisting of opaque conceptual formulations that deserve to be understood, as these concepts underpin the broader meanings of various evaluative practices. The discursive functioning of such productions operates from constructions that seek to present themselves between the political and technical dimensions, showing themselves as endowed with wide acceptability and unquestionable relevance, under the argument that the evaluation needs to meet quality criteria, such as those mentioned above.

What these forms of inventorying evaluation approaches have in common, in terms of the logic of organizing their reasoning, include, first and foremost-as identified by Furtado and Vieira-da-Silva^{4545 Furtado JP, Vieira-da-Silva LM. A avaliação de programas e serviços de saúde no Brasil enquanto espaço de saberes e práticas. Cad Saúde Pública. 2014;30(12):2643-2655. DOI: https://doi.org/10.1590/0102-311X00187113
https://doi.org/10.1590/0102-311X0018711...} , when analyzing Guba and Lincoln’s generational methodological chronology, but applicable to other cited actors-the use of an artifice that provides linearity and homogeneity to a supposedly historical reading, as if generations succeeded each other without conflicts, contradictions, or historical determinations, which is associated with a traditional way of interpreting the history of sciences^{4646 Althusser L. Apresentação. In: Canguilhem G. O normal e o patológico. Rio de Janeiro: Forense Universitária; 2020. p. 137-139.}. Second, we observe that these proposals for chronological inventories of evaluation approaches, in which methodological characteristics are imperative for delineating distinctions, share an emphasis on authors and their individual trajectories, their inventive capabilities, and formulations that supposedly broke away from problematic paradigms considered traditional, outdated, or limited.

We understand that the logic employed in the literature aims to support a rational discourse for the historical development of evaluation and, although we can recognize the merits in systematizing the proliferation of applications that this field has accumulated over the years, it is based on an epistemological position that emphasizes the primacy of subjects as producers of their social practices. This is how the direct association between authors practicing evaluation and their methodological formulations emerges in the literature cited above as an imaginative flow capable of setting in motion the gears of an established and autonomous area of practice, in which the subjects would be the source of their own formulations.

Similarly, in the set of documents that seek to standardize evaluation practice-both in its professional dimension as a service sold in the consulting market and in research practices within universities and research centers-we observe a set of standards that are presented as self-evident and unquestionable, thus justifying their universalization (standardization). In this type of discourse, the intentionality of associations and professional networks (and even nations), as well as national and international organizations and institutions, remains opaque. These entities, from their privileged positions, determine what should or should not be the norm for a field of practices as diverse and contextually varied as evaluation, as previously observed^{3030 Furtado JP. Por uma meta-avaliação sem metafísica. Rev Bras Aval. 2022;11(2):e112022. DOI: http://dx.doi.org/10.4322/rbaval202211020
http://dx.doi.org/10.4322/rbaval20221102...} .

In contrast to this logic, and based on a historical epistemological perspective, it seems to us that the practice of program evaluation finds its meanings in its objects, in its favored methodological approaches, in the specific aspects of interventions it delineates to establish its focus, and in the types of conduct it defines for its practitioners, which are the conceptual relationships that determine its meanings, rather than the formulation of subjects. It is in the relationship with the object and the favored choices that we can capture discursive remnants revealing intersections of meaning. In other words, it is via the plurality of possible determinations that evaluation begins to emerge, from its foundational concepts, or more precisely, from the issues that justify the development of these concepts. It is in this erasure of the functioning of concepts, to the detriment of subjects as sources of their knowledge and norms, that we bring forth our specific reading approach.

Final considerations

By mobilizing Discourse Analysis and Historical Epistemology as theoretical assumptions to address program and service evaluation, based on their formulations in the context of U.S. program evaluation-which, as mentioned, produces meaning effects in health evaluation in Brazil-we start from a sense of estrangement from what is most familiar to us, as it constitutes our primary object of investigation and our own professional practice. Such a choice requires a constant process of critical reflexivity and analytical implication, precisely because we understand that we are not oblivious to the idealistic trap of the subject as a producer of knowledge, as already discussed. It is also not a matter of placing oneself above the conflicts, interests and the reproduction of possible erasures of meanings that the ideological functioning of the discourse produces in any and all analysts. What we aim in this work is to clarify how this functioning operates and to propose reading approaches that, if not overcoming, at least challenge and deconstruct a discourse that presents itself as truth and obscures other perspectives, in the process of building theoretical knowledge.

In this essay, we aim to present the theoretical and analytical foundations for an initial understanding and critique of the dominant discourse in program and service evaluation. This discourse, via its official documentation concerning approaches, methods, and models, as well as norms and standardizations applied to professional practice in various contexts, stabilizes a discourse and renders it opaque in its intentions and historical disputes over meaning.

Ultimately, this is an initial effort to disrupt the discursive stability of a field whose assumptions and intentions are in constant dispute and represent distinct, even antagonistic, societal proposals. Program and service evaluation, particularly in the health sector, can either produce material effects that support social transformations and bolster struggles for political and epistemological independence and agency, or it can reproduce conservative meanings aimed at maintaining existing power relations and reinforcing privileges.

The materialist reading approach for Discourse Analysis of health evaluation, as presented in this essay, represents the first step toward understanding that this discourse is guided by the establishment of concepts which, in turn, provide meanings that warrant further analysis. Since this field utilizes a profusion of concepts to structure its scientific discourse (methods and approaches) and political discourse (norms and standards of conduct), as well as the various associations between them, a general initial approach seemed necessary.

Whether via a discursive construction aiming to reconstruct a linear and objective history, or via the definition of established norms and standards seen as parameters for good evaluations, what we have observed in the standardized discourse on program evaluation-via our specific reading approach-is an attempt to rationalize the field through constructs that render its contradictions invisible. Ultimately, chronologies, inventories of approaches, and conduct parametizations are not the result of material arrangements and disputes forged within the practice of evaluation and its relationships with specific discursive formations. Instead, they impose themselves as forces of truth supported by the logic of technicality and the harmonious development of evaluation approaches and models over time. The concepts mobilized are not treated as historical or immersed in networks of other significant concepts, which give them material meaning, but rather as propositions seen as bolder technical solutions, depending on each context and intentionality of the evaluation and its practitioners.

We believe that it is possible, from this first impulse, to dive into specific concepts, whose meanings deserve to be analyzed and disputed, in a movement of destabilization and search for explanatory syntheses, always transitory and complex, about the determinations and ideological implications of such concepts for the discourse of health evaluation in Brazil.

References

Publication Dates

History