DEBATE DEBATE

Evaluation in health: regulation, research, and culture in the challenges of institutionalization

I certainly do not intend to give a rebuttal to the discussants' comments, since they are both pertinent and relevant, and I am thus tempted to reiterate them. However, I will merely highlight a few points to avoid redundancy. I thought it would be interesting to organize my remarks as clues to answers or treatment of the questions raised by Yunes concerning the applicability of the French experience, so as to form a preliminary list of ingredients in a basic recipe for Institutionalization, inspired by international cuisine, yet with a Brazilian flavor. Gerard de Pouvourville sheds considerable light on the matter when he identifies the limits of this experience "...we are still far short of many objectives..." and makes suggestions to implement institutionalization in France, since I believe that the potential for such "generalization" is reinforced by the agreement amongst the various colleagues' participating in this debate:

1) evaluation as an intrinsic part of public services management, a requisite for accountability and modernization of the state. In this sense, evaluation provides the tools for the state's regulatory role, crucially important to ensure "equity" in health care in the case of privatization of providers and hiring of local partners in decentralized interventions (which would certainly include, but not be restricted to, the "old IPDA circuit" mentioned by Yunes). Regulation, as an act to facilitate governance and quality improvement, an issue also approached by Claudia Travassos, would require the use of more participatory strategies, with flexible, decentralized evaluation structures. Ligia Vieira adds to the debate by recalling that the use of local standards should not rule out the possibility of comparing problems and interventions on national and international scales, and I feel that techno-scientific committees, together with specific health programs or councils at various levels, can provide such important back-up. It is thus interesting to highlight the different forms of regulatory logic (techno-scientific, professional, economic, and democratic), characterized exceptionally well by André-Pierre Contandriopoulos, since they define the prime methodologies orienting the focus of evaluation and thus, in a sense, its results. This approach reappears in Ligia's commentary, evoking intersections between the scientific and professional fields and power, expressing "the dispute over which methodologies are most valid" and underscoring the French preference for the "sur mesure" approach. I feel it is also necessary to point out that public policy and program evaluation performed (in)directly by executive branch agencies should be the object of regulation (meta-evaluation or auditing of evaluation effectiveness) by a different branch of government, just as with any other activity.

2) funds to promote evaluation research, including the development of a community with a structure to define proper scientific work, exploring the impact of public sector activities on society, reducing the incidence of "quick and dirty studies". This prioritization agrees with that of Claudia Travassos, who, given the "chronic and severe lack of academic and technical specialists" in the Brazilian context, is concerned over the resources needed to stimulate the production of knowledge and training of expert personnel in this field, with the new technical skills required by evaluation teams, including communications, teaching, and policy-making. Another problem in promotion of research, in addition to the issue of scientific legitimacy identified by Gérard de Pouvourville, is the ability to promote the connection between thinking and acting, knowledge and action, essential to legitimize the evaluation of programs and technologies, "whether they aspire to scientific research status or not" (Maria Novaes). According to Ligia, this relationship between evaluation and decision-making is a political and ethical (rather than theoretical and methodological) issue, involving choices in which, faced with the various rationales, "the institutionalization of evaluation for a public health system means seeking to ensure the hegemony of the technical/health rationale...". The counterpoint by Gastão Wagner is indispensable to avoid turning the institutionalization proposal into a "rationalist delirium", since political decisions will continue to be moved by "the motor force of desire, interests, and needs". As a woman from the hinterlands, I also see the "long and winding roads" (with clearings and turns along the way) in this rather wild territory of evaluators. Such meandering pathways appear when one becomes aware of the on-going challenge of (de)constructing our field of activity, in keeping with the policies and programs that evolve (like the institutions) in their efficacy in the "wager against previous evidence, against warnings that the proposal will never work", without losing the spirit of advocacy, to use what we know to be an unparalleled word.

3) evaluation as a process fostering democratic debate, which implies better redistribution of "access to evaluation" by the various actors who, whose own means are insufficient for them to evaluate public services and use such evaluation to counterbalance opposing interests. With regard to this approach, the comment by André-Pierre Contandriopoulos is quite "daring" in that it points to the emergence of a "true" culture of evaluation, or the generalization of its practice with the hegemony of democratic logic, like institutionalization, at all levels of society, processes fostering individual and collective learning in such a way that all actors can overcome the exclusive logic of regulation. Institutionalization of evaluation would thus foster "the subordination of vested interest groups' power to that of individuals who collectively constitute society". I see a similarity between this approach and that of a "social warning" (Gastão Wagner), contributing to a "radical democratization of political life at both the national and internal institutional levels".

In concluding this difficult task of choosing highlights, given the wealth of contributions to the debate, I would call on readers to share in the proposal raised by Yunes, viewing this debate as part of work that should be continued with the desirable exchange of ideas concerning the applicability of the French experience and other initiatives to help respond to and reformulate questions identified by him. I also consider it crucial to reaffirm my conviction that "knowledge of the reality of others fosters a better understanding of our own", as stated so well by Maria Novaes concerning the justification for my article, but I agree with her that analysis of programs and policies should view them as "socially and technically constructed alternatives for specific contexts, and not as universal models".