Abstract

INTRODUCTION

When considering the ingenuity of the criteria of meritocracy engendered by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) to qualify post-graduate programs, it is necessary both to improve the quantitative mechanisms and to examine new ways of evaluating the impact of the researchers and programs.

On following the process of accreditation of a Masters as post-graduate coordinator, I published an opinion piece entitled: “There are many doctors, but few ‘social entrepreneurs’”. ¹1 . Akerman M. Há muitos doutores, mas faltam empreendedores sociais. Diario Grande ABC. 2005 ago 7:13.

Entrepreneurship or taking a risk would be the expected result of the “research – teaching- extension” triad and combined mechanisms of evaluation would need to be practiced. As mentioned in Ribeiro, RJR “the university will truly become university when the prince (research) kisses the frog (extension)” (Almeida Filho, 2007). ⁸8 . Almeida Filho N. Universidade nova: textos críticos e esperançosos. Brasília/Salvador: Editora UnB/EDUFBA; 2007.

And how is this to be done? Perhaps by identifying when research also “serves humanity and not just science”.

Borrowing Packer’s expression, ^a b Akerman M. Diferenciais intra-urbanos de saúde: estudo de caso de macrolocalização de problemas como estratégia de influenciar políticas públicas In: Equidade e saúde: contribuições da epidemiologia. Rio de Janeiro: ABRASCO; 1997. v.1, p. 177-86. that “a doctor may have read one scientific article and have saved two lives”, by not reporting this in a systematic manner to the scientific community, “he helps humanity, but not science”. Thus, the author of said article read by the doctor will never know of the impact caused by article.

Moreover, Packer ^a a Packer A. Comunicação pessoal; 2012. (2012) states that the scientific community recognizes the need for this type of qualification of research or of its repercussions on the life of the subjects; but, for lack of a validated methodology, we continue to use the enshrined measure of scientometrics ³3 . Akerman M, Bousquat A. Mapa de risco da violência das cidades de São Paulo, Curitiba, Salvador e Rio de Janeiro. Sao Paulo Perspec. 2000;13(4):112-20. DOI: 10.1590/S0102-88391999000400012
https://doi.org/ 10.1590/S0102-883919990... to evaluate the impact of journals, articles and researchers.

Would this attempt to qualify the impact of research/article/researcher not be the subterfuge of someone who would never achieve the competitive quantitative levels, hiding behind qualitative raptures? We have not reached a conclusion on this insinuation. According to Packer ^a a Packer A. Comunicação pessoal; 2012. (2012), the “science of citations” is also susceptible to Pareto’s Law, known as the “80-20 principle”, in which 80% of citations would always be attributed to 20% of researchers. Coimbra Jr. ¹⁰10 . Coimbra Jr CEA. Produção científica em saúde pública e as bases bibliográficas internacionais. Cad Saude Publica. 1999;15(4):883-8. DOI: 10.1590/S0102-311X1999000400022
https://doi.org/ 10.1590/S0102-311X19990... argues that the field of Public Health is not represented in the “lions share” of the most cited journals, not even in international journals, such as the American Journal of Public Health and the International Journal of Epidemiology.

So the “crumbs” of research promotion would be left for those 80% of researchers with an inferior number of citations? Of that contingent of research, what has the greatest potential to help humanity? Are there other forms of evaluating the impact of articles that does not involve simply counting citations?

The aim of this article was to suggest possibilities for the qualitative analysis of the impact of scientific production.

A POSSIBLE STARTING POINT

In 1993, a broad piece of research entitled “Health and Environment: a study of intra-urban differences in developing countries, in Acra, capital of Ghana and Sao Paulo, Brazil”, ¹⁶16 . Stephens C, Timaeus I, Akerman M, Avle S, Maia PB, Campanário P, Harpham T. Field study: São Paulo, Brazil. In: Stephens C, organizador. Environment and health in developing countries: an analysis of intra-urban differentials using existing data. London: London School of Hygiene and Tropical Medicine; 1994. p. 49-80. demonstrated the existence of a social gradient with regards the risk of dying from various selected chronic diseases, external causes, respiratory disease and communicable diseases when comparing different geographical territories in the cities studied. ⁷7 . Akerman M, Stephens C, Campanario P, Maia PB. Saúde e meio ambiente: uma análise de diferenciais intra-urbanos enfocando o Município de São Paulo, Brasil. Rev Saude Publica. 1994;28(4):320-5. DOI: 10.1590/S0034-89101994000400013
https://doi.org/ 10.1590/S0034-891019940... ^{, 14}14 . Stephens C, Akerman M, Maia PB. Health and environment in São Paulo, Brazil: methods of data linkage and questions for policy. World Health Stat Q. 1995;48(2):95-107. ^{, 15}15 . Stephens C, Akerman M., Avle S, Maia PB, Campanário P, Doe B, Tetteh D. Urban equity and urban health: using existing data to understand inequalities in health and environment in Accra, Ghana and Sao Paulo Brazil. Environ Urban. 1997;9(1):181-202.

More than an ecological epidemiological study of intra-urban differences, it practiced the exercise in negotiation of constructing composite indicators, ^b b Akerman M. Diferenciais intra-urbanos de saúde: estudo de caso de macrolocalização de problemas como estratégia de influenciar políticas públicas In: Equidade e saúde: contribuições da epidemiologia. Rio de Janeiro: ABRASCO; 1997. v.1, p. 177-86. ^{, c}cAkerman M. Metodologia de construção de indicadores compostos: um exercício de negociação intersetorial In: Condições de vida e saúde. Rio de Janeiro: ABRASCO; 1997. v.1, p. 95-113. which influenced a set of research into violence, quality of life and social exclusion/inclusion. ⁴4 . Akerman M, Massei W, Cabral S, Broch A, Cremonese A, Alves TS, et al. A concepção de um projeto de observatório de qualidade de vida: relato de uma experiência realizada em Campinas. Saude Soc. 1997;6(2):83-100. DOI: 10.1590/S0104-12901997000200010
https://doi.org/ 10.1590/S0104-129019970...

5 . Akerman M. Qualidade de vida e exclusão: quanto custa incluir? Debates Socioambient. 1996;2(4):8-9. ^{- 6}6 . Akerman M, Bousquat A Mapa de risco da violência da cidade de São Paulo. São Paulo: Centro de Estudos de Cultura Contemporânea; 1996.

One of those pieces of research was the Violence Risk Map of Sao Paulo, SP, Southeastern Brazil. ⁶6 . Akerman M, Bousquat A Mapa de risco da violência da cidade de São Paulo. São Paulo: Centro de Estudos de Cultura Contemporânea; 1996. This initiated maps of violence risk in the cities of Curitiba, PR, Rio de Janeiro, RJ, Southeastern Brazil and Salvador, BA, ³3 . Akerman M, Bousquat A. Mapa de risco da violência das cidades de São Paulo, Curitiba, Salvador e Rio de Janeiro. Sao Paulo Perspec. 2000;13(4):112-20. DOI: 10.1590/S0102-88391999000400012
https://doi.org/ 10.1590/S0102-883919990... Northeastern Brazil, forming part of Federal Government efforts to draw up a National Human Rights Plan, created in 1996 by the Ministry of Justice.

It was decided to report the results of these colored maps in the form of an easy-to-read bulletin, divulged as a tool for informing policies and raising society’s awareness of the magnitude of urban violence.

Following this logic, it was decided to report the results of mapping of violence in the city of Sao Paulo, exclusively, in that city’s press.

The material, carrying the headline “Jardim Ângela is the most dangerous place in the world”, ^ddFolha de São Paulo. Jardim Ângela é o lugar mais perigoso do mundo, 1996, caderno cotidiano, página 3. introduced the report with the results of the research based on a small chart which compared mortality rates from homicide in Cali and Medellin, Colombia, with neighborhoods in the city of Sao Paulo, where the rate in Jardim Ângela exceeded the rates of the Colombian cities, at the time considered to be extremely violent.

Following this, there was a steep decline in the violence rates in Jardim Ângela which persists to this day. There was a 73.3% drop in the homicide rate, according to the text below:

It is not intended that this narrative establishes a cause-effect relation between the headline in 1996 and the significant decrease in homicide rates in 2005. However, this headline has repercussions up to the present day in newspapers and seems to play a social marketing role, influencing the mobilization and rolling out of actions aimed at this area, which was pushed into the spotlight by the headline, which did nothing more than illuminate a situation that had existed long before the headline.

It seems that a newspaper headline, which does not come within the scope of science, indicated that the research may have had a “great impact on humanity”.

OTHER TYPES OF IMPACT

Who has not heard the expression “This had a huge impact on me”? With this expression, the subject is talking of something that affected them, something they experienced.

According to Bondía: ⁹9 . Bondia JL. Notas sobre a experiência e o saber da experiência. Rev Bras Educ. 2002;19(1):20-9. DOI: 10.1590/S1413-24782002000100003
https://doi.org/ 10.1590/S1413-247820020...

Thus, Bondía ⁹9 . Bondia JL. Notas sobre a experiência e o saber da experiência. Rev Bras Educ. 2002;19(1):20-9. DOI: 10.1590/S1413-24782002000100003
https://doi.org/ 10.1590/S1413-247820020... alerts us that a society “constituted under the sign of information” makes experiencing difficult.

There is no room here for the conflict between knowledge/information and knowing from experience/feeling, each pair with their purpose, their ethics and aesthetics, but we must advocate for adequate tension between the two pairs.

However, Bondía ¹⁶16 . Stephens C, Timaeus I, Akerman M, Avle S, Maia PB, Campanário P, Harpham T. Field study: São Paulo, Brazil. In: Stephens C, organizador. Environment and health in developing countries: an analysis of intra-urban differentials using existing data. London: London School of Hygiene and Tropical Medicine; 1994. p. 49-80. comments that modern science distrusts experience and creates a safe path translated by the method which reduces experience to experimentation.

If the logic of experimentation is agreement, consensus and homogeneity, in knowing from experience, difference, heterogeneity and singularity are important. ⁹9 . Bondia JL. Notas sobre a experiência e o saber da experiência. Rev Bras Educ. 2002;19(1):20-9. DOI: 10.1590/S1413-24782002000100003
https://doi.org/ 10.1590/S1413-247820020...

If the scientometric base ¹³13 . Pinto AC, de Andrade, JB. Fator de impacto de revistas científicas: qual o significado deste parâmetro? Quim Nova. 1999;22(3):448-53. DOI: 10.1590/S0100-40421999000300026
https://doi.org/ 10.1590/S0100-404219990... is the impact factor and other bibliometric indicators producing objective, regular, repeatable, comparable, impersonal, infinite, universal results, it would be very daring of Brazilian researchers to be able to explore the development of “experiencimetrics”, which would produce personal and comparable results and which would also contribute to better understanding of the scientific research mechanism as a social activity.

RESISTENCE TO A DECLARED ELITISM?

In the “preamble” of the “rodízio” traffic law in Sao Paulo – limiting which days cars with particular number plates can be used – it states: “according to research by Prof. Pedro Jacobi, who pointed out the perceived increased pollution in the city by housewives in a neighborhood by the quantity of soot forming on the sheets hanging out on the clothes lines”. ^ffCentro de Estudos de Cultura Contemporânea. Coordenadoria de Educação Ambiental. Respira São Paulo: resultado da pesquisa sobre a Operação Rodízio. São Paulo; 1997.

On seeing this described in the law, it is interpreted immediately as being a genuine impact of scientific research.

Universities in Europe are in the process of creating ways of evaluating impact that are not limited to citation indexes, which frequently lead to opinions that the academic world is split between those who do “little science” and those who do “real” science when comparing the citation indexes for different researchers.

What needs to be done is to find forms of inducing/designing a wider repertoire of indicators for evaluating the career of a university professor, considering other fields of activity, measured with the same value as the capacity for disseminating scientific articles, so that the quantitative criterion of citations is not solely predominant.

FINAL CONSIDERATIONS

First, it is necessary to recognize the provisory character of this work – tentative and precarious, in the words of Deleuse & Guattarri: ¹¹11 . Deleuze G, Guatarri F. Mil Platôs. Rio de Janeiro: Editora 34; 1995.

Second, it is necessary to recover the context of academic life so as to “tighten” meritocracy in research as a determining principle in evaluating academic performance and advocate that other areas of academic performance also acquire the same degree of evaluative relevance as research and publishing.

The third point that needs to be highlighted is: what is the purpose of science? To help humanity or to self- reproduce among peers? How is it possible to know if it is really helping humanity? At the moment we only have the quantitative methods already validated, derived from scientometrics, but we lack methods which could also qualify the impact of science/research in the sense of its “contribution to humanity”. Moreover, the most enshrined method – the impact factor measured by number of citations – has its limitations and critics such as, for example, that attributed by Pinto & Andrade: ¹³13 . Pinto AC, de Andrade, JB. Fator de impacto de revistas científicas: qual o significado deste parâmetro? Quim Nova. 1999;22(3):448-53. DOI: 10.1590/S0100-40421999000300026
https://doi.org/ 10.1590/S0100-404219990...

In order to achieve the aspiration that research’s main purpose is to “help humanity”, indirect criteria could be established, for example, potential or realized capacity of research to influence public policies: (1) level of participation of social players in the process of drawing up the research; (2) communication strategies between the social players and the researchers involved; (3) clarity in defining objectives; (4) visibility of the research process and dissemination of the results; (5) meeting deadlines; and (6) achieving previously established objectives. ²2 . Akerman M. Examinado elementos que possam influenciar a formulação de políticas em estudos que utilizam indicadores compostos: o chão contra o cifrão. Cienc Saude Coletiva. 2000;5(1):115-24. DOI: 10.1590/S1413-81232000000100010
https://doi.org/ 10.1590/S1413-812320000...

These criteria are only a first approximation for evaluating the elements research that aim to influence policies. Other elements may not have been identified, and other studies exploring these issues are welcome, as would a clearer definition of indicators which allow the association between these elements and influence on public policies to be certified. In this discussion, the emphasis is on the fact that, in the case of there being genuine “scientific willingness” on the part of the researchers in influencing policy through their work, the abovementioned criteria need to be directly and explicitly included in the design of the research as elements that could potentially influence the formulation of public policies. ²2 . Akerman M. Examinado elementos que possam influenciar a formulação de políticas em estudos que utilizam indicadores compostos: o chão contra o cifrão. Cienc Saude Coletiva. 2000;5(1):115-24. DOI: 10.1590/S1413-81232000000100010
https://doi.org/ 10.1590/S1413-812320000...

Thus, these elements indicate the possibility of searching for a mode of analysis to highlight “knowledge from experience” of the research in influencing public policies, blazing trails to constitute evaluative “experiencometrics”.

A fourth thread of discussion, connected to the previous one, proposes inserting perceived and documented “narratives” (knowledge from experience) of the impacts of the research, for example, in subjects, population groups, territories or laws such as those exemplified here: (1) in the case of the “Rodízio” law and the influence of research by Prof. Pedro Jacobi and (2) the case of the Violence Risk Map of Sao Paulo City and its connection with “Jardim Ângela”.

Therefore, it would become the role of researchers to position themselves as a collector of experiences, inspired by the willingness to share the perceived and observed impact of their research. Such an exercise may even be encouraged by preconceived scripts during the evaluation processes, and incorporated into CAPES or into the Currículo Lattes network, e.g., and they would have some weight in evaluating research, post-graduate programs and requests for subsidies to development agencies.

Grossman & Cardoso ¹²12 . Grossman E, Cardoso MHCA. As narrativas em medicina: contribuições à prática clínica e ao ensino médico. Rev Bras Educ Med. 2006;30(1):6-14. DOI: 10.1590/S0100-55022006000100002
https://doi.org/ 10.1590/S0100-550220060... give some indications that may serve for proposing this script when they affirm that “narratives are structured on five elements, namely: the facts, the characters, time, space and the narrator”. Moreover, it is possible to suggest creating a script which would come with documental material in order to support and state in detail the facts.

References

Publication Dates

History