Abstract

INTRODUCTION

Given the immense effort dedicated to scientific investigation and to ensure that its results are not limited to “advances in knowledge”, it is expected that at some point this knowledge is transformed into action, yielding tangible benefits (material or non-material) for society. This long, and far from linear process between knowledge and action, although acknowledged and desirable, has not proved to be very practicable and there are difficulties with regards perception, consensus, measurement and assessment. This process, generically known as transfer, translation or, more recently, knowledge translation, ⁷7 . Guimarães R. Pesquisa Translacional: uma interpretação. Cienc Saude Coletiva. 2013;18(6):1731-44. DOI:10.1590/S1413-81232013000600024
https://doi.org/10.1590/S1413-8123201300... has been understood and interpreted in various ways. For some, it has always been the tempting idea that knowledge produced can be immediately translated into a tangible action, it being enough that we desire it to be so. This version leaves out the large cognitive, epistemological and political distances between producing the knowledge and producing the action.

Another issue which has often been a source of confusion concerns the differences between knowledge and information. Scientific knowledge involves a production process with various stages and ultimately aims to change the state of knowledge that was previously available. With regards information, its objective is to present a fact and how that fact is configured at a given moment and in a given context. Information may be knowledge, however, it is not scientific knowledge.

It is in this context, in which information, knowledge and action are mixed, that programs for evaluating academic performance, and specifically the production of scientific knowledge, are situated. The configuration of the assessment of scientific programs should, obviously, be linked to the configuration of the scientific programs. However, on reviewing current scientific programs it is notable that only some characteristics which are central to the scientific activity are included in the evaluation processes, whereas others, no less important, are either not included or are secondary.

Science, in whatever area, is disseminated in a written form; any new piece of scientific knowledge needs to be written, revised, reviewed and verified in order to be accepted by peers. Add to this the fact that the evolution of science contains cumulative elements: each new project needs to record and cite the prior knowledge needed to substantiate and justify new proposals. These characteristics formed the basis for developing the systems of bibliographic cataloging and, derived from these, the systems of counting citations, which made it possible for various current bibliometric indexes (e.g., number of citations, impact factor and h-index) to be developed and universally used in systems of scientific evaluation. With the exception of the systems of registering patents, which are limited to those areas of applied science that produce patentable products and which are used in scientific evaluation, nothing else has been developed to systematically record the potential effects of activity in the different scientific fields, apart from bibliographic effects. It is not by chance that bibliometrics, with its various indexes, became the reference for scientific evaluation and the principal strategy for “objective” measurement of the scientific impact of an individual, institution or journal, eventually becoming confused with scientometry. This occurred in spite of everyone being aware of its principal drawback: the fact that the indexes measured only the modus operandi part of the science (publishing and citations), leaving aside its various potential non-bibliographic effects on the world.

The quantity and the form of publishing and the intensity of citation varies between the scientific fields. However, it is important to recognize that there is no such thing as science without publishing or science without citation. Thus, bibliometric evaluation has taken on great relevance for scientific evaluation in general and is central to some scientific fields or ways of doing science. ⁶6 . Gibbons M, Limoges C, Nowotny H, Schwartzmann S, Scott P, Trow M. The new production of knowledge: the dynamics of science and research in contemporary societies. London: Sage; 1994. However, one would expect this approach to have less significance in those fields of ways of doing science in which other products in addition to bibliographic production or citation are expected. This difference between ways of doing science was well established by Gibbons, ⁶6 . Gibbons M, Limoges C, Nowotny H, Schwartzmann S, Scott P, Trow M. The new production of knowledge: the dynamics of science and research in contemporary societies. London: Sage; 1994. separating traditional academic science, based on disciplines and centered around a system of merit based on publication and on effects identified using bibliometry (named mode 1), from scientific practice aimed at problems, not restricted to specific disciplines and the products of which are not limited to publications, although of course not excluding them (named mode 2). However, in spite of proposals such as those of Gibbons having a strong influence on the way in which the organization of science is viewed, what is also notable is the tendency not to differentiate the evaluation processes, which continue to be strongly centered on bibliometry.

In 2000, in one of the first attempts to evaluate the excellence of research in area in which mode 2 predominates, the English Royal Academy of Engineering presented an innovative set of ideas about a way of perceiving and evaluating the impact of this way of doing science. ⁹9 . Royal Academy of Engineering. Measuring excellence in engineering research. London; 2000. It is interesting that it reaffirms the centrality of bibliometric indexes in mode 1 evaluations and, where deemed necessary, withdraws its importance and centrality to mode 2 ways of doing science. In this mode, elements such as the practical results of the research are emphasized, these being considered essential in evaluating excellence.

Similarly, and using the earlier document as the main point of reference, in 2002 the Dutch Royal Academy of Arts and Sciences published a document ¹⁰10 . Royal Netherlands Academy of Arts and Sciences, Council for Medical Sciences. The societal impact of applied health research: towards a quality assessment system. Amsterdam; 2002. which clearly highlighted the importance of non-bibliometric impact in evaluating applied research in the area of health. Firstly, it emphasized broadening the coverage of bibliometric impact to cover more than just peer citations, in other words, exclusively among scientific manuscripts, so as to include other forms in which they are used, for example, in documents on which policies, norms and guidelines for professional practice are based. The need to focus the impact of applied research in health on health care providers and on health care organization processes and the process of designing, implementing and monitoring decisions was highlighted. In order to establish such advances more clearly, it was proposed that potential impact ( ex-ante ) – transforming social issues into research – be discriminated from effective impact ( ex-post ) – how scientific questions established previously were answered – and the subsequent scientific conclusions translated into practical solutions or political implications.

More than a decade after these seminal documents appeared, little advancement has been seen concerning bringing these proposals into being. With the exception of being able to use search engines (such as Google scholar), which recover bibliographic citations without limiting the search to scientific journals, ³3 . Barreto ML, Aragão E, Souza LEPF, Santana TM, Barata RB. Diferenças entre as medidas do índice-h geradas em distintas fontes bibliográficas e engenhos de busca. Rev Saude Publica. 2013;47(2):231-8. little has been produced in the way of new proposals for scientific evaluation, especially in the applied areas, or mode 2.

If the evaluation of scientific impact, internationally, has been focused on bibliometric indicators, in Brazil this process has taken on some very specific characteristics. There, academic research has been linked to the development of post-graduate courses. Post-graduate programs, enclosed in universities that, if not entirely rejecting them, regarded them as foreign bodies, have become epicenters of scientific investigation activities and sources of prestige in academic activity. Within the systems of scientific evaluation, bibliometric indicators (number of publications, number of citations, impact factor the scientific journals, h-index of the researchers or the journals, among others) have been avidly assimilated into the system of evaluating scientific activities and academics, as in the international scene, also known as “quantified control”. ⁴4 . Burrows R. Living with the h-index? Metric assemblages in the contemporary academy. Sociol Rev. 2012;60(2):355-72. DOI:10.1111/j.1467-954X.2012.02077.x.
https://doi.org/10.1111/j.1467-954X.2012...

The principal system of scientific evaluation in the country is the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – CAPES (Coordination of Improvement of Higher Education Personnel) and the unit of evaluation is post-graduate programs and scientific activities linked to them. Brazil is perhaps the only country in the world to use such an approach. Although this evaluation effort, which began over 30 years ago, has been central in kick starting post-graduate activities and scientific activities on which they are based in the country, the need for substantial changes has arisen, accompanying changes in society’s focus on scientific development, coming to include evaluative parameters such as applications and innovations produced by scientific activity. The current focus based on number of articles, impact factor or journals’ h-index has drawbacks and is unsatisfactory, as has been shown in terms including public demonstrations calling for its exclusion as a criterion for scientific evaluation. ¹1 . Alberts B. Impact factor distortions. Science . 2013;340(6134):787. DOI:10.1126/science.1240319 ^{, 5}5 . Ferreira RC, Antoneli F, Briones MRS. The hidden factors in impact factors:aperspective from Brazilian science. Frontiers in Genetics 2013; 4:130. DOI:10.3389/fgene.2013.00130
https://doi.org/10.3389/fgene.2013.00130...

In Brazil, the need to develop new parameters for scientific evaluation which seek to recognize the contribution made by different modes of doing science, by different communities, institutions and scientific groups to material and non-material advances expected to occur in society has become evident, as has the need to transform this evaluative process into an activity which includes the perspective of the various direct sources of financing scientific investigation, of the institutions which carry it out (universities and research institutes) and of society’s expectations of scientific activities.

For a project of such change to be realized, it also becomes necessary to understand the base upon which the projects and relationships of power within and among the scientific communities in the country are configured. Thus, Barata et al ²2 . Barata RB, Portela LE, Aragão E, Barreto ML. The configuration of the Brazilian scientific field. Ann Acad Bras Cienc. 2013. No prelo. have shown that, based on current evaluation criteria, the area of health care, despite its importance to society and its good performance when compared with other scientific areas using traditional parameters, has been, historically, one of the areas that had the lowest allocation of investment in research. The proposal of constructing a new Brazilian Unified Health System (SUS), in order to fulfill the principle established in the 1988 Constitution that very citizen has a right to health care and that this is a duty of the State, produced a huge political, ⁸8 . Paim J, Travassos C, Almeida C, Bahia L, Macinko J. The Brazilian health system: history, advances, and challenges. Lancet . 2011;377(9779):1778-97. DOI:10.1016/S0140-6736(11)60054-8
https://doi.org/10.1016/S0140-6736(11)60... scientific and technological challenge. ¹¹11 . Victora CG, Barreto ML, Leal MC, Monteiro CA, Schmidt MI, Paim J, et al. Health conditions and health-policy innovations in Brazil: the way forward. Lancet . 2011;377(9782):2042-53. DOI:10.1016/S0140-6736(11)60055-X
https://doi.org/10.1016/S0140-6736(11)60... They include the search for scientifically sound alternatives for a diverse set of activities ranging from diagnosis, cure and prevention of a variety of health problems to the organization of a macrostructure capable of providing comprehensive and equal access to the recourses necessary for the health of the population. These are complex challenges for members of the scientific community in the country, demanding expertise and creativity and, also, production which includes, but is by no means limited to scientific publications.

References

Publication Dates

History