Abstracts

HISTORICAL BACKGROUND

The need to evaluate what has been produced under the aegis of different funding sources is as old as the emergence of Modern Science itself. In its beginnings, evaluation was basically defined as a system of obscure (in the eyes of the public) rewards and vetoes applied to findings that might please or challenge the status quo and the points of view of nobles and priests, as clearly illustrated by Galileo Galilei’s trial. ^aaSuch historical facts will not be analyzed here. Interested readers may consult Galileo Galilei biographies such as the recent one written by John L. Heilbron. Heilbron JL. Galileo. New York: Oxford University Press; 2012. Available from: http://www.amazon.com/Galileo-John-L-Heilbron/dp/0199655987/ref=sr_1_2?s=books&ie=UTF8&qid=1377282006&sr=1-2

The long and winding road between the emergence of Modern Science in Galileo’s times and our own age is characterized by:

1. the definition of people who carry out research as “scientists” as late as 1834 (in the case of the English language) by the Rev. W. Whewell (1794-1866). Whewell was an influential thinker in the era of Charles Darwin (1809-1882) and John Snow (1813-1858), so the very word “scientist” emerged at the same time as modern biology and epidemiology;

2. the gradual replacement of doing research under the aegis (patronage) of a given member/circle of the clergy/nobility or on behalf of a given state ^b b The latter idea was aggressively promoted by the most different dictatorial regimens and peaked on controlling and intolerance in the so-called “Gleichschaltung” [forcible coordination], imposed by the Nazis upon the 3 centuries-old Prussian Academy in June 1939. by the alternative perspective of doing science “supported” by public agencies, understood, to a large extent, as an independent task pursued by individual scientists/research groups;

3. the establishment of societies, institutions and agencies with the explicit aim of promoting science and debating its concepts and findings. One cannot imagine French or British science without institutions like the French Academy of Science, founded in 1666, or the Royal Society, in 1660 (about the latter, see a review of its major achievements in its first 350 years in Bryson). ²2 . Bryson B, editor. Seeing further: the story of science, discovery, and the genius of the Royal Society. London: William Morrow Paperbacks; 2011. Available from: http://www.amazon.com/Seeing-Further-Science-Discovery-Society/dp/0061999776/ref=sr_1_13?s=books&ie=UTF8&qid=1377283447&sr=1-13&keywords=bill+bryson
   http://www.amazon.com/Seeing-Further-Sci...

SCIENCE AFTER WORLD WAR II

Science as we know and practice it – as a large, regularly funded complex set/network of individuals and teams, institutions and dedicated journals, supported by public agencies and private foundations in the context of democratic societies – gained momentum after World War II, in what is usually known as “big science” (a term coined by Alvin Weinberg (1915-2006), an American physicist and science manager).

In Brazil, big science was born with the initiative launched by Brazilian physicists to establish stable, well-endowed institutions to foster science on a “big scale”. ¹1 . Andrade AMR. Físicos, mésons e política: a dinâmica da ciência na sociedade. Rio de Janeiro: Hucitec/Museu de Astronomia e Ciências Afins; 1999. Of course, the practical effects of such a quest for big-scale science in Brazil should not be compared with the huge amounts of money invested in science by governments and foundations from richer countries, such as the US government and US-based foundations. Nevertheless, such efforts resulted in the creation, in 1951, of the Conselho Nacional de Pesquisa (National Research Council), later renamed as Conselho Nacional de Desenvolvimento Científico e Tecnológico (National Council for Scientific and Technological Development).

The systematic support and accreditation of graduate and postgraduate programs dated from the 1970s, when Coordination for Enhancement of Higher Education Personnel (CAPES – Coordenação de Aperfeiçoamento de Pessoal de Nível Superior ), until then defined as a “campaign” ^c c Historical facts about CAPES are available at: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – CAPES. História e missão. Brasília (DF); 2013 [cited 2013 Jul 23]. Available from: http://www.capes.gov.br/sobre-a-capes/historia-e-missao. (what in English would be roughly the equivalent of a taskforce) became a permanent institution and launched its regular evaluations of graduate courses.

WHAT DO EVALUATIONS MEAN, WHAT DON’T THEY

Evaluation of academic work aims to assess the intrinsic value of research from the perspective of peers; to make science and its technological applications and social impacts accountable to governments and tax payers; and to provide guidance for policymaking in any area of societal life which may interact with science & technology (what might be roughly coincident with contemporary life in a broad sense; but see Feynman ⁵5 . Feynman RP. The meaning of it all: thoughts of a citizen-scientist. Nova York: Basic Books; 2005. on what he called our “unscientific” age).

Some of the problems, frustrations and anxiety scholars and students feel respecting evaluation are secondary to (mis-)expectations. For instance, as a teacher and former coordinator of a graduate program, I heard many times that evaluations at the national level should include assessments of individual classes/teachers. Desirable and sorely needed as such specific assessments might be, they are clearly local and focal, and by no means compatible with a comprehensive cross-evaluation of dozens of programs in each given area, nationwide. In this sense, evaluations should be multiple and should take place on different scales, varying from those targeting local concrete experiences to broad national assessments.

THE DISUNITY OF SCIENCE AS A CHALLENGE TO INTEGRATED EVALUATIONS

We are used to speaking of Science with a capital S, as such capitalized Science could muster aims, methods and procedures across the most different disciplines. “Disunity of science” is an expression coined by the American physicist and historian of science Peter Galison ^d d As discussed, among other papers and books in: Galison PL, Stump DJ, editors. The disunity of science: boundaries, contexts, and power. Stanford: Stanford University Press; 1996. to document the fact different disciplines have boundaries, are propelled or hindered by different, idiosyncratic forces, and use methods and procedures tailored to their specific aims.

Brazil has a national evaluation system and a nationwide on-line curricula vitae (CV) platform (unified as the Lattes platform ^e e Full documentation and historical background are available at: Conselho Nacional de Desenvolvimento Científico e Tecnologia – CNPQ. Plataforma Lattes. Brasília (DF); 2013 [cited 2013 Jul 23]. Available from: http://lattes.cnpq.br/ ), unlike other large federative countries, such as the US and Germany. The Brazilian system roughly resembles Canadian funding institutions in terms of profiting from unified, on-line CV platforms. ^f f See, for instance, information at: Canadian Institutes of Health Research. Canadian Common CV: on-line training now available. Ottawa; 2013 [cited 2013 Jul 23]. Available from: http://www.cihr-irsc.gc.ca/e/45622.html But key differences should be highlighted here: Canada is much less populated and heterogeneous (in both social and geographical terms) than Brazil, and its on-line systems cover specific areas, such as research on health. ^f g As described in much detail at: US National Institutes of Health. Grants & funding. Bethesda; 2013 [cited 2013 Jul 23]. Available from: http://grants.nih.gov/grants/electronicreceipt/preparing.htm On the other hand, Canada is a bilingual country, what asks for English and French versions of every single document. Both differ markedly from the topical nature of US-based institutions and agencies (compare, for instance, the specificity and conciseness of the National Institutes of Health “eRA Commons” biosketches ^g h Available at: Sistema Integrado CAPES. WEBQualis. Brasília (DF); 2013 [cited 2013 Jul 23]. Available from: http://qualis.capes.gov.br/webqualis/principal.seam. with Brazilian or Canadian comprehensive CVs).

Brazil’s challenge is enormous and comprises the need to accommodate the demands of the most different sciences and disciplines, varying from the most abstract ones, such as philosophy and theoretical physics, to empirical sciences, such as most research carried out in the field of public health or engineering, as well as properly handle the pronounced heterogeneity of its institutions, society and geographic regions.

FOCUSING ON PUBLIC HEALTH

In a critical review of scientometrics in public health, Coimbra Jr. highlighted its comparatively low impact ( vis-à-vis frontier science) and its contextual and applied nature (his analysis focused on Latin American scientific output). ³3 . Coimbra Jr CEA. Produção científica em saúde pública e as bases bibliográficas internacionais. Cad Saude Publica. 1999;15(4):883-8. DOI:10.1590/S0102-311X1999000400022
https://doi.org/10.1590/S0102-311X199900... The author is right in his criticism and his proposals toward broader and more “applied” criteria of scientific relevance. Not as a coincidence, fierce battles shake every single field of knowledge from time to time over the attribution of “Qualis scores” to journals, as listed at the CAPES webpage. ^h i As described in detail at: Lopes JL. Richard Feynman e a física no Brasil. Rio de Janeiro: Centro Brasileiro de Pesquisas Físicas; 1988 [cited 2013 Jul 23]. Available from: http://cbpfindex.cbpf.br/publication_pdfs/Cs00588.2010_09_09_17_16_11.pdf From my own perspective, any stable consensus regarding such scores would rather reflect coercion than consensus. So, controversy (sometimes harsh) tend to (re)emerge here for ever and ever, but this is the price Brazilian academy will always pay to have a unified national evaluation system across different disciplines, graduate programs, and localities.

In a similar way, on-line CV platforms seem to be condemned to be on the wing, pulled from one side towards conciseness and practicality and from other side towards comprehensiveness and capacity to accommodate different activities and demands. The recent incorporation by CNPq of a brand new module to Lattes platform on the dissemination of science seems to be a compromise between the straightjacket of peer review and the need for scientists to disseminate to the society at large using language as free of jargon and technicalities as possible. The many different books written by Richard Feynman on physics and computation for non-specialists remain paradigmatic in terms of scientific soundness combined with clarity of exposition and conciseness, even when he focused on what he called “not-so-easy” pieces. Of course, to demand from all of us to be as talented and didactic as Feynman would push it too far. But maybe the sharp criticisms he published on teaching and doing physics in Brazil, as he observed first-hand during his sabbatical at the Centro Brasileiro de Pesquisas Físicas (CBPF), in the early 1950s, ⁱ may help us to better integrate academic excellence, academic teaching and the dissemination of science to the public at large.

“I don’t think non peer-reviewed publications (such as reports in newspapers or television, blogs or posts in the web) could ever replace peer-reviewed articles. They should, rather, complement and foster a dynamic dialogue with them. Peer-reviewing is far from perfect, as has been discussed in the most different forums, many of them launched by the editors of peer- reviewed journals themselves. But I do agree with a recent comment by the German researchers and editors Maerthens & Baethge ⁶6 . Mertens S, Baethge C. Standards in the face of uncertainty: peer review is flawed and under-researched, but the best we have. Dtsch Arztebl Int. 2012;109(51-52):900-2. DOI:10.3238/arztebl.2012.0900
https://doi.org/10.3238/arztebl.2012.090... [that] peer-review is “flawed and under-researched, but the best we have”. In the same way, democracy has many flaws and caveats, but for those who once lived in dictatorships it seems to be well defined in Winston Churchill’s famous quotation: “[…] democracy is the worst form of government except all the others that have been tried”.

Examples of proper and improper uses of the most different media in their attempts to disseminate science are so numerous, conflicting and confusing that any attempt to base evaluations on such examples are not likely to reach any valid conclusion or even a provisional agreement. On a personal note, I remember here the positive influence of the Brazilian journal on popular science “Ciência Hoje” ^j j Instituto Ciência Hoje. Rio de Janeiro; 2013 [cited 2013 Jul 23]. Available from: http://cienciahoje.uol.com.br (launched in 1982), from which I have learned so many wonderful things about science (especially about fields I will never have the expertise, time or talent to explore), over many years.

On the negative side, I think any Brazilian citizen minimally concerned with science and their country should read about the defamatory newspaper campaign launched by a powerful group of Academia Nacional de Medicina against Carlos Chagas in the early 1920s. Lacking any compromise with scientific integrity, a group of renowned physicians launched against Chagas – the most distinguished biomedical scientist of that period, as well as against his landmark discovery (later on, named after him as Chagas’ disease) – personal insults as well as biased criticisms. ^k k Biblioteca Virtual Carlos Chagas. Trajetória. Rio de Janeiro: Casa de Oswaldo Cruz; 2013. A polêmica [cited 2013 Jul 23]. Available from: http://www.bvschagas.coc.fiocruz.br/php/trajetoria.php#polemica Such examples, so close to our own efforts as biomedical researchers, is a clear warning that any evaluation should be as comprehensive as possible, always comparing and contrasting different evidence and perspectives. There is no magical solution, no easy fix… as regarding civilization itself (as pointed by Sigmund Freud). Any evaluation will generate as a necessary corollary “its discontents”. ^l l I made a pun here with Freud’s book “Civilization and its Discontents”. The original German title “Das Unbehagen in der Kultur” is rather translated into English as “The Uneasiness in Culture”, but “Civilization and its Discontents” became such a popular title to the point of obfuscating the original one. Freud S. Das Unbehagen in der Kultur. Wien: Internationaler Psychoanalytischer Verlag; 1930.

I would like to conclude this article with an example from a field of research which is situated so far from my own field of expertise and from research published by Revista de Saúde Pública that at first sight it may seem pure nonsense. Anyway, without some unusual ideas science remains stalled.

Living and working in the UK in the period I submitted these comments to Revista de Saúde Pública , I began to read in that same period a book published by the American theoretical physicist Lisa Randall. After reading the accolades that usually embellish the back covers of books and are so many times mistaken, I decided to browse her homepage at Harvard and then entered the Scopus database to double-check what Lisa Randall designated in her CV as “recent and highly cited papers” ^m m Randall L. Curriculum vitae. Cambridge: Harvard University; s.d. [cited 2013 Jul 23]. Available from: http://randall.physics.harvard.edu/CV.html (after all, epidemiologists are a kind of scientifically-endorsed gossipers). In the Scopus database, I realized that, as of May 30, Dr. Randall had been cited by 14,511 indexed papers. ⁿ n The link refers to a consultation performed as of May 30 2013 and may be no longer valid and/or accurate when opened in subsequent moments: http://www.scopus.com.scopeesprx.elsevier.com/cto2/main.url?origin=AuthorProfile&stateKey=CTOF_423753892 Curiously, such a hard-nosed scientist, dealing with abstract concepts and sophisticated mathematics has authored two books on popular science (one of them, the book that launched my quest for additional information [ Knocking on Heaven’s Door ]), contributed to many different newspapers and blogs, radio broadcasts, and – most surprising – wrote a libretto for an Opera. Definitively, Dr. Randall never attended the classic debates which took place repeatedly in our graduate forums, where wars are waged against “the highly cited scientists who have alienated themselves from real life problems”.

I think it’s time to turn old prejudices upside down and realize that old dichotomies (such as the one between “ivory tower intellectuals vs. people who care about life as it really is”) are not as simple as people would like to think. Maybe such overly simplistic contrasts should instead be viewed upside-down, as in the funny sentence from Feynman (who was a bongo drummer and played in Rio de Janeiro’s samba schools), introducing one collection of his lectures on physics: “It is odd, but on the infrequent occasions when I have been called upon to play the bongo drums in a formal place, the introducer never seems to find it necessary to mention I also do theoretical physics.” (Feynman, 1992, p. 13). ⁴4 . Feynman RP. The character of physical law. London: Penguin Books; 1999.

REFERENCES

Publication Dates

History