Abstract

This article aims to contribute to reflection on using and appropriating the term “meta-evaluation” in health research, especially regarding the importance of adopting the controlled vocabulary. It is assumed that the DeCS/MeSH thesaurus is widely used for indexing and retrieving scientific articles in health research. However, the term “meta-evaluation” does not appear as a controlled descriptor option in this database. The text is presented in the form of a theoretical essay, discussing the relevance of evaluative practices and the need to expand and consolidate studies on meta-evaluation in the health area. Including the term "meta-evaluation" in the DeCS/MeSH database is fundamental and proposed.

Keywords:
Meta-evaluation; Health assessment; Descriptors in health sciences; Controlled vocabulary

Introduction

The evaluation of programs and other health interventions plays a fundamental role in decision-making, resource allocation, and continuous improvement of services. In this context, the institutionalization of the evaluation process as a management practice integrated into an organizational system capable of absorbing the results and knowledge generated is essential to consolidate it as a relevant instrument of social control (Hartz, 2012HARTZ, Z. M. A. Meta-avaliação da gestão em saúde: desafios para uma "nova saúde pública". Ciência & Saúde Coletiva, Rio de Janeiro, v. 14, n. 4, p. 832-834, abr. 2012. Disponível em: https://doi.org/10.1590/S1413-81232012000400004. Acesso em: 1 ago. 2023.
https://doi.org/10.1590/S1413-8123201200... ).

However, it is necessary to recognize that the quality and validity of evaluations can vary significantly. Santos . (2021SANTOS, E. M. dos et al. Meta-avaliação: quais padrões de qualidade, por quê, de quem e para quê? Revista Brasileira de Avaliação, v. 10, n. 2, e101721, 2021. https://doi.org/10.4322/rbaval202110017
https://doi.org/10.4322/rbaval202110017... ) highlight that the concern with the quality of evaluations stems not only from the resources and efforts they require but also from the influence of their results on the validation of strategic choices and the need to ensure that they are conducted according to the best research practices.

Evaluators have been seeking to establish standards of legitimacy, technical-scientific quality, and ethics for evaluative studies since the early 1960s. In this context, the concept of meta-evaluation, according to Scriven (1991SCRIVEN, M. Evaluation thesaurus. 4. th. California: Sage, 1991.), emerges as a systematic approach to evaluate evaluations through the description, judgment, and synthesis of the functions and practices of an evaluation using previously established and validated criteria and standards. A similar definition is proposed by Stufflebeam (2001), describing it as a process of describing, judging, and synthesizing a study or any other evaluative procedure based on proposed and validated standards and criteria by professional associations and other auditing bodies. It is noteworthy that the expression "evaluate the evaluation," coined and popularized by Scriven in 1969, was previously proposed by the educator Pedro Orata in the 1940s when discussing the terminological transition from test and measure to evaluation at the time (Furtado, 2022FURTADO, J. P. Por uma meta-avaliação sem metafísica. Revista Brasileira de Avaliação, v. 11, n. 2, e112022, 2022. DOI: https://doi.org/10.4322/rbaval202211020
https://doi.org/10.4322/rbaval202211020... ).

From a practical standpoint, the institutionalization of meta-evaluation enables organizations to implement and govern ethical, effective, and efficient evaluation systems. Meta-evaluation is the professional imperative of evaluation; it demonstrates that evaluation is self-referential and transcends mere application to third parties (Scriven, 2018).

Thus, meta-evaluation is a field of study dedicated to the systematic and critical evaluation of existing evaluations, which must be conducted based on predefined procedures or criteria to collect, analyze, and interpret data in an organized and methodological manner. Therefore, it is not based on subjective impressions or casual observations but on a structured and planned reflective process, including questioning underlying positions, identifying potential biases, analyzing logical consistency, and the strength of the evidence presented. This pursuit of developing solid references, guides, standards, and norms to guide evaluative practice has been increasingly incorporated and developed in global, regional, and national organizations (Silva ., 2020SILVA, R. R.; JOPPERT, M. P.; GASPARINI, M. F. V. (org.). Diretrizes para a prática de avaliação no Brasil (livro eletrônico). 1. ed. Rio de Janeiro: Rede Brasileira de Monitoramento e Avaliação, 2020. Disponível em: Diretrizes-para-a-pratica-de-avaliacao-no-Brasil_20ago20.pdf (rbma.site); Furtado, 2022FURTADO, J. P. Por uma meta-avaliação sem metafísica. Revista Brasileira de Avaliação, v. 11, n. 2, e112022, 2022. DOI: https://doi.org/10.4322/rbaval202211020
https://doi.org/10.4322/rbaval202211020... ).

Hartz, Rehem and Craveiro (2021HARTZ, Z. M. A.; REHEM, T. C. S. B.; CRAVEIRO, I. Meta-avaliação em tempos de pandemia: para que servem as intervenções da covid-19? In: SANTOS, A. O.; LOPES, L. T. (Org.). Principais elementos. Brasília: Conselho Nacional de Secretários de Saúde, 2021. p. 168-180. (Coleção Covid-19, v. 1). Disponível em: https://www.conass.org.br/biblioteca/volume-1-principais-elementos/. Acesso em: 29 jun. 2023.
https://www.conass.org.br/biblioteca/vol... ) recommend encouraging the use of meta-evaluation as a tool to improve public health to prevent inconsistent or biased evaluations from negatively influencing managers' decisions. However, Hartz (2008) emphasizes the importance of not underestimating the complexity of meta-evaluation despite the apparent simplicity of definitions about its realization; he emphasizes the need for careful reflection and thorough investigation of the academic and functional practices involved. Malta et al. (2019), for example, offer a methodological contribution to the improvement of systematic meta-evaluative studies in four stages: planning the meta-evaluation; selecting and compiling the material to be meta-evaluated; analyzing the quality of studies; and synthesizing the evaluation's conclusions.

In Brazil, according to Santos . (2021SANTOS, E. M. dos et al. Meta-avaliação: quais padrões de qualidade, por quê, de quem e para quê? Revista Brasileira de Avaliação, v. 10, n. 2, e101721, 2021. https://doi.org/10.4322/rbaval202110017
https://doi.org/10.4322/rbaval202110017... ), meta-evaluation is influenced by four important references: the guidelines of the Organisation for Economic Cooperation and Development (OECD/DAC), the standards established by the Joint Committee on Standards of Educational Evaluation (JCSEE), the Evaluation Guidelines for Latin America and the Caribbean (DALAC), and the Guidelines for Evaluation Practice in Brazil (DPAB). However, these references do not establish conceptual or methodological standards among themselves - including the Brazilian Monitoring and Evaluation Network highlights that the guidelines for evaluation in Brazil (DPAB) do not directly address the concepts of evaluability and meta-evaluation (Silva ., 2020SILVA, R. R.; JOPPERT, M. P.; GASPARINI, M. F. V. (org.). Diretrizes para a prática de avaliação no Brasil (livro eletrônico). 1. ed. Rio de Janeiro: Rede Brasileira de Monitoramento e Avaliação, 2020. Disponível em: Diretrizes-para-a-pratica-de-avaliacao-no-Brasil_20ago20.pdf (rbma.site)).

The conceptual and methodological diversity poses an additional challenge for the progress of scientific studies on meta-evaluation. To promote a deeper understanding of the topic and drive the development of studies in meta-evaluation, it is essential to consolidate the epistemological foundations and premises of scientific research in the health domain. In this context, Dodebei (2002DODEBEI, V. L. D. L. M. Tesauro: linguagem de representação da memória documentaria. Rio de Janeiro: Intertexto, 2002.) highlights that indexing languages are crucial for representing the content of informational resources in information organization and retrieval systems.

Thesauri, one of the most classical indexing languages, play a fundamental role in terminological control; their function is to translate natural language into controlled language, making the indexing service and the accuracy of search results more effective (Curras, 1995CURRÁS, E. Tesauros, linguagens terminológicas. Trad.: Antônio Felipe Corrêa da Costa. Brasília: IBICT, 1995.). Gomes (1990GOMES, H.E. (coord.). Manual de elaboração de tesauros monolíngues. Brasília: Programa Nacional de Bibliotecas de Instituições de Ensino Superior, 1990.) defines the thesaurus as a documentary and dynamic language that contains terms semantically and logically related, covering a domain of knowledge. Thus, the controlled vocabulary plays a fundamental role in scientific research. If the publication does not use descriptors that align with the nomenclature of the databases, it risks not being found and consequently little cited, valued, recognized, and promoted (Brandau; Monteiro; Braile, 2005BRANDAU, R.; MONTEIRO, R.; BRAILE, D. M. Importância do uso correto dos descritores nos artigos científicos. Braz J Cardiovasc Surg., São Paulo, v. 20, n. 1, jan 2005. Disponível em: https://doi.org/10.1590/S0102-76382005000100004. Acesso em: 10 ago. 2023.
https://doi.org/10.1590/S0102-7638200500... ).

In this sense, this essay aims to discuss the importance and benefits of incorporating the term "meta-evaluation" as a controlled descriptor, considering its benefits for the field of meta-evaluative research in health. The aim is to argue about the challenges in defining an assertive search strategy in research development as well as the importance of structured vocabulary for meta-evaluation research given the diversity of previously mentioned concepts, proposals, and methodological criteria. Finally, after careful analysis of the structure of the main health thesaurus, it is proposed to include and categorize the term in four hierarchical branches of the DeCS/MeSH conceptual tree.

It should be noted that an essay, according to Meneghetti (2011MENEGHETTI, F. K. O que é um ensaio-teórico? Revista de Administração contemporânea, João Pessoa, v. 15, n. 2, p. 320-332, mar. 2011. Disponível em: https://www.scielo.br/j/rac/a/4mNCY5D6rmRDPWXtrQQMyGN/?lang=pt. Acesso em: 1 ago. 2023.
https://www.scielo.br/j/rac/a/4mNCY5D6rm... ), is characterized by its reflective and interpretive nature, different from the classificatory form of science. Therefore, the intention is to foster reflection and debate without pretending to exhaust the topic.

Challenges in defining an assertive search strategy

The dissemination of scientific information plays a crucial role in advancing science, as it is through the interaction between researchers and society that knowledge progresses. This knowledge, in turn, is built on a foundation of pre-existing knowledge. Thus, the need arises to communicate research results in a way that establishes a solid basis for future scientific investigations (Gäal; Martins, 2022GÄAL, L. P. M.; MARTINS, M.S. Acesso aberto no contexto da pesquisa em Ciência da Informação. Transinformação, v. 34, e220016, 2022. DOI: https://doi.org/10.1590/2318-0889202234e220016. Acesso em: 4 mar. 2024.
https://doi.org/10.1590/2318-0889202234e... ). Moreover, the volume of available information and studies grows exponentially. When searching for answers to a scientific question, it is common to encounter a myriad of results pointing in various directions, sometimes even antagonistic. Additionally, the researcher faces concerns beyond the content of their study, such as formatting, document normalization, and especially the appropriate selection of descriptors.

According to Latorraca . (2019LATORRACA, C. O. C. et al. Busca em bases de dados eletrônicas da área da saúde: por onde começar. Diagnóstico & Tratamento, São Paulo, v. 24, n. 2, p. 59-63, 2019. Disponível em: https://docs.bvsalud.org/biblioref/2019/09/1015338/rdt_v24n2_59-63.pdf. Acesso em: 1 ago. 2023.
https://docs.bvsalud.org/biblioref/2019/... ), the process of developing a search strategy for health-related databases can be structured in four initial steps: identifying the structured question, choosing the database, choosing and using descriptors, and choosing and using Boolean operators. Mastering these steps is fundamental to creating an appropriate search strategy capable of retrieving the studies of interest and providing support to answer the proposed question. Thus, descriptors are terms that allow a more precise finding of what is being sought in the databases. The correct choice of these terms is essential for a review to be representative of the existing knowledge on the intended topic.

Brandau, Monteiro, and Braile (2005BRANDAU, R.; MONTEIRO, R.; BRAILE, D. M. Importância do uso correto dos descritores nos artigos científicos. Braz J Cardiovasc Surg., São Paulo, v. 20, n. 1, jan 2005. Disponível em: https://doi.org/10.1590/S0102-76382005000100004. Acesso em: 10 ago. 2023.
https://doi.org/10.1590/S0102-7638200500... ) highlight the importance of differentiating keyword and descriptor, arguing that the former does not follow any structure and is conceived from free language. They further emphasize that for a keyword to become a descriptor, it must undergo strict control of synonyms, meaning, and importance in the tree of a particular subject.

It is observed that the correct use of controlled descriptors aids in the precision and standardization of health research, facilitating the retrieval of relevant information and contributing to the quality of research and decision-making in the area. Therefore, constructing a balanced search strategy in terms of sensitivity and specificity becomes more challenging in the absence of controlled descriptors, requiring the use of similar terms.

In this regard, the challenge faced during the development of a protocol for a scoping review on meta-evaluation in health is highlighted. It was identified that the term “meta-evaluation” is not present as a controlled term in the DeCS/MeSH systems, which are widely used to index and categorize information in the health area (Cagliari ., 2023CAGLIARI, E. B.; SOUSA, N. P.; CRAVEIRO, I.; REHEM, T. C. M. S. B. Overview of meta-evaluation studies in Health: a scoping review protocol. OSF Home, Virginia, 2023. Disponível em: https://doi.org/10.17605/OSF.IO/YW6EQ. Acesso em: 1 ago. 2023.
https://doi.org/10.17605/OSF.IO/YW6EQ... ). The fact that the term is not a controlled descriptor required the research team to search for approximate and/or related terms that had also been used in articles and bibliographic references previously researched to construct the best search strategy.

This led to the realization of two relevant scenarios for discussion: the broad and unspecific search for approximate and/or related terms (such as “health evaluation,” “evaluation studies as topic,” “program evaluation”) resulted in numerous unspecific studies due to the breadth and lack of specificity of the terms. In reading some scoping review works on meta-evaluation, a large number of identified works were observed, but the inclusion of studies for effective analysis was very low. In the scoping review on meta-evaluation in Primary Heath Care, for example, the researchers began their search with 11,641 studies and ended up analyzing only 23 as eligible (Bay Junior et al., 2022).

On the other hand, the focused search for the specific term “meta-evaluation” or “metaevaluation” resulted in more specific studies, although there is a risk of missing studies not indexed with this “keyword.” Additionally, another aspect was highlighted: the terms meta-evaluation and meta-analysis are used as synonyms by some health researchers. Scriven (2009) warns of this risk and emphasizes the need to avoid the mistake of confusing meta-evaluation with meta-analysis. According to him, meta-analysis is a statistical technique conceived and introduced by Gene Glass in the 1970s, applicable only to a set of quantitative studies (which may or may not be evaluative) by synthesizing their results in terms of statistical significance. Meta-evaluation, on the other hand, only applies to evaluations, being able to evaluate an evaluation that is entirely qualitative.

It is believed that both scenarios would be better adjusted if the term “meta-evaluation” became a term in the indexing base. A broad and unspecific search would not be necessary, nor would the term be underutilized or used inappropriately. The indexing of the term would contribute to defining a more assertive search strategy and consequently in the recovery and appreciation of meta-evaluative research in the health field.

The importance of structured vocabulary for meta-evaluation research

As previously discussed, the search for scientific information available in the literature can become unproductive or confusing without a basic understanding of how knowledge is organized or indexed.

An essential characteristic of scientific research is its temporal progression; each advance is based on previously accumulated knowledge, forming a complex network of interconnected information that coherently defines the structure of knowledge. Thus, constructing a specific vocabulary is not a mere accessory in science. This terminology plays a crucial role in scientific advancement, facilitating a more efficient and precise exchange of information. According to Dias (2000DIAS, C. A. Terminologia: conceitos e aplicações. Ciência da Informação, v. 29, n. 1, p. 90-92, abr. 2000.), terminology represents technical-scientific knowledge in an organized manner through manuals and glossaries and unifies this knowledge through norms and standards. Without adequate terminology, specialists would face difficulties in communicating, transmitting, and organizing this knowledge. However, a critical analysis of the control and management of scientific information reveals a complex scenario permeated by power dynamics and interests.

The origin of thesauri is linked to the informational demands of the political, social, and technological changes of the 1950s. Standardizing descriptor terms and creating relationships between concepts were crucial for facilitating the location and retrieval of information. The creation of thesauri was influenced by historical, cultural, and ideological factors, highlighting two main theoretical strands: one originating in North America with an alphabetical approach and another influenced by the European tradition of bibliographic classification (Lancaster, 1968).

According to Miranda, Medeiros, and Sujii (1990MIRANDA, L. M. C. de; MEDEIROS, M. B. B.; SUJII, M. K. Elaboração de tesauros utilizando-se o programa de elaboração de tesauros em microcomputador (Tecer). Revista de Biblioteconomia De Brasília, v. 18, n. 2, p. 185-192, 1990. Disponível em: https://periodicos.unb.br/index.php/rbbsb/article/view/46693. Acesso em: 4 mar. 2024.
https://periodicos.unb.br/index.php/rbbs... ), the conceptual or terminological thesaurus is a specialized language composed of linguistic terms that enables the standardized description of subjects in documents in information retrieval systems in accordance with international rules and standards, including the definition of semantic relationships between concepts. In the health area, the DeCS (Health Sciences Descriptors) and MeSH (Medical Subject Headings) thesauri are important tools used in indexing and information retrieval, playing a fundamental role in organizing and standardizing the terms used in specialized scientific literature. DeCS was created in 1986 by Bireme based on MeSH, which emerged in 1963 and is produced by the U.S. National Library of Medicine (Pellizzon, 2004PELLIZZON, R. F. Pesquisa na área de saúde. 1. base de dados DeCS (Descritores em Ciências da Saúde). Acta Cirúrgica Brasileira, São Paulo, v. 19, n. 4, p. 153-163, 2004. Disponível em: https://red.bvsalud.org/decs/wp-content/uploads/sites/4/2016/11/Pellizzon-Pesquisa-na-%C3%A1rea-da-sa%C3%BAde-Base-de-dados-DeCS.pdf. Acesso em: 1 ago. 2023.
https://red.bvsalud.org/decs/wp-content/... ).

Figueira (2018FIGUEIRA, C. S. Ideologia e discursos sobre trabalho, educação e saúde nos descritores em ciências da saúde. Trabalho, Educação e Saúde, [S. l.], Rio de Janeiro, v. 16 n. 3, p. 1.337-1.360, set./dez. 2018. DOI: http://dx.doi.org/10.1590/1981-7746-sol00148.
http://dx.doi.org/10.1590/1981-7746-sol0... ) emphasizes that the documentary language of DeCS is also a political and ideological object, disseminating a way of thinking about the field of work, education, and health to the health sciences area. In other words, DeCS, by attributing synonymy to certain terms, influences the conceptions and interpretations in dispute in society, configuring itself as a linguistic-ideological instrument. In this sense, a careful and propositional analysis of the bases is essential to contribute to its evolution and adequacy over time.

According to DeCS, a new category can be established in situations involving a new area of knowledge, an area where DeCS terminology is inadequate, or an area where terms are dispersed among existing categories. Thus, the purpose of adopting the term “meta-evaluation” as a controlled descriptor is multifaceted and aims to achieve significant benefits in conducting meta-evaluative research.

Firstly, the inclusion of “meta-evaluation” as a controlled descriptor seeks to promote terminology standardization, ensuring that the terms used by researchers are uniform. This is especially relevant in multidisciplinary areas where different terminologies may be used to describe similar concepts. Another important aspect is the precise retrieval of information: using controlled descriptors saves time and effort in research, increasing efficiency in information retrieval and mitigating selection bias. Additionally, they are generally organized in a hierarchical structure, where broader terms are subdivided into more specific terms, allowing hierarchical and relational exploration of concepts and enabling more precise and comprehensive investigation. Furthermore, controlled descriptors tend to be interrelated, facilitating the exploration of related concepts. Finally, the use of these descriptors contributes to the comparability and replicability of studies - by providing precise information about the terms used in the research, controlled descriptors enable other researchers to replicate and compare studies. This aspect is essential for the validation and replicability of research as well as for building a solid and reliable body of knowledge over time (Fujita; Tolare, 2019FUJITA, M. S. L.; TOLARE, J. B. Vocabulários controlados na representação e recuperação da informação em repositórios brasileiros. Informação & Informação, [S. l.], v. 24, n. 2, p. 93-125, 2019. DOI: 10.5433/1981-8920.2019v24n2p93.
https://doi.org/10.5433/1981-8920.2019v2... ; Cruz , 2022CRUZ, M. C. A.; FERNEDA, E.; FUJITA, M. S.L. A disponibilização de vocabulário controlado aos usuários para a recuperação da informação. Revista Ibero-Americana de Ciência da Informação, [S. l.], v. 15, n. 1, p. 266-282, 2022. DOI: 10.26512/rici.v15.n1.2022.42464.
https://doi.org/10.26512/rici.v15.n1.202... ).

In summary, controlled descriptors play a crucial role in scientific research by facilitating precise information retrieval, standardizing terminology, enabling hierarchical and relational exploration of concepts, supporting systematic reviews, and promoting the comparability and replicability of studies. Thus, the importance of adopting structured vocabulary for meta-evaluation research is reinforced once again.

Meta-Evaluation as a DeCS/MeSH Descriptor - A Categorization Proposal

At this point, it is necessary to minimally detail the structure of the DeCS/MeSH thesaurus, which is composed of a vast vocabulary of terms organized in a hierarchy and related to each other, available in four languages (English, Portuguese, Spanish, and French).

According to the DeCS website, the DeCS/MeSH hierarchical tree is organized into three main levels: broad category, subcategory, and descriptor. Broad categories are the main topics that group related descriptors. Each category covers a specific set of subjects related to health, represented by letters. Subcategories are one level below broad categories and serve to organize descriptors more specifically within each category, represented by specific numbering. Descriptors are the specific terms that represent concepts and subjects within each subcategory, possessing a unique number in MeSH/DeCS.

This hierarchical structure allows descriptors to be systematically organized, facilitating navigation and search for specific information at different levels of detail. Broader descriptors are found in broad categories, while more specific ones are located in the lower levels of subcategories and descriptors. This makes MeSH/DeCS a powerful tool for indexing and accessing information in the health area in a structured and efficient manner.

In addition to the hierarchy, DeCS/MeSH uses relationships between terms such as synonyms and related terms to improve search precision. Synonyms allow different words to be used to represent the same concept, while related terms indicate concepts that are related but not direct synonyms.

The DeCS/MeSH thesaurus is dynamic and updates its base annually, allowing suggestions for the inclusion of new terms through a form available on its website under the "Suggest a new term" tab titled "Form to suggest the creation of new terms to DeCS." Thus, the steps outlined in the form were followed to consolidate the proposal of the term meta-evaluation. Highlights of the form's items include the following aspects:

Each broad category was analyzed in its subdivisions and concepts to seek the best hierarchy for the term meta-evaluation. In this sense, the following alternatives were identified as shown in Table 1:

As the thesaurus allows the indication of one or more hierarchical branches of the DeCS conceptual tree in which the new term should be inserted, the option was to indicate inclusion in the four above-mentioned suggestions.

Conclusion

Given the relevance and critical reflection on evaluative practices, combined with the need for expansion and consolidation of studies on meta-evaluation in the health area, it is considered fundamental to include the term "meta-evaluation" in the DeCS/MeSH thesaurus.

It is believed that including the term in four hierarchical branches of the DeCS/MeSH thesaurus will bring greater visibility to meta-evaluative studies, allowing for better accuracy in the adoption of terms by researchers, optimizing search strategies, and contributing to the appropriate application of the concept. It is expected that in the next update of the thesaurus, scheduled for 2024, the term will be incorporated and made available as requested.

Moreover, it is important that other meta-evaluators also suggest the inclusion of the term in the DeCS/MeSH thesaurus and reinforce the importance of this step - fundamental to qualifying searches and expanding meta-evaluation studies in health.

References

Publication Dates

History