Abstract

The need for universal access to health care and the failure of the pedagogical model centered on only the transmission of knowledge has led to changes in the training of health professionals. The aim of this study was to provide a new alternative for evaluation of Dentistry courses based on the National Curriculum Guidelines (NCG) of the area. For this, a logical model was formulated on the need for training in the oral health pathway which allowed for the construction of a criteria matrix, validated by Delphi consensus technique and modified by the participation of 33 “experts.” The matrix dimensions presented as a pedagogical approach the profile of graduates, the teaching-service integration and orientation of health care. The detailing of these dimensions into sub-dimensions and of measurable criteria allowed for deepening structural elements of the NCG unexplored in other studies evaluating undergraduate courses in Dentistry. The final instrument proposed in this study is differentiated alternative assessment training, for both dentists and other professionals, considering that the NCG of all healthcare courses provide for the training focused on the health needs of the population, integrated to Unified Health System (SUS) and based on student-centered learning.

Keywords
Dental education; Educational assessment; Human resources in health

Introduction

Several aspects contribute to the current findings, although emerging/developing, that are changing in the training of health professionals. Among them the need for universal access to health care may be cited or discredited on a purely assistant model and focused on procedures, as well as the finding that the pedagogical model centered only on the transmission of knowledge has failed.

In discussing today the differences between human resources and health workers, a number of questions related to how the education of health professionals has overcome the tensions in its structure involving work, the worker and society in favor of developments in the health system, the flexibility of labor relations and the development of social policies¹1 Assunção AA, Belisário SA, Campos FE, D’Ávila LS. Recursos humanos e trabalho em saúde: os desafios de uma agenda de pesquisa. Cad Saude Publica 2007; 23(Supl. 2):S193-S201.. In view of these paradigm shifts, consequent demands for investigations have been proposed, with the perspective of elucidating phenomena employment/ work and education¹1 Assunção AA, Belisário SA, Campos FE, D’Ávila LS. Recursos humanos e trabalho em saúde: os desafios de uma agenda de pesquisa. Cad Saude Publica 2007; 23(Supl. 2):S193-S201.. In research with students of Dentistry, Medicine and Psychology, an imaginary liberal-privatized operation became evident, articulated by its role in the public sphere simply for greater professional experience²2 Ceccim RB, Armani TB, Oliveira DLLC, Bilibio LF, Moraes M, Santos ND. Imaginários da formação em saúde no Brasil e os horizontes da regulação em saúde suplementar. Cien Saude Colet 2008; 13(5):1567-1578..

The reference that differs this movement and incorporates an expanded dimension in health is translated in the National Curriculum Guidelines (NCG), instituted in undergraduate courses from 2001³3 Brasil. Ministério da Educação (MS). Conselho Nacional de Educação. Câmara de Educação Superior. Parecer nº 583 de 04 de abril de 2001. Aprova a orientação para as diretrizes curriculares dos cursos de graduação. Diário Oficial União 2001; 29 out.. By analyzing the difficulties of dentists in public health services, Costa and Araújo⁴4 Costa ICC, Araújo MNT. Definição do perfil de competências em saúde coletiva a partir da experiência de cirurgiões-dentistas atuantes no serviço público. Cien Saude Colet 2011; 16(Supl. 1):1181-1189. suggest the need for changes in curricula, forming health professionals who reflect on their practice and develop competencies of learning, know-how and behaviour.

The NCG reinforce the approach between Education and Health, with emphasis on promotion, prevention, recovery and rehabilitation, and training of professionals capable of working with quality, efficiency and resolution, consistent with the terms of the construction of the SUS, which projects this system as a training order for human resources in health⁵5 Brasil. Constituição da República Federativa do Brasil de 1988. Diário Oficial da União 1988; 5 out.. For both, it points to the challenge of promoting autonomous and permanent intellectual and professional development in the student, competence that does not end at the completion of course⁴4 Costa ICC, Araújo MNT. Definição do perfil de competências em saúde coletiva a partir da experiência de cirurgiões-dentistas atuantes no serviço público. Cien Saude Colet 2011; 16(Supl. 1):1181-1189..

With the establishment of the NCG and curriculum changes taking place in higher education institutions, it was necessary to develop evaluation processes which suggest an analysis of how the guidelines were met, with regard to the quality, relevance and suitability of courses to the development needs of country⁶6 Haddad AE, Pierantoni CR, Ristoff D, Xavier IM, Giolo J, Silva LB, organizadores. A trajetória dos cursos de graduação na área da saúde: 1991-2004. Brasília: Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira; 2006.. This review is benchmarked on the orientation of authorization procedures and recognition of courses. However, in addition to aspects related to the regulatory process, studies are needed to further investigate essential elements in the everyday development of these institutions.

In 2006, the Teixeira National Institute for Educational Studies (INEP) published a study on the trajectory of undergraduate courses in health. It sought to stimulate reflection and discussion on evaluating the training of upper-level courses in health from the Census of Higher Education, the Teacher's Registry, the Register of Institutions and Courses, the National Survey of Student Performance (ENADE) and Audit Reports of Evaluation Commission (CPA)⁶6 Haddad AE, Pierantoni CR, Ristoff D, Xavier IM, Giolo J, Silva LB, organizadores. A trajetória dos cursos de graduação na área da saúde: 1991-2004. Brasília: Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira; 2006.. It found a significant increase in the availability of Dentistry courses throughout the country, mainly in the private sector.

In 2006, the Ministry of Health and of Education investigated the adoption of the National Curriculum Guidelines in the context of the Pedagogical Projects Course (PPC)⁷7 Brasil. Ministério da Saúde (MS). Ministério da Educação (ME). A aderência dos cursos de graduação em enfermagem, medicina e odontologia às diretrizes curriculares nacionais. Brasília: MS; 2006. based on the reports of the evaluation processes of these PPC courses, conducted by INEP in the period 2002-2006. According to this study, the NCG for Dentistry courses signified an important step forward by establishing the principles and rationale for the formation of the dental surgeon.

However, even after more than ten years since the establishment of NCG and the beginning of the curriculum changes in undergraduate Dentistry, several questions remain about the true effectiveness of the guidelines in the training of dentists. Therefore, the need to think of criteria that can verify how the training of health professionals has responded to the recommendations by the NCG emerges, even in the face of subjectivities and scope of factors for proper training to meet the health needs of the population. Consistent with this need, the objective of this study was to propose the construction of a logical model for the formation of dentists based on National Curriculum Guidelines and from this, the construction of an evaluation criteria matrix and its validation, providing a new alternative for evaluation of undergraduate courses in dentistry.

Methodology

In evaluation studies, considering the complexity of the object, it is necessary, notwithstanding the methodological rigor, to use flexibility, triangulation of research approaches, and complementarity of quantitative and qualitative techniques⁸8 Januzzi PM. Avaliação de programas sociais no Brasil: repensando práticas e metodologias das pesquisas avaliativas. Planejamento e políticas públicas 2011; (36):251-275..

This study consisted of a pre-evaluation or assessment-ability, which arise as a possibility to verify the extent to which an intervention can be evaluated⁹9 Bezerra LCA, Alves CKA, Reis YAC, Samico I, Felisberto E, Carvalho ALB, Silva GS. Identificação e caracterização dos elementos constituintes de uma intervenção: pré-avaliação da política Participa SUS. Cien Saude Colet 2012; 17(4):883-900.. It provides a favorable environment to the extent that builds understanding among stakeholders about the nature and goals of the evaluation objective, seeks agreement between the interests of the assessment and potential users of the study, thereby increasing the possibilities of using results¹⁰10 Mendes MFM, Cazarin G, Bezerra LCA, Dubeux LS. Avaliabilidade ou pré-avaliação de um programa. In: Samico A, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em Saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010. p. 57-64.. For this study, the pre-evaluation consisted of the following steps: construction of the logical model, the development of an evaluation criteria matrix, and validation of the criteria matrix.

Construction of the Logical model

The representation of the research objective, which in the case of this study is to train dentists based on the NCG, their movements, and their relationships, is likely to be modeled with reference to a building theory¹¹11 Medina MG, Silva GAP, Aquino R, Hartz ZMA. Uso de modelos teóricos na avaliação em saúde: aspectos conceituais e operacionais. In: Hartz ZMA, Silva LMV, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. p. 65-102..

The construction of the model began with research and reading documents related to the objective, from NCG³3 Brasil. Ministério da Educação (MS). Conselho Nacional de Educação. Câmara de Educação Superior. Parecer nº 583 de 04 de abril de 2001. Aprova a orientação para as diretrizes curriculares dos cursos de graduação. Diário Oficial União 2001; 29 out.,⁴4 Costa ICC, Araújo MNT. Definição do perfil de competências em saúde coletiva a partir da experiência de cirurgiões-dentistas atuantes no serviço público. Cien Saude Colet 2011; 16(Supl. 1):1181-1189.,¹²12 Brasil. Ministério da Educação. Conselho Nacional de Educação. Câmara de Educação Superior. Resolução 3, de 19 de fevereiro de 2002. Institui Diretrizes Curriculares Nacionais do Curso de Graduação em Odontologia. Diário Oficial União 2002; 4 mar., publications from the Ministries of Health and Education⁶6 Haddad AE, Pierantoni CR, Ristoff D, Xavier IM, Giolo J, Silva LB, organizadores. A trajetória dos cursos de graduação na área da saúde: 1991-2004. Brasília: Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira; 2006.,⁷7 Brasil. Ministério da Saúde (MS). Ministério da Educação (ME). A aderência dos cursos de graduação em enfermagem, medicina e odontologia às diretrizes curriculares nacionais. Brasília: MS; 2006.,¹³13 Brasil. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira – Inep. Diretoria de Avaliação da Educação Superior – Daes. Sistema Nacional de Avaliação da Educação Superior – Sinaes. Instrumento de Avaliação de Cursos de Graduação presencial e a distância. Brasília: maio 2012. [acessado 2013 jun 13]. Disponível em: http://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2012/instrumento_com_ alteracoes_maio_12.pdf
http://download.inep.gov.br/educacao_sup... ,¹⁴14 Brasil. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. SINAES – Sistema Nacional de Avaliação da Educação Superior: da concepção à regulamentação. 5ª ed. revista e ampliada. Brasília: Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira; 2009., the Brazilian Association of Dental Education¹⁵15 Morita MC, Kriger L, Carvalho ACP, Haddad AE. Implantação das diretrizes curriculares nacionais em odontologia. Maringá: Dental Press, Abeno, Opas, MS; 2007., and other studies and publications on the assessment of undergraduate education and dentistry¹⁶16 Lampert JB. Avaliação institucional nos cursos de graduação da área da saúde: avaliar o quê e para quê? Cadernos ABEM 2009; 5:45-55.–²²22 Carvalho ACP, Kriger L. Educação Odontológica. São Paulo: Artes Médicas; 2006..

The construction of a theoretical / logical model needs to be convincing and present plausibility of established associations¹¹11 Medina MG, Silva GAP, Aquino R, Hartz ZMA. Uso de modelos teóricos na avaliação em saúde: aspectos conceituais e operacionais. In: Hartz ZMA, Silva LMV, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. p. 65-102.. In this perspective, based on the literature and research objectives, several possible elements of the model were raised so that this could become a ‘conceptual schematic representation of a thing or real situation or perceived as such’²³23 Bunge M. Os conceitos de modelo. In: Bunge M. Teoria e realidade. São Paulo: Perspectiva; 1974. p. 11-40.. In this case, it could as closely as possible represent the training of dentists according to what is recommended in the NCG.

The logical model therefore presented the need for training as a structuring element linked to the shifting paradigm facing health; the health needs of the population (Figure 1).

Development of matrix Criteria

Based on the logical model constructed, a matrix of evaluation criteria was developed covering the main issues to be addressed from the theoretical framework. These were considered relevant to the formation of a “objective image” assessment of Dentistry courses based on the NCG. For the construction of the matrix, each structural element highlighted in the logical model was transformed into a dimension of evaluation. The matrix was then built on four dimensions: Profile of Graduated, Health Care Orientation, Service-Learning Integration and Pedagogical Approach. Still based on the logical model, sub-dimensions describing evaluation criteria that express the expected objective image standard were proposed for each of these dimensions.

The matrix drawn from the logical model, with dimensions, sub-dimensions, criteria and their respective scores initially proposed by researchers is expressed in Chart 1.

Validation of the criteria matrix by consensus among “experts” (modified Delphi method)

To validate the evaluation criteria of dentistry courses proposed by the authors, an adaptation of the matrix technique of consensus among “experts” (Delphi method) was used.

The Delphi method is a technique for formal consensus among “experts” on a particular topic using their theoretical knowledge, experience and creativity converted to scientific information through a methodologically structured application²⁴24 Esher A, Santos EM, Magarinos-Torres R, AzeredoTB. Construindo critérios de julgamento em avaliação: especialistas e satisfação dos usuários com a dispensação do tratamento do HIV/AIDS. Cien Saude Colet 2012; 17(1):203-214.,²⁵25 Valdés MG, Marín MS. Empleo del método Delphi en investigaciones sobre salud publicadas en revistas científicas cubanas. Rev Cubana de Información en Ciencias de la Salud 2013; 24(2):133-144.. In health, this technique has been used to obtain consensus for various purposes, such as technology assessment, selection of quality of service indicators, and the development of curricula for medical education²⁵25 Valdés MG, Marín MS. Empleo del método Delphi en investigaciones sobre salud publicadas en revistas científicas cubanas. Rev Cubana de Información en Ciencias de la Salud 2013; 24(2):133-144..

The Delphi consensus consists of a structured process in rounds of consultations with experts in order to obtain a collective and qualified opinion on certain issues until a consensus is reached²⁶26 Fonsêca GS, Junqueira SR. Programa de Educação pelo Trabalho para a Saúde da Universidade de São Paulo (Campus Capital): o olhar dos tutores. Cien Saude Colet 2014; 19(4):1151-1162.,²⁷27 Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. Plos One 2011; 6:e20476.

For this study, we opted for an adaptation called “modified Delphi,” which is the inclusion of at least one round of face-to-face discussion between specialists²⁵25 Valdés MG, Marín MS. Empleo del método Delphi en investigaciones sobre salud publicadas en revistas científicas cubanas. Rev Cubana de Información en Ciencias de la Salud 2013; 24(2):133-144.,²⁷27 Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. Plos One 2011; 6:e20476 through association with the consensus conference technique recommended by Souza et al.²⁸28 Souza LEPF, Silva LMV, Hartz ZMA. Conferência de consenso sobre a imagem-objetivo da descentralização da atenção à saúde no Brasil. In: Hartz ZMA, Silva LMV, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. p. 65-102.. In short, a first round with the application of the traditional Delphi technique took place in the study, while the second and third rounds corresponded to the implementation of the consensus conference.

For the first round sixty experts were invited consisting of thirty PhD professors with active inclusion in discussions on education in dentistry with the Brazilian Association of Dental Schools (ABENO), and thirty doctorate professors with membership in the Working Group on Oral Health of the Association of Collective Health (ABRASCO). To identify these participants the understanding was that “experts” are people connected to organizations able to provide conclusive values about a particular problem with utmost competency, make objective predictions about the effects, applicability, feasibility and relevance that can contribute to solving this problem, and suggesting recommendations about what to do to perfect them²⁹29 Crespo T. Respuestas a 16 preguntas sobre el empleo de “expertos” en investigación pedagógica. Lima: San Marcos; 2007..

The ideal number of participants for achieving Delphi consensus is variable. While Okoli and Pawlowski³⁰30 Okoli C, Pawlowski S. The Delphi method as a research tool: an example, design considerations and applications. Information & Management 2004; 42:15-29. indicate that between 10 and 18 “experts” are sufficient for method development, Valdes and Marín²⁵25 Valdés MG, Marín MS. Empleo del método Delphi en investigaciones sobre salud publicadas en revistas científicas cubanas. Rev Cubana de Información en Ciencias de la Salud 2013; 24(2):133-144. suggest a minimum of seven “experts,” noting that the consensus error is remarkably decreased by every expert added. However, studies recommend prudence in inviting more members to the panel of “experts” than provided because usually some quit or don't attend²⁵25 Valdés MG, Marín MS. Empleo del método Delphi en investigaciones sobre salud publicadas en revistas científicas cubanas. Rev Cubana de Información en Ciencias de la Salud 2013; 24(2):133-144.,²⁷27 Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. Plos One 2011; 6:e20476. Bloor et al.³¹31 Bloor M, Sampson H, Baker S, Dahlgren K. Useful but no Oracle: Reflections on the use of a Delphi Group in a multi-methods policy research study. Qualitative Research 2014; DOI: 10.1177/1468794113504103.
https://doi.org/10.1177/1468794113504103... reported that more important than the size of a Delphi group is the balance of the shares represented by the range of views, knowledge and interests placed.

Individual messages were sent to the “experts” via email, including the Free and Informed Consent Form (ICF), an explanatory text containing the study objectives and method of building consensus by Delphi method, as well as instructions for completing the instruments research. The logical model and the matrix of criteria were presented accompanied by the following question: “What are relevant criteria for the training of dentists based on prescribing to the national curriculum guidelines for Dentistry?” Each criterion in the matrix, as well as other studies³²32 Higa EFR, Hafner MLMB, Tonhom SFR, Taipeiro EF, Moreira HM, Guimarães APC, Pinheiro OL. Indicadores de avaliação em gestão e saúde coletiva na formação médica. Rev Bras Educ Médica 2013; 37(1):52-59.,³³33 Gontijo ED, Alvim C, Megale L, Melo JRC, Lima MECC. Matriz de competências essenciais para a formação e avaliação de desempenho de estudantes de Medicina. Rev Bras Educ Médica 2013; 37(4):526-539. included a score from 0 to 10, where 0 meant the exclusion criteria and 10 meant the maximum importance for the evaluation of courses based on the NCG. We asked the “experts” to judge each criterion and to attribute a score 0-10 following the same logic, but also to describe the reasons for their persistence or change in the criterion score. In the matrix, criteria averaging less than 7 (seven) and standard deviation value of less than or equal to 3 (three) were maintained²⁸28 Souza LEPF, Silva LMV, Hartz ZMA. Conferência de consenso sobre a imagem-objetivo da descentralização da atenção à saúde no Brasil. In: Hartz ZMA, Silva LMV, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. p. 65-102..

From the results flagged in the first round of the Delphi, the second round was developed in the consensus conference, for which four experts who were not participants in the first round were invited; doctors and professors with active inclusion in discussions on education in dentistry along ABENO, with extensive academic achievement in public health and publications in the area of vocational training of dentists, and those who had previously received the matrix with the criteria and their scores scaled in the Delphi consensus.

Just as in traditional Delphi, there is no consensus regarding the optimal number of participants in consensus conferences. A third variation of 3 to 15 experts has been observed in recent studies³²32 Higa EFR, Hafner MLMB, Tonhom SFR, Taipeiro EF, Moreira HM, Guimarães APC, Pinheiro OL. Indicadores de avaliação em gestão e saúde coletiva na formação médica. Rev Bras Educ Médica 2013; 37(1):52-59., ³⁴34 Costa EAM, Costa EA, Graziano KU, Padoveze MC. Reprocessamento de produtos médicos: uma proposta de modelo regulatório para hospitais brasileiros. Rev Esc Enferm USP 2011; 45(6):1459-1465.–³⁷37 Reis YAC, Cesse EAP, Carvalho EF. Consensos sobre o papel do gestor estadual na regionalização da assistência à saúde no Sistema Único de Saúde (SUS). Rev. Bras. Saúde Matern. Infant. 2010; 10(Supl. 1):S157-S172., which conclude that representativeness should be more evaluated for quality by a panel of experts than by the number of participantes³⁸38 Powell C. The Delphi technique: myths and realities. J Adv Nurs 2003; 41(4):376-382., as occurred in this study.

A consensus conference was developed at an in-person meeting in which experts reviewed and discussed about the dimensions, sub-dimensions and proposed evaluation criteria for the matrix. From the debate, the consensus of this group was recorded.

The experts at the consensus conference were invited to individually analyze the resulting matrix of their work, which consisted in the third round of the Modified Delphi.

Ethical Aspects

The project was submitted to the Ethics Committee at Hospital Onofre Lopes and registered in the Brazil Platform, following all the guidelines of Resolution 466/12³⁹39 Brasil. Ministério da Saúde (MS). Conselho Nacional de Saúde. Resolução n° 466, de 12 de dezembro de 2012. Diário Oficial da União 2013; 13 jun. of the National Health Council's Project and was approved.

Results

The logical model in Figure 1 is the result of the preliminary construction by researchers, plus the suggestions forwarded by the “experts” in the first round of the Delphi consensus. To ensure the configuration of the proposed image-goal, the following components were entered into the model with aspects related to training: pedagogical approach, the graduate profile, teaching-service integration and orientation of health care, all organized to establish relationships and routes that were relevant to the objective. The logical model obtained allowed for a better understanding of the structure and evaluation elements of the issues involved and the possible relationships between these, guiding the construction of a proposed evaluation criteria matrix for courses of Dentistry.

This matrix was sent to “experts” who were able to judge the consistency of the criteria with the corresponding dimension and also indicate their suggestions for improving the matrix, either by exclusion criteria, appointment of new criteria, new dimensions, with respective scores or any other reformulations they deemed necessary.

The consulting of the ‘experts’ was performed by Delphi during the months of July to August 2013. Of the sixty invited experts, a total of thirty-three professors responded to the matrix, a number deemed satisfactory by other studies with the same characteristics²⁵25 Valdés MG, Marín MS. Empleo del método Delphi en investigaciones sobre salud publicadas en revistas científicas cubanas. Rev Cubana de Información en Ciencias de la Salud 2013; 24(2):133-144.,³⁰30 Okoli C, Pawlowski S. The Delphi method as a research tool: an example, design considerations and applications. Information & Management 2004; 42:15-29..

The scores given by the “experts” to each criterion matrix were entered into a spreadsheet and analyzed descriptively using the mean and standard deviation. The higher the average, the higher the importance given to the criteria. The standard deviation verified the estimate of the degree of agreement being inversely proportional to this, regardless of the importance attributed to criteria²⁸28 Souza LEPF, Silva LMV, Hartz ZMA. Conferência de consenso sobre a imagem-objetivo da descentralização da atenção à saúde no Brasil. In: Hartz ZMA, Silva LMV, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. p. 65-102..

The results are shown in Table 1, which indicates the proposed dimensions and sub-dimensions in the matrix, the average and standard deviation and its final score to be considered after the evaluation by “experts.”

Among the twenty-five initial criteria proposed, five criteria were considered non-essential to the evaluation of training dentists and were therefore excluded from the matrix: disease prevention (Dimension of orientation of health care); inclusion of students in the SUS, SUS theoretical framework, reference and counter-reference, and planning and evaluation of services (Dimension of teaching-service integration).

In the second round corresponding to the consensus conference, the four experts confirmed the dimensions and sub-dimensions validated in the previous step and reconsidered part of the sub-dimensions defining criteria. The analyzes obtained from experts in the third round confirmed the criteria discussed throughout the actual meeting, setting the final matrix of evaluation criteria for Dentistry courses according to the NCG, shown in Chart 2.

Discussion

In a study of pre-evaluation, steps were developed such as a detailed description of the intervention, evaluative questions relevant to the process, building the logical model and the consensus on the items to be evaluated to provide clarification and reflection on boundaries and possibilities of the proposed evaluation¹⁰10 Mendes MFM, Cazarin G, Bezerra LCA, Dubeux LS. Avaliabilidade ou pré-avaliação de um programa. In: Samico A, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em Saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010. p. 57-64..

Among the three types of models ranked by Champagne et al.⁴⁰40 Champagne F, Brousselle A, Hartz Z, Contandriopoulos AP. Modelizar as intervenções. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z, organizadores. Avaliação: conceitos e métodos. Rio de Janeiro: Editora Fiocruz; 2011. p.61-74., the logical operating model is the closest to the model obtained in this study, since it represents the way the goals of the intervention can be achieved through resource and processes. In the constructed model, the detail of the image-goal allowed for a list of the aspects perceived in the NCG that are priorities for training better to the health needs of the population. In this perspective, Lampert¹⁶16 Lampert JB. Avaliação institucional nos cursos de graduação da área da saúde: avaliar o quê e para quê? Cadernos ABEM 2009; 5:45-55. states that the “train of thought” for those who will move toward the evaluation of institutions that purport to train health professionals begins with the health needs of society.

The logical model enabled the construction of the matrix of criteria for evaluating the training of dentists in the four dimensions proposed. There was consensus and similarity in value judgment criteria, and most of these were considered important to assess in view of the “experts” by Delphi consensus technique. One of the biggest challenges for the application of this technique was asking stakeholders to strengthen the principles of NCG, especially focused on the SUS as a privileged setting for learning and student centered teaching approaches.

The Delphi consensus facilitates access to a broad group and simultaneously a geographically dispersed population and allows the exchange of information between a large number of important people²⁴24 Esher A, Santos EM, Magarinos-Torres R, AzeredoTB. Construindo critérios de julgamento em avaliação: especialistas e satisfação dos usuários com a dispensação do tratamento do HIV/AIDS. Cien Saude Colet 2012; 17(1):203-214.. Because there are no universally accepted and standardized requirements for the Delphi technique, a large variability can be found in investigations that use the method. Authors²⁵25 Valdés MG, Marín MS. Empleo del método Delphi en investigaciones sobre salud publicadas en revistas científicas cubanas. Rev Cubana de Información en Ciencias de la Salud 2013; 24(2):133-144. argue that this variability does not consist in the failure of developing the method, it can be considered an opportunity due to its adaptability to conditions in which the study is situated. In a systematic review of the literature for the selection of quality indicators in health using the Delphi technique²⁷27 Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. Plos One 2011; 6:e20476, most of the 80 studies analyzed describe modifications of the original method in which at least one physical meeting between the participants was included, just as occurred in the present study for the validation which describe sub-dimension criteria.

The option for the association of consensus techniques occurred with the view to reach consensus on all the criteria in the first round, for the possibility of adding the benefits related to classroom discussion of the criteria already submitted to initial individual consensus²⁸28 Souza LEPF, Silva LMV, Hartz ZMA. Conferência de consenso sobre a imagem-objetivo da descentralização da atenção à saúde no Brasil. In: Hartz ZMA, Silva LMV, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. p. 65-102., which allowed to clarify possible misunderstandings, while still respecting individual opinions in the third round²⁷27 Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. Plos One 2011; 6:e20476.

The dimensions “profile of graduate” and “pedagogical approach” had all the criteria considered relevant to the evaluation of training dentists, the latter dimension to that obtained by the average in all high scores of relevance criteria from the experts.

Although currently coexisting, teaching trends change over time and suggest a better way for education, including health professions, of questioning or criticising pedagogies, because the value of the student to know and to institute a change in reality and of itself in the development of human skills and techniques⁴¹41 Pereira ALF. As tendências pedagógicas e a prática educativa nas ciências da saúde. Cad Saude Publica 2003; 19(5):1527-1534..

In the dimension “orientation of health care,” the only criterion excluded was “disease prevention.” The prevention of diseases, mainly dental caries, is already a practice worked at extensively in dentistry training which may have influenced the lack of association of this criterion to the changes required in the curriculum aligned with current requirements of the NCG. Furthermore, the presence of the criterion “health promotion” may have led to the understanding that preventive practices should be incorporated from the strengthening of people's capacity to adopt the various factors that influence health.

Most of the criteria considered non-essential belonged to “teaching-service integration” dimension. Contrary to what may initially seem to be, it is believed that there was a devaluation of this magnitude by the “experts” as to its relevance in the evaluation of the training of health professionals. This finding may be explained by the fact that an overlap of interpretations between the proposed criteria in the initial matrix for this dimension was observed. It is noticed that the four criteria excluded (inclusion of students in the SUS, SUS theoretical framework, reference and counter-reference, and planning and evaluation services) are included in the evaluation criteria of “experiences in SUS” and “internship activities.”

The need for assessment systems with quantitative and qualitative indicators to identify strengths and weaknesses and guide the redirection of institutional planning are closely related to the mission of the institution formed by actions of teaching, research and extension based on the relationship between state and society and political institutes¹⁷17 Braid LMC, Machado, MFAS, Aranha AC. Estado da arte das pesquisas sobre currículo em cursos de formação de profissionais da área da saúde: um levantamento a partir de artigos publicados entre 2005 e 2011. Interface (Botucatu) 2012; 16(42):679-692.. Public policies in the field of evaluation are potential inducers of transformation aligned to fulfill the NCG and constitutional guidelines of the SUS in the graduation of the health professional, reinforcing social responsibility and response to contemporary health demands of the population⁴²42 Ceccim RB, Feuerwerker LCM. Mudança na graduação das profissões de saúde sob o eixo da integralidade. Cad Saude Publica 2004; 20(5):1400-1410..

The Federal Constitution of 1988 brings the notion of the state as an evaluator, putting the government as responsible for approval of the entry and permanence of courses in the education system, based on the evaluation process conducted by them⁵5 Brasil. Constituição da República Federativa do Brasil de 1988. Diário Oficial da União 1988; 5 out.,⁴³43 Ristoff D, Giolo J. O Sinaes como sistema. Rev Bras Pós-Grad 2006; 3(6):193-213.. Currently this regulation is carried out by the National System of Higher Rating (SINAES), proposed in 2002 and established as a state policy in 2004, which results from various initiatives introduced for higher education evaluation since the late 70's⁴³43 Ristoff D, Giolo J. O Sinaes como sistema. Rev Bras Pós-Grad 2006; 3(6):193-213..

In the instrument for assessing undergraduate courses used by SINAES, the evaluative dimensions are didactic-pedagogic organization, teaching faculty and infrastructure. Still herein, compliance with the NCG is characterized as a legal and regulatory mandatory requirement, however, to be an essentially regulatory item, not part of the calculation concept of course evaluation¹³13 Brasil. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira – Inep. Diretoria de Avaliação da Educação Superior – Daes. Sistema Nacional de Avaliação da Educação Superior – Sinaes. Instrumento de Avaliação de Cursos de Graduação presencial e a distância. Brasília: maio 2012. [acessado 2013 jun 13]. Disponível em: http://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2012/instrumento_com_ alteracoes_maio_12.pdf
http://download.inep.gov.br/educacao_sup... .

The indicators and criteria recommended in the assessment tool of undergraduate INEP courses allow to affirm values that ensure adherence to the PPC of NCG⁴³43 Ristoff D, Giolo J. O Sinaes como sistema. Rev Bras Pós-Grad 2006; 3(6):193-213., but also make valid judgments on the proposed dimensions. Nevertheless, despite the efforts of joint action between Ministries of Health and Education to include the NCG and SUS standards in higher education evaluation in health care based on SINAES⁷7 Brasil. Ministério da Saúde (MS). Ministério da Educação (ME). A aderência dos cursos de graduação em enfermagem, medicina e odontologia às diretrizes curriculares nacionais. Brasília: MS; 2006., for specific evaluation of courses in the area of health, and in this case dentistry courses, attention to the aspects inherent to the training peculiarities in this area of work is necessary. Therefore, the matrix of criteria proposed in this study brings the relative issues of health care guidance in their current perspective and the dimension of teaching-service integration facing the SUS, a condition stipulated in its own NCG and indispensable to the professional profile required by it.

Pinheiro et al.¹⁸18 Pinheiro FMC, Nobrega-Therrien SM, Almeida MEL, Almeida MI. A formação do cirurgião-dentista no Brasil: contribuições de estudos para a prática da profissão. RGO 2009; 57(1):99-106. discussed the production of academic papers on the training of dentists in Brazil between 1992 and 2005. Studies found by the authors dealt with the profile, the social role of the dentist, the characteristic of work related to training, and evaluating curricular internship, curriculum structure, analysis, trends and contradictions, all related to NCG. Furthermore, Casotti et al.¹⁹19 Casotti E, Ribeiro VMB, Gouvêa MV. Educação em odontologia no Brasil: produção do conhecimento no período 1995-2006. Hist Cien. saude-Manguinhos 2009; 16(4):999-1010. have presented results of the production of knowledge about dental education between the years 1995 and 2006 based on 52 dissertations and 20 theses from the database of the Coordination of Improvement of Higher Education Personnel. They concluded that production in this field is still in its infancy, but with multiple issues, such as disciplinary innovative experiences, teacher training, curriculum and graduate profile.

Among the few studies that purport to work with validated criteria for evaluating dentistry courses, Zilbovicius et al.²⁰20 Zilbovicius C, Araujo ME, Botazzo C, Frias AC, Junqueira SR, Junqueira CR. A paradigma shift in predoctoral dental curricula in Brazil: evaluating the process of change. J Dent Educ 2011; 75(4):557-564. aimed to analyze the trend of changes in Brazilian dental education in the face of the need for NCG implementation. Quantitative data was gathered from the assessment tool used in the ABENO workshops in 2005 and 2006. Indicators of this instrument are based on radial analysis⁴⁴44 Campos FE, Ferreira JR, Feuerwerker LCM, Sena RR, Campos JJB, Cordeiro H, Cordoni Júnior L. Caminhos para aproximar a formação de profissionais de saúde das necessidades da atenção básica. Rev Bras Educ Médica 2001; 25(2):53-59. consisting of three axes: theoretical orientation, pedagogical approach and practice scenarios. At the end of the radial assessment, most of the courses showed an developing degree of curricular innovation²¹21 Araujo ME. Palavras e silêncios na educação superior em odontologia. Cien Saude Colet 2006; 11(1):179-182..

In the study titled “The Adhesion of Undergraduate Courses in Nursing, Medicine and Dentistry to National Curriculum Guidelines”⁷7 Brasil. Ministério da Saúde (MS). Ministério da Educação (ME). A aderência dos cursos de graduação em enfermagem, medicina e odontologia às diretrizes curriculares nacionais. Brasília: MS; 2006., quantitative indices were created following dimensions and criteria of INEP reports that were more associated with prescribing to the NCG, and the evaluation reports and PPC courses which achieved lesser and greater adherence to the NCG were also qualitatively assessed. For courses of Dentistry, the results indicate that the barriers to NCG implementation are primarily related to the integration of teaching and service in the SUS, the difficulty of integration and curricular modification and difficulty in developing innovative, pedagogical projects different from the traditional teaching model. They concluded based on the pedagogical projects analyzed that the vocational training of dentists is still largely concentrated on the diagnosis and treatment of disease⁷7 Brasil. Ministério da Saúde (MS). Ministério da Educação (ME). A aderência dos cursos de graduação em enfermagem, medicina e odontologia às diretrizes curriculares nacionais. Brasília: MS; 2006..

In both studies cited⁷7 Brasil. Ministério da Saúde (MS). Ministério da Educação (ME). A aderência dos cursos de graduação em enfermagem, medicina e odontologia às diretrizes curriculares nacionais. Brasília: MS; 2006.,²¹21 Araujo ME. Palavras e silêncios na educação superior em odontologia. Cien Saude Colet 2006; 11(1):179-182., important results could be obtained on the trend of changing the training of dentists since the implementation of NCG. However, many shortcomings remain on how the guidelines can be effected into curricula and educational practices, and which can be assessed by a greater number of ways through more sensitive and specific valid indicators⁴⁵45 Alves CKA, Natal S, Felisberto E, Samico I. Interpretação e análise das informações: o uso de matrizes, critérios, indicadores e padrões. In: Samico A, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em Saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010. p.89-107., and verified by accessible, legitimate and close sources to the everyday training spots. It is also essential to understand the public health services and the world of work as central elements of a new pedagogical approach, capable to contribute to the new health care professional acting with the capacity of understanding reality and intervening on it⁴⁶46 Werneck MAF, Senna MIB, Drumond MM, Lucas SD. Nem tudo é estágio: contribuições para o debate. Cien Saude Colet 2010; 15(1):221-231..

The final matrix proposed in this study brings dimensions, sub-dimensions and is capable of being used to conduct evaluative studies of dentistry courses based on proposals of the NCG criteria. To do so, these criteria can be understood as cognitive categories²⁴24 Esher A, Santos EM, Magarinos-Torres R, AzeredoTB. Construindo critérios de julgamento em avaliação: especialistas e satisfação dos usuários com a dispensação do tratamento do HIV/AIDS. Cien Saude Colet 2012; 17(1):203-214., constructed from the NCG guidelines for training of health professionals and may be used in evaluating them.

The dimensions shown in this evaluative proposal had a different purpose as they deepened structural elements unexplored in the scientific literature. Still, one can glimpse the possibility of the matrix to be used, with appropriate adaptations, to other courses in the area of health. There is also possibility to use different methodological approaches in studies which intend to use the validated criteria herein.

The criteria presented here have duly validated scores for evaluation in quantitative studies, showing the sensitivity provided on a scale of 1 to 10. Still, the improvement of these criteria could be accomplished through consensus conference, where subjective reflections on aspects mentioned as inherent to the evaluation of the object in question were made possible since the attempt to insert subjective elements of the different actors in the construction of criteria enables increased validity of the parameters analyzed⁴⁷47 Minayo MCS. Construção de indicadores qualitativos para avaliação de mudanças. Rev Bras Educ Médica 2009; 33(Supl. 1):83-91..

Faced with the need of evaluative culture of health education implementation based on new paradigms and training needs of the health system and the population, and consisting of a process that results in renewals for improving training activities, limitations are explicit as to the applicability of criteria built here. An example is the very purpose and evaluative understanding of the definition of competent authorities for these actions in higher education institutions (HEIs). Self-assessment was thought of as an important strategy since it allows the HEI self-awareness that favors the construction of a participatory and inclusive culture of assessment based on self-evaluative management of this process, enriching the planning and subsidizing the construction of internal development policies and offering higher quality education and public education policies⁴⁸48 Polidori MM, Marinho-Araújo CM, Barreyro GB. Sinaes: perspectivas e desafios na avaliação da educação superior brasileira. Ensaio: aval pol pub educ 2006; 14(53):425-436..

This leads itself to the reflection on the importance of careful planning and methodological evaluative research on health in general, including in the educational field. Difficulties on the availability of theoretical guidance for methodological construction of evaluation studies in health that are not limited to programs and services are currently found in the literature.

The final instrument proposed in this study, legitimized by various participants who comprised the steps of construction and validation, is a differentiated alternative assessment of training dentists and other health professionals, to enable an assessment by a larger number of aspects, through valid, sensitive and specific indicators, and that can be checked by accessible, legitimate and closer everyday spots of the training sources, allowing for better approximation between evaluation methods of health education and the reality of the effectiveness of NCG.

References

Publication Dates

History