Abstract

The potential of the reflective collective portfolio as a method of teaching, learning and assessment needs to be emphasized. The objectives is to assess and validate the quality of the collective portfolio as a method of teaching, learning and assessment in the training of health professional. Portfolio quality was analyzed using an inventory of analysis that was adapted and tested for this study. In total, 70 portfolios were analyzed. The Kappa test was used to analyze reproducibility, confirming the level of agreement between the appraisers. Descriptive analysis and the t-test were performed to compare results from the years the portfolios were produced. The constructs (layout and organization, comprehensive, reflective, critical and creative thought) exhibited Cronbach alpha values above 0.7, confirming the high reliability of the tool. The weighted Kappa test revealed an elevated and significant agreement (+90%) between the appraisers. Notably, certain items and constructs exhibited significant differences over the years. All of the constructs were assessed better from 2011 onwards, when the assessment and self-assessment tool was implemented. The success of the portfolio depends on strategies that provide students with greater clarity and forms of developing the method.

Evaluation; Health education; Medical education

Resumo

O potencial do portfólio coletivo reflexivo como método de ensino, aprendizagem e avaliação precisa ser enfatizado. Os objetivos consistem em avaliar e validar a qualidade do portfólio coletivo como um método de ensino, aprendizagem e avaliação na formação de profissionais de saúde. A qualidade do portfólio foi analisada utilizando um inventário de análise que foi adaptado e testado para este estudo. No total, foram analisados 70 portfólios. O teste Kappa foi utilizado para analisar a reprodutibilidade, confirmando o nível de concordância entre os avaliadores. Análise descritiva e teste t foram realizados para comparar a construção do portfólio nos diferentes anos. Os construtos (layout e organização, compreensão, reflexão, crítica e pensamento criativo) apresentaram valores de alfa de Cronbach acima de 0,7, confirmando a alta confiabilidade da ferramenta. O teste Kappa ponderado revelou uma concordância elevada e significativa (+ 90%) entre os avaliadores. Notavelmente, certos itens e construtos apresentaram diferenças significativas ao longo dos anos. Todas as construções foram avaliadas melhor a partir de 2011, quando a ferramenta de avaliação e autoavaliação foi implementada. O sucesso do portfólio depende de estratégias que proporcionem aos alunos maior clareza e formas de desenvolver o método.

Avaliação; Educação em saúde; Educação médica

Introduction

Portfolios have been defined as a collection of evidence and reflections that students use to demonstrate the results of specific learning and are an effective method of assessing personal and professional development ¹1. Exley K , Dennick R . Ensenanza em pequeños grupos em Educación Superior . Tutorias, seminários e outros agrupamentos . 2nd ed. Madrid : Narcea ; 2009 . . Thus, the portfolio is a method designed by the students who must identify evidence that allows them to assess their knowledge, skills and attitudes (competence), as well as the learning process itself ²2. Gonzalez S . Técnicas docentes y sistemas de evaluación em Educación Superior . Madrid : Narcea ; 2010 .

3. Cotta RMM , Costa GD , Mendonça ET . Portfólio reflexivo: uma proposta de ensino e aprendizagem orientada por competências . Cien Saude Colet . 2013 ; 18 ( 6 ): 1847 - 1856 .^-44. Cotta RMM , Mendonça ET , Costa GD . Portfólios reflexivos: construindo competências para o trabalho no Sistema Único de Saúde . Rev Panam Salud Pub 2011 ; 30 ( 5 ): 415 - 421 . . In this context, the students are considered as agents: “somebody who acts and causes change and whose achievements can be judged by their own values and objectives, regardless of assessments associated with external criteria” ⁵5. Sen AK . Desenvolvimento como liberdade . São Paulo : Companhia das letras ; 2000 . .

There are many different types of portfolios ⁶6. Tartwijk JV , Driessen EW . Portfolios for assessment and learning: AMEE Guide no. 45 . Med Teach 2009 ; 31 ( 9 ): 790 - 801 .^,77. Tekian A , Yudkowsky R . Assessment portfolios . In: Downing EM , Yudkowsky RBY , editors . Assessment in Health Professions Education . London : Routledge ; 2009 . p. 287 - 304 . , including the collective portfolio, which involves the acquisition of competence in a group, encouraging teamwork and training future professionals in problem-solving and decision-making. This version of social reality presupposes that professional training leads to a new rationality. Students must be prepared to provide the most adequate, competent, democratic and effective answers to the enormous challenges that await them in contemporary society, particularly in the area of health. Therefore, the development of skills, in terms of making people and organizations more reflective, competent and efficient, is indispensable in the training of new professionals ⁸8. Alarcão I . Escola Reflexiva e nova racionalidade . Porto Alegre : Artmed Editora Ltda ; 2001 . .

This emphasizes the potential of the reflective portfolio as a method of teaching, learning and assessment, through the development of the following skills: decision-making; knowing how to work as part of a team, including inter-disciplinary teams and occasionally trans-disciplinary teams; communication skills; development of critical, reflective and creative thought; strengthening autonomy in the learning process.

However, in spite of all of the positive points in favor of the reflective portfolio, there is a consensus about the fragility of the assessment and validation of this method in the day-to-day operation of universities. Driessen et al. ⁹9. Driessen EW , Overeem K , Tartwijk J , Vleuten CPM , Muijtjens AMM . Validity of portfolio assessment: which qualities determine ratings? Med Educ 2006 ; 40 ( 9 ): 862 - 866 . stated that assessing a portfolio is complex due to its open, dynamic and flexible format. The same authors defended more global assessments with trained appraisers and stated that the richness and complexity of portfolios cannot be captured by analytical assessment criteria and extremely technical checklists can easily trivialize the assessment. From this perspective, it is essential to create portfolio analysis tools that ensure each portfolio is designed for the specific learning context of the student or group in question.

With this fragility of the assessment and validation of portfolios in mind, the aim of the present study was to assess and validate the collective portfolio as a method of teaching, learning and assessment, in terms of professional health training, as well as to present a tool created and tested for this purpose.

Materials and methods

Context

The construction of a collective portfolio is used as a didactic method of teaching, learning and assessment in the discipline of Health Policy for graduate courses in a Brazilian public university. The aim is to use this active method in large groups. The primary purpose of the portfolio is to promote learning associated with health policies, stimulating critical thought and enabling students to decipher reality, acquire autonomy of judgment and share accountability in knowledge building while part of a team. The portfolios are built collectively in groups composed of approximately six students.

At the beginning of the semester, the learning objectives, related to the construction of the portfolio, are designed jointly by teachers and students. In order to guarantee a clear structure with well-defined criteria and guidelines, students are instructed in all of the phases that should be carried out while constructing a portfolio: planning (before the activity); monitoring activities and the monthly assessment process (four assessments per semester); real-time assessments and feedback, with constant dialogue between teachers and students and between the students themselves ³3. Cotta RMM , Costa GD , Mendonça ET . Portfólio reflexivo: uma proposta de ensino e aprendizagem orientada por competências . Cien Saude Colet . 2013 ; 18 ( 6 ): 1847 - 1856 .^,44. Cotta RMM , Mendonça ET , Costa GD . Portfólios reflexivos: construindo competências para o trabalho no Sistema Único de Saúde . Rev Panam Salud Pub 2011 ; 30 ( 5 ): 415 - 421 .^,1010. Cotta RMM , Silva LS , Lopes LL , Gomes KO , Cotta FM , Lugarinho R , Mitre SM . Construção de portfólios coletivos em currículos tradicionais: uma proposta inovadora de ensino-aprendizagem . Cien Saude Colet 2012 ; 17 ( 3 ): 787 - 796 . .

The teacher who is accompanying the post-graduate students meets the groups, analyzes and discusses the items in the portfolio and promotes debate and questions about the evidence used. It is important to highlight that the assessment tool for the professor and the self-assessment tool for students was created and implemented in 2011, based on the experience of the professor and broad theoretical foundations, having been tested and discussed with students. Its structure is composed of learning objectives and aspects of classification and assessment (needs to improve, progressed adequately, progressed remarkably, exceptional).

Cotta et al. ¹⁰10. Cotta RMM , Silva LS , Lopes LL , Gomes KO , Cotta FM , Lugarinho R , Mitre SM . Construção de portfólios coletivos em currículos tradicionais: uma proposta inovadora de ensino-aprendizagem . Cien Saude Colet 2012 ; 17 ( 3 ): 787 - 796 . proposed four stages in the construction of a portfolio: 1) Search and conceptual production: the first proposed activity suggests that students search the scientific literature for the concepts, definition and aims of the portfolio (extra-curricular activity) and elaborate an individual and collective concept based on the literature consulted, including a list of the characteristics that a reflective portfolio should contain; 2) My trajectory: memories written at the beginning and end of the semester; 3) Learning with the group: activities designed and conducted in a group using narratives (reports of experiences in different practical scenarios), according to the theses involved; 4) Creative space: free space where the group can express their creativity using cartoons, poems, songs, photos and artwork from written and electronic media, associated with critical thought.

Assessment inventory

An inventory of analysis of the quality of portfolios was created as a result of the studies conducted by Driessen et al. ⁹9. Driessen EW , Overeem K , Tartwijk J , Vleuten CPM , Muijtjens AMM . Validity of portfolio assessment: which qualities determine ratings? Med Educ 2006 ; 40 ( 9 ): 862 - 866 . , Gadbury-Amyot et al. ¹¹11. Gadbury-Amyot C , Kim J , Palm R , Mills E , Noble E , Overman P . Validity and reliability of portfólio assessment of competency in a baccalaureate dental hygiene programme . J Dent Educ . 2003 ; 67 ( 9 ): 991 - 1002 . and Cotta et al. ³3. Cotta RMM , Costa GD , Mendonça ET . Portfólio reflexivo: uma proposta de ensino e aprendizagem orientada por competências . Cien Saude Colet . 2013 ; 18 ( 6 ): 1847 - 1856 .^,1010. Cotta RMM , Silva LS , Lopes LL , Gomes KO , Cotta FM , Lugarinho R , Mitre SM . Construção de portfólios coletivos em currículos tradicionais: uma proposta inovadora de ensino-aprendizagem . Cien Saude Colet 2012 ; 17 ( 3 ): 787 - 796 . with the aim of assessing portfolios using documentary analysis.

The items (20 questions, Table 1 ) in the inventory were formulated based on previously referenced studies, as well as the learning objectives proposed by the professor responsible for the discipline at the beginning of the process and the assessment and self-assessment criteria contained in the assessment tool.

A Likert scale was created to measure the criteria. The scale was designed with five points and was ordered as follows: clearly not contemplated; partially not contemplated; neutral; partially contemplated; clearly contemplated.

The constructs were formulated based on the theoretical assumptions of Lizarraga ¹²12. Lizarraga MLSA . Competencias cognitivas em educación superior . Madrid : Narcea Ediciones ; 2010 . adapted by Cotta et al. in 2011 ¹⁰10. Cotta RMM , Silva LS , Lopes LL , Gomes KO , Cotta FM , Lugarinho R , Mitre SM . Construção de portfólios coletivos em currículos tradicionais: uma proposta inovadora de ensino-aprendizagem . Cien Saude Colet 2012 ; 17 ( 3 ): 787 - 796 . and 2013 ³3. Cotta RMM , Costa GD , Mendonça ET . Portfólio reflexivo: uma proposta de ensino e aprendizagem orientada por competências . Cien Saude Colet . 2013 ; 18 ( 6 ): 1847 - 1856 . in which cognitive skills were divided as follows: comprehensive thought, which promotes skills such as comparison, classification, analysis, synthesis and the creation of arguments; critical thought, which promotes skills such as the investigation of the reliability of sources, the interpretation of causes, analogical reasoning and deduction; creative thought, which refers to the possibility of generating ideas, establishing associations, producing images, creating metaphors and establishing goals. In addition, theoretical aspects related to studies by Driessen et al. ⁹9. Driessen EW , Overeem K , Tartwijk J , Vleuten CPM , Muijtjens AMM . Validity of portfolio assessment: which qualities determine ratings? Med Educ 2006 ; 40 ( 9 ): 862 - 866 . and Gadbury-Amyot et al. ¹¹11. Gadbury-Amyot C , Kim J , Palm R , Mills E , Noble E , Overman P . Validity and reliability of portfólio assessment of competency in a baccalaureate dental hygiene programme . J Dent Educ . 2003 ; 67 ( 9 ): 991 - 1002 . were included.

A pilot study was carried out by the research team in order to adjust the formulation of the inventory items and their respective instructions. Ten post-graduate student portfolios were analyzed from the area of health promotion. It is worth noting that the appraiser used the items of the inventory scale for each activity in the portfolio to help analyze the portfolios.

Procedure

In total, 70 portfolios, designed between 2008 and 2013, were analyzed. Three properly trained appraisers performed the analysis. In order to increase inter-appraiser reliability, the appraisers discussed the items of classification of two portfolios, which were excluded from the present study. The analysis was performed blind by all appraisers.

Data analysis

When assessing the measurement properties of an adapted tool, Guillemin et al. ¹³13. Guillemin F , Bombardie C , Beaton D . Cross-cultural adaptation of health related quality of life measures: literature review and proposed guidelines . J Clin Epidemiol 1993 ; 43 ( 12 ): 1417 - 1432 . proposed an assessment of the reliability and validity of the tool.

The Kappa test, weighted between the three appraisers, was used to analyze reproducibility, confirming the level of agreement between the appraisers. The Kappa values were divided as follows: 0 – poor agreement; from 0 to 0.20 – mild agreement; from 0.21 to 0.40 – considered agreement; from 0.41 to 0.60 – moderate agreement; from 0.61 to 0.80 – substantial agreement; from 0.81 to 1.0 – excellent agreement.

Cronbach’s alpha was used to analyze internal consistency. Index values above 0.70 indicated a good construct. Pearson’s correlation coefficient was used to correlate each of the items in relation to each other and their constructs.

Descriptive analysis (mean and standard deviation) and the t-test were performed with the data to compare results from the years the portfolios were produced (2008 to 2013). The data were analyzed using SPSS 20.0 and Stata 11.0 software.

Ethical approval

The present study received approval from the Ethics Committee of the Universidade Federal de Viçosa (UFV). According to resolution 466/2012 of Brazil’s National Health Council, which regulates studies involving humans, a statement of informed consent was signed by those who agreed to participate in the research. The participants were guaranteed confidentiality and anonymity.

Results

The most significant finding of the present study was the high reliability of the analysis tool (Inventory of collective portfolio quality), as shown in Table 1 . The constructs (layout and organization, comprehensive, reflective, critical and creative thought) exhibited Cronbach alpha values greater than 0.7, which confirmed the high reliability of the tool in terms of analyzing the portfolio.

The weighted Kappa test revealed an elevated and significant agreement between the appraisers, taken two by two. The mean level of agreement was greater than 90%, as shown in Table 2 .

Therefore, it is worth noting that the analysis tool and the analysis performed by the appraisers were reliable in terms of the assessment conducted, which involved a careful and detailed pre-test, as well as the knowledge and experience of the appraisers in relation to the construction of portfolios.

With regards to the quality of the portfolio, the constructs present in the inventory (layout and organization, comprehensive, reflective, critical and creative thought) can be seen in Table 3 , along with their mean and standard deviation values.

As can be seen in Table 3 , most of the items exhibited values higher than 3 (neutral and generally contemplated) and these values increased considerably in the years 2011 and 2012. The items with values below 3 were associated with reflective thought when assessing the students capacity to build and rebuild concepts (item 9), thereby demonstrating the student´s difficulty in presenting literature related to the adequate construction of concepts. Another item with values below 3 (also associated with reflective thought) was the presentation of strengths and weaknesses during the construction of the portfolio (item 11). Some of the portfolios omitted this item entirely or only included it in one phase. The critical thought construct (item 16), which deals with deep and theoretical coherence of the evidence in the portfolio, also exhibited a mean value below 3. This was due to the fact that the bibliographical references were not evident or there were few connections between texts.

In the creative thought construct, item 20 exhibited a value below 2 due to the fact that the creation of a group in virtual communities was not obligatory and was meant to facilitate communication and teamwork, as well as helping decision-making and allowing students to make optimal use of the short time they had for meetings. This decreased the final mean value of this construct somewhat. Notably, this item also exhibited the greatest variability between the portfolios analyzed, particularly in the years after 2010, bearing in mind that no portfolio had previously exhibited this characteristic. When analyzing the entire period of the years studied (2008 to 2103), the greatest variability was associated with items 3, 11 and 19.

In spite of the fact that these items exhibited mean values below 3, the vast majority of the items had a final mean value greater than 3. Over the years, a number of items and constructs have reached values higher than 4, including the following: layout and organization (item 1 from 2009 to 2012, 2 and 3 from 2010 to 2012); comprehensive thought (item 5, 6, 7 and 8 in 2011 and 2012); reflective thought (item 10 in 2011); critical thought (item 15 in 2011 and 2012) and creative thought (item 19 and 20 in 2011 and 2012). Figure 1 displays the mean values for the constructs by year for the portfolios assessed.

One notable finding of the present study is the growth of the total mean value of the constructs over time, which indicates appropriation of the process of portfolio construction. The t-test of the mean values, displayed in Table 4 , reveals a significant difference, indicating that all of the constructs were assessed more adequately from 2011 onwards. This in turn indicates the construction of better portfolios, as a result of the acquisition of skills and an improvement in the method. The capacity for comprehensive, reflective, critical and creative thought improved in 2011 and 2012. This improvement may have been due to the use of the assessment and self-assessment tool that was created, implemented and used in the last two years (2011 and 2012). This assessment tool involves learning objectives and assessments for each item, including competence (skills, knowledge and attitudes). This tool enabled a clarification of the method, the assessment criteria and the understanding of the reflective, critical and creative processes.

The portfolios exhibited significantly higher values in the items of reflective and critical thought, which indicates that the students analyzed the views of different studies and scholars to form their own opinions. The analysis of the evidence (cartoons, reports, scientific articles) involves more than a list of facts: it explores themes; it questions the opinions provided; it makes connections and inter-connections between the texts and/or documents of the portfolio; ideas are formed leading to a creative and innovative presentation.

The construct with the greatest final mean value in each year was layout and organization. This demonstrates the ease with which the students organized the portfolio using the required steps, providing supplementary evidence and working in a team to develop and create the portfolio.

Discussion

The findings of the present study indicate improvements in the collective portfolio over the years as a method of teaching, learning and assessment, in terms of the acquisition of important skills on behalf of students. The portfolio contributed to an understanding of the content related to health policy and helped the development of the student’s skills and attitudes, particularly in relation to reflective, critical and creative thought ³3. Cotta RMM , Costa GD , Mendonça ET . Portfólio reflexivo: uma proposta de ensino e aprendizagem orientada por competências . Cien Saude Colet . 2013 ; 18 ( 6 ): 1847 - 1856 .^,1010. Cotta RMM , Silva LS , Lopes LL , Gomes KO , Cotta FM , Lugarinho R , Mitre SM . Construção de portfólios coletivos em currículos tradicionais: uma proposta inovadora de ensino-aprendizagem . Cien Saude Colet 2012 ; 17 ( 3 ): 787 - 796 . .

Gadbury-Amyot et al. ¹¹11. Gadbury-Amyot C , Kim J , Palm R , Mills E , Noble E , Overman P . Validity and reliability of portfólio assessment of competency in a baccalaureate dental hygiene programme . J Dent Educ . 2003 ; 67 ( 9 ): 991 - 1002 . suggested that portfolios are a valid and reliable method of assessing the competence of a student. Likewise, Driessen et al. ⁹9. Driessen EW , Overeem K , Tartwijk J , Vleuten CPM , Muijtjens AMM . Validity of portfolio assessment: which qualities determine ratings? Med Educ 2006 ; 40 ( 9 ): 862 - 866 . stated that the portfolio is a precise tool when assessing reflective competence.

One of the great authors of contemporary pedagogy, Dewey ¹⁴14. Dewel J . Democracia y Educacion . Madrid : Morata ; 2004 . was cited by Rué ¹⁵15. Rué J . El Aprendizaje Autónomo em Educación Superior . Madrid : Narcea Ediciones ; 2009 . , who identified four central aspects that are fundamental to all learning: experience; data for reflection; management and generation of ideas; retention of what was learned. Dewey ¹⁴14. Dewel J . Democracia y Educacion . Madrid : Morata ; 2004 . said that reflection is more than simply an ingredient that is necessary for success. It is in fact an intelligent experience in itself. Thus, reflective training becomes indispensable in the context of professional training. However, it demands conditions of effectiveness that incorporate the desire to reflect, as well as the development of knowledge, using determined activities and materials ¹⁵15. Rué J . El Aprendizaje Autónomo em Educación Superior . Madrid : Narcea Ediciones ; 2009 . .

The changes noticed in portfolios over the years demonstrate the importance of strategies created to help students in the process of acquiring skills (assessment and self-assessment tool, definition of learning objectives), as demonstrated by Driessen et al. ⁹9. Driessen EW , Overeem K , Tartwijk J , Vleuten CPM , Muijtjens AMM . Validity of portfolio assessment: which qualities determine ratings? Med Educ 2006 ; 40 ( 9 ): 862 - 866 . , and highlight the significance of the capacity and work of students during the reflection process. Based on the increased score of the constructs, it is clear that the items of the constructs were enhanced after the implementation of the assessment and self-assessment tool in 2011. The portfolio has improved as a method of learning and is now more than just an archive folder (with significant characteristics in the layout and organization). It now constitutes a method that focuses the triad of critical-reflective-creative thought and has become an effective method of assessing skills.

With regards to the teaching, learning and assessment processes promoted by the portfolio, the student should receive all of the necessary guidelines at the beginning, including the learning objectives to be achieved, the skills to be developed and the assessment criteria, all of which should be transferred in a transparent and consensual manner. Coll et al. ¹⁶16. Coll C , Martin E , Mauri I , Miras , M , Onrubea J , Solé I , Zabala A . O construtivismo em sala de aula . São Paulo : Ática ; 2004 . stated that in order for the student to be interested and motivated, they must understand clearly what they are supposed to do and why: “If a student does not know the reason for an exercise or understand what the task involves and what they themselves need, it will be extremely difficult to accomplish an in-depth successful study” ¹⁶16. Coll C , Martin E , Mauri I , Miras , M , Onrubea J , Solé I , Zabala A . O construtivismo em sala de aula . São Paulo : Ática ; 2004 . . Therefore, the student must act differently, research, create and recreate their knowledge and become autonomous in the learning process. At this moment, the professor abandons his role as a detainer and becomes a guide and epistemological provoker, instructing the students on how to build learning.

Conclusion

The collective portfolio is an important method in skills training. It requires organizational and structural resources that provide the quality necessary to achieve the proposed objectives. In addition to the skills exercised in an individual portfolio, the collective portfolio also promotes teamwork, contributing to the reflection process and the management of conflicts that are inherent in the decision making process. It also contributes to the development of leadership skills and strategies that involve the effective participation of all members of a team in collective tasks and activities. These factors help to prepare the student to work in a multi-professional environment. In addition, feedback related to the collective portfolio is not only provided by the professor, as is the case in individual portfolios, but by the students partners as well, thereby contributing to their process of self-assessment. Further studies are required to ascertain significant differences among the skills promoted by collective and individual portfolios.

The success of the portfolio depends on strategies that provide the students with greater clarity and forms of developing the method, particularly in terms of reflection and critical thought. The analysis of the 70 portfolios revealed that reflective and critical thought have improved in the years 2011 and 2012. This may be the result of the introduction of the assessment and self-assessment tool.

The structure (layout and organization) exhibited significant values, including some greater than 4 (which is the measurement for the scale that is usually used), during all of the years studied. Particular and specific characteristics were presented in the way ideas were organized and presented, which also indicated the development of creativity. In the portfolio assessments highlighted by the literature, there is little evidence in this category. However, the present study presents this aspect as an essential and determining characteristic, within a traditional curricular context, of creating innovative alternatives.

Thus, the inventory of analysis of portfolio quality, which was designed to analyze collective portfolios, was successful in terms of innovating and guiding the analysis of the portfolio and could be adapted to the realities and contexts of different learning processes.

Acknowledgments

The present study was conducted with the support of the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), which is an entity of the Brazilian Government focused on the training of human resources.

References

Publication Dates

History