ARTICLE ARTIGO
The impact of health-management training programs in Latin America on job performance
El impacto de los programas de formación en gestión de la salud en el desempeño laboral de los gerentes de salud en América Latina
Sonia Janeth Díaz-MonsalveI, II
ILatin America Centre, Liverpool School of Tropical Medicine, Liverpool, UK
IIFacultad de Ciencias de la Salud, Universidad Francisco de Paula Santander, Cúcuta, Colombia
ABSTRACT
A study was undertaken in Mexico, Colombia, and El Salvador to determine the impact of a management training program on health managers' job performance. A quasi-experimental design was used where in the baseline study an intervention group of 85 district health managers in the three countries was compared with a control group of 71 managers who did not receive the training program. After the implementation of an 18-month training program (which included 5-day training workshops and a series of tasks to be carried out between the workshops), the outcome in terms of improved job performance (i.e. use of predefined management techniques) was measured through twelve management performance indicators. The data collection tools were two questionnaires, participant observation in managers' workplaces, focus group discussions, staff interviews, and document analysis. In Mexico, the control group showed 8.3 times weaker management performance compared to the intervention group; in Colombia the value was 3.6 and in El Salvador 2.4. Factors associated with a successful training outcome were: (a) training techniques, (b) strengthening of enabling factors, and (c) reinforcement mechanisms.
Key words: Health Services Administration; Management Capacity; Management; Training
RESUMEN
Se desarrolló un estudio en México, Colombia y El Salvador para determinar el impacto de un programa de capacitación en el desempeño de trabajo de los gerentes de salud. El diseño del estudio fue quasi-experimental (no-randomizado) comparando dos grupos: el grupo de intervención con 85 gerentes en los tres países que recibieron la capacitación y, el grupo control con 71 gerentes que no recibieron la capacitación. Después de ejecutar el programa de enseñanza de 18 meses de duración (el cual incluyó talleres de cinco días de duración y compromisos), el mejoramiento en el desempeño fue medido utilizando 12 indicadores. Los resultados mostraron que, en México, el grupo control mostró 8.3 veces más debilidades en el desempeño que el grupo de intervención; en Colombia, el valor fue 3.6 y en El Salvador 2.4 veces mayor. Factores asociados con el éxito de los resultados del programa de capacitación fueron: (a) las técnicas de enseñanza utilizadas, (b) el fortalecimiento de factores asociados con el buen desempeño y, (c) mecanismos de refuerzo.
Palabras claves: Administración de los Servicios de Salud; Capacidad de Gestión; Gerencia; Capacitación
Introduction
Ongoing health reforms, which include decentralizing the sector in Latin America and elsewhere, have increased the importance of district health management. A number of authors 1,2,3,4,5 have stressed the importance of good management at the district level and the need for training.
There is a broad literature on how to measure the impact of health education and management training 6,7,8. The information about the training outcome of courses related to the management of specific diseases is available, but in most cases the outcome indicators were "correct use of standard treatments" 9,10,11. In only very few cases, control groups were used and the measurement of change was generally done shortly after the end of the training 12,13. Much information is given about how the training was done and about participant satisfaction 14,15,16,17,18, but no studies on the impact of management training on job performance were identified. No rigid evaluation criteria or performance indicators were applied, and only one study 19 used a control group to compare management skills acquired in the training course to similar staff members who did not take part in the training.
It was the objective of this research to measure the direct results of a new 18-month training program on district health managers' knowledge and performance in three Latin American countries.
Methodology
Study areas and study design
Study groups: the study areas and groups of health managers attached to the Ministry of Health (MoH) and Social Security System (SSS) in three Latin American countries are shown in Table 1.
Decision on the intervention-control design: ideally, a sample of managers from different regions should have been assigned randomly to the intervention group (with a training program) and to the control group (without the training program or with a different intervention). Unfortunately, random allocation in real-life situation where a health manager's training program is envisaged is not feasible. Therefore a quasi-experimental design was chosen. Health managers who experienced the training program were compared to a control group with similar characteristics but without the educational intervention. Both the baseline studies and the follow-up studies (after the training program for the intervention group) were undertaken among the intervention and control groups. In Mexico, the intervention and control groups were followed through the whole process, but in Colombia and El Salvador a second group of untrained managers had to be selected from the same management teams for the follow-up assessment. The first and second control group had the same general characteristics in terms of kind of posts, educational level, and time of employment (p < 0.05) and there were no statistically significant differences in indicators measuring job performance 20 between them.
Baseline study and outcome measurement in the follow-up study
Measuring health managers' knowledge: this was done in each of the training workshops by using knowledge tests with ten open-ended questions related to needs assessment, planning, communication, systems management, monitoring, and evaluation (see below).
Measuring job performance: the quantitative baseline study before the start of the training program collected information about health managers' job performance; it included two self-applied questionnaires, which were filled in by all participants in the intervention and control groups. Internal consistency checks showed a consistency of 88.6% between key interview answers in the two questionnaires. The questionnaires were piloted in Chile and slightly modified. Twelve management performance indicators based on a literature review 20 measured if district health managers used essential management techniques or not; they included:
(a) Needs assessment (identifying health needs of different population groups): 1. Use of health indicators and health services indicators; 2. Prioritization of problems for taking actions; 3. Identification of risk groups (risk assessment).
(b) Planning: 4. Design of district health plans; 5. Implementation of these plans; 6. Use of visual aids (e.g. maps for specific programs or risk maps for identifying priority areas).
(c) Coordination and communication: 7. Coordination meetings for human resource management; 8. Adequate structure of meetings (use of agendas and minutes); 9. Appropriate communication with peripheral levels.
(d) Systems management: 10. Involvement of communities in health actions; 11. Coordination with other sectors (education, agriculture, NGOs, district councils, and others).
(e) Monitoring and evaluation: 12. Use of monitoring and evaluation techniques.
Qualitative assessment of job performance: the qualitative baseline study included structured observation in the district health offices, semi-structured interviews with staff (for measuring the reliability of health managers' interview answers), and document analysis (health plans, maps, minutes of meetings, and others) both in the intervention and control groups. Previously trained national public health specialists carried out this study in order to decrease observer bias with 10% of quality checks. The external consistency between quantitative and qualitative study showed a high reliability of data.
Measuring outcome in the follow-up study: 18 months after the start of the training program, the same questionnaires were applied to the intervention and control groups in the three countries. The twelve indicators were also used for the outcome measure additionally to the information obtained in the qualitative study.
Sample size calculation: The expected increase of adequate management performance (in terms of use of key management techniques) due to the educational intervention was from 60% beforehand to 95% afterwards, or from 60% in the control group to 95% in the intervention group after the training program. In order to find a significant difference between the two groups, the power of the test to be 80%, and the significance level (alpha) to be 5%, a sample size for each group of 22 persons was calculated (using EpiInfo, EpiCalc; see Table 1 for managers included).
The training program (educational intervention)
Structure and objectives: the training program lasted 18 months. It contained five formal training courses, lasting five days each, with specific tasks and supervision by national facilitators in the interval periods of four months. The training topics were: how to do needs assessment, local planning, monitoring and evaluation, drug management, risk approach, quality assurance, resource management, program management, information systems, and community participation. The principal objectives of the training programs were to: improve district health managers' knowledge and management skills; to strengthen creativity in responding flexibly to unexpected challenges; and to work as a team. The expected overall training outcome was better job performance.
The methodology used a participatory approach with different visualization techniques (work with cards on pin-boards), group work, didactic games, and socio-dramas for simulating real health problems.
The replication workshops at district level: one important aim of the program was to replicate the contents and methodologies of the workshop program to the whole management team at district level. The replication workshops carried out by our trainees were a copy of the managers' training workshops and used the same training materials and evaluation forms as the "master" workshop. These are, however, not the subjects of this study.
Data processing and analysis
The data entry and processing was done using the EpiInfo program package. For the calculation of the risk ratio with their confidence intervals Statcalc of the EpiInfo program were used. The differences between before-and-after and intervention-control groups were calculated and the chi-squared test applied for statistical significance testing of the overall proportion of management techniques (or functions), which had not been used by health managers. Example from Figure 1: in a group of 33 health managers, each has to use 12 techniques, i.e. ideally these techniques have to be applied 396 times. However, in 13 cases they were not applied, resulting in a proportion of 3.3% of techniques which were not used. Additionally, the t-test was applied comparing mean number of practices adopted in the control and intervention groups.
Results
Knowledge gain
The average knowledge gain in the workshops (as measured through knowledge tests) for the three intervention groups is presented in Figure 1. All three groups started at the same low level of knowledge (expressed as percentage of correct answers) and reached a similar high level of knowledge (in all three groups the differences in percentages of correct answers before and after the training were statistically significant; p < 0.001; the 95%CI were larger before the training [18-32; 13-29; 12-30] than afterwards [77-83; 80-88; 73-81]).
Training impact on job performance
Outcome measurement: the main quantitative outcome indicators of the training program were the increased use of 12 key management techniques by the trainees (comparing Figure 2 with Figure 3). No specific weight was given to specific functions in order to avoid an arbitrary judgment about, which of these key functions are more important than others. In Figure 2 the differences between intervention and control groups in the baseline study are presented. None of them was statistically significant at the beginning of the training program. When comparing management performance in control groups "before and after" (averages in Figure 2 and Figure 3), the average management deficiencies did not change significantly (p > 0.05) although there was a certain fluctuation of individual management indicators in the three groups over time.
Comparative impact measurement of the educational intervention: this was based on the 12 outcome variables indicating deficient use of 12 management techniques (Figure 3) and complemented by the qualitative study, which shed more light on the improvement in quality of management performance. From these 12 variables, the average proportions of management techniques, which have not been applied by district health managers, were calculated Figure 3).
The averages of the control group were divided by the averages of the intervention group in order to obtain the relative risk (RR, risk ratio). This was used as an estimate for the impact of the training program. The highest impact of the educational intervention was achieved in the Mexican group: the amount of managerial deficiencies was 8.3 times higher in the control group, compared to the intervention group. In Colombia, this factor was lower (3.6) as well as in El Salvador (2.4) (in all three cases p < 0.01) (Figure 4). The differences between Mexico and the other two countries was statistically significant (p < 0.01) but not between Colombia and El Salvador. The same significance level was obtained in the before-after comparison in the intervention groups; in the control groups no change occurred.
Additionally, the comparison was done by means of the t-test comparing mean number of management practices not carried out in the control and intervention groups. This takes into account that the unit of intervention was the person, and whether or not a specific practice is adopted is likely to be clustered within individuals. Also, in this case the differences between control and intervention groups were highly significant (p < 0.001).
The significant reduction in non-use of management techniques among trainees (intervention group) can be observed when comparing the black bars in Figures 2 and 3. The issue of "how adequately" these techniques have been applied by managers has been assessed through the visits to health districts in the qualitative follow-up study, where important changes in the intervention groups could be observed; staff interviews, discussions with management teams and document analyses reconfirmed the validity of interview answers. No change in the control groups compared to the baseline assessments was found.
Other measures of program success: an indirect measure of the success of the program was the number of replication workshops to district health teams carried out by the workshop participants. Mexico: 147 replicated workshops for a total of 2,048 participants (on average 13.9 participants per workshop); Colombia: 62 replicated workshops for a total of 918 participants (14.8 participants per workshop) and El Salvador: no replicated workshops, as this was not yet a program element.
Discussion
Training for better management: prospects and limitations
Issues around training: does training lead to better district health management? The work by Rojas 19 and country examples by Pervilhac & Seidel 21 suggest that isolated training courses seem to have little or no impact. Also, in view of the many possible factors influencing health managers' performance, it is unlikely that training alone has a substantial impact if the general context is not favorable. Why then were the training programs of this study apparently successful, at least for the limited period of impact assessment? The answer is most likely: (a) that the delivery of the programs and the training techniques was quite unique; (b) that factors associated with good management performance were strengthened; and (c) that powerful reinforcement mechanisms were in place.
(a) Delivery of the training program: factors which contributed to the positive training outputs included participatory training methods; problem-based learning (using local examples for group work); joint development of indicators for monitoring management performance, practical work using operational research tools, exercises with a newly developed information system, field exercises on information synthesis and presentation and collection of insect vectors, participatory evaluation of each training workshop, and providing the right documents and literature.
Furthermore, the 18 months of the program provided enough time for trainers to better understand training needs and to adjust some training elements. This responds to Rogers' suggestion 22 (p. 37) that "ideally, adult learners themselves should play an active role in training programs, identifying their training needs, participating in planning and implementation of their own training".
(b) Strengthening variables associated with good performance: two variables which have been shown to be associated with good management performance 20, have to do with training: "Knowledge of the local situation" and in a more general way "management training received". In the training program "knowledge of the local situation" was clearly strengthened through acquiring new tools to analyze the local situation and to summarize and present the information in an understandable way. Management training generally seems to have a positive impact on managers' job performance.
(c) Reinforcement mechanisms: the most powerful reinforcement mechanism of the training in Mexico and Colombia was strong support from the central (national) level. In both countries, the program was financed by national sources. This meant that top managers (in Mexico, the General Director of IMSS-OPORTUNIDADES and in Colombia, the Minister of Health and the Director of Public Health) asked for progress and impact reports on the training program in their visits to districts and regions. Likewise, the different tasks district health managers had to comply in the time between the training workshops, contributed to reinforcing the training; these included the production of a poster and document on district health problems; the design of a district health plan and the above mentioned replication workshops. The different achievements of the training program in the three countries seem to reflect these differences in reinforcement mechanisms: El Salvador was successful but less so than Mexico. The program was financed by foreign aid money and not directly integrated into the MoH structure. Participants had no time to do the replication workshops, and university lecturers (instead of MoH staff) did the supervision of tasks. In Colombia, supervision was less strict and the line management not as clear as in Mexico, but replicated workshops were conducted by most of the trainees and the supervision by MoH staff was important. Mexico was highly successful regarding training outcomes; all the reinforcement mechanisms described above were in place.
Final appraisal of the training program and its outcome: the three expectations after training are: increase of knowledge; improvement of skills; and the practical application of both in future work. This has been achieved quite successfully in the three countries, but the long-term effects are unknown. It can be expected that, due to the high turnover rate of district health officers in El Salvador and some areas of Colombia, the positive effect of the training on district health management in these places is not sustainable. However, the recommendation will not be to discard such training but to institutionalize it in a systematic way. In addition, it is advocated that management training be integrated into quality assurance programs. The concepts used in performance management and quality assurance focusing on district health managers are similar; while activities in both can strengthen each other. The same participatory training methods can be applied with emphasis on teamwork and user involvement 23 such as were applied in the training programs in this research.
This experience with in-service management training confirms Stefanini's assumption 24 that management training cannot be separated from the work and activities being managed; that the different elements of management (such as human resources, finance or information) are best developed together and, that the organizational background has to be taken into account in management training.
Towards a refined methodology of measuring district health managers' job performance
Limitations of the study: the lack of randomization of intervention and control groups as well as the difficulties with the control group in two countries means that the conclusions concerning the success of the educational intervention have to be taken with caution. However, this study is the first to measure the impact of management training in a quasi-experimental design and is in a position to provide considerably stronger evidence than has previously been available.
Outlook: in this research only the use or non-use of key management techniques by district health managers has been quantified, complemented by qualitative descriptive studies about managers' job performance. A more sophisticated way of performance measurement would be to quantify how appropriately these techniques are being applied and which impact they have on staff, people, and people's health. However, the literature has shown that this is difficult to establish and that the effect of appropriate management on people's health might be impossible to determine because many other powerful factors are in play, such as resources available, job imperatives limiting the flexibility of the manager's decisions, socioeconomic conditions of the target population, disease trends, and others 25. Nevertheless, using the knowledge advanced by this research and complemented by more qualitative studies, it will be the challenge for the future to construct indicators on the appropriate use of key management tools, their direct effect on some outcome indicators, and how management training can modify them.
Acknowledgements
Financial support was provided by the Mexican and Colombian governments, KfW (in El Salvador), the British Council in Mexico and Colombia, and by the European Commission, INCO-DEV (Contract No IC18CT 980339) in Colombia. Many thanks to all trainees who participated with great enthusiasm in the educational program and to all colleagues who organized it. Institutionally the training program in Colombia was supported by the Ministry of Health, in El Salvador by the Department of Public Health at the Universidad Centroamericana "José Simeón Cañas", and in Mexico by IMSS-OPORTUNIDADES and partially by ECOSUR in Chiapas. I am indebted to Nigel Bruce, Department of Public Health, and University of Liverpool, for his continuing support and advice and to all my colleagues who participated in the training program.
References
1. Monekosso GL. A health district as the operational unit for primary health care. Trop Doct 1993; 23:7-8.
2. Collins C, Green A. Decentralisation and primary health care: some negative implications in developing countries. Int J Health Services 1994; 24: 459-75.
3. Comninellis N. Managing district finances. Trop Doct 1994; 25:87-91.
4. Beracochea E, Dickson R, Freeman P, Thomason J. Case management quality assessment in rural areas of Papua New Guinea. Trop Doct 1995; 25: 69-74.
5. Agyepong I. Coverage and quality of care from the perspective of a district medical officer. In: World Health Organization, editor. Meeting on common programme issues. Geneva: World Health Organization; 1996.
6. Abbat FR. Teaching for better learning. Geneva: World Health Organization; 1992.
7. Dignan MB. Measurement and evaluation of health education. Springfield: Charles C. Thomas; 1995.
8. Harter M, Vauth R, Tausch B, Berger M. Goals, content and evaluation of training seminars for quality circle moderators. Z Aerztl Fortbild (Jena) 1996; 90:394-9.
9. Bailey JE, Coombs DW. Effectiveness of an Indonesian model for rapid training of Guatemalan health workers in diarrhoea case management. J Community Health 1996; 21:269-76.
10. Kundi MZ, Ahmad I, Anjum M. Evaluation of diarrhoea management of health professionals trained at the diarrhoea training unit of Rawalpindi general hospital. J Pak Med Assoc 1997; 47:3-6.
11. Bojalil R, Guiscafre H, Espinosa P, Viniegra L, Martínez H, Palafox M, et al. A clinical training unit for diarrhoea and acute respiratory infections: an intervention for primary health care physicians in Mexico. Bull World Health Organ 1999; 77:936-45.
12. Jones JE, Nelson EA. Evaluation of an education package in leg ulcer management. J Wound Care 1997; 6:342-3.
13. Onion CW, Bartzokas CA. Changing attitudes to infection management in primary care: a controlled trial of active versus passive guideline implementation strategies. Fam Pract 1998; 15:99-104.
14. Kuchler F. The diploma course in health care management in tropical countries and the advanced modules towards a MSc international health. In: Pervilhac C, Seidel W, editors. Training for better district health management. Berlin: German Foundation for International Development; 1996. p. 19-25.
15. Muita J, Maritim G, Omwega A, Wang'ombe J, Njoroge P, Tenambergen E. Which tools for district health managers? Rationale, contents and methods of a short course in tropical public health. In: Pervilhac C, Seidel W, editors. Training for better district health management. Berlin: German Foundation for International Development; 1996. p. 55-63.
16. Nissack-Onloun F, Schmidt-Ehry B, Owona Essomba R, Onana M, Omboto S. Course on applied epidemiology and district health management in Central America. In: Pervilhac C, Seidel W, editors. Training for better district health management. Berlin: German Foundation for International Development; 1996. p. 65-9.
17. Pervilhac C, Kloss-Quiroga B, Anyangwe S, Maude G, Adjei S, Bichmann W. Improving district health management in Africa through continuing education: five years experiences of the DSE Berlin course. In: Pervilhac C, Seidel W, editors. Training for better district health management. Berlin: German Foundation for International Development; 1996. p. 41-53.
18. Zapata Y, Rodríguez C, Kloss-Quiroga B. Training programme on health management: an experience on continuing education for health services. In: Pervilhac C, Seidel W, editors. Training for better district health management. Berlin: German Foundation for International Development; 1996. p. 71-83.
19. Rojas Z. The search for a satisfactory approach to the evaluation of health management training: a case study from Central America [PhD Thesis]. Liverpool: University of Liverpool; 2002.
20. Díaz-Monsalve SJ, Doyle V. Modelo de garantía de calidad para Latinoamérica: manual de capacitación e implementación. San José: Instituto Centroamericano de la Salud/Liverpool School of Tropical Medicine; 2002.
21. Pervilhac C, Seidel W, editors. Training for better district health management. Berlin: German Foundation for International Development; 1995.
22. Rogers A. Adults learning for development. London: Cassell; 1992.
23. Díaz-Monsalve SJ. Measuring health managers' performance in Latin America. Ann Trop Med Parasitol 2003; 97:299-311.
24. Stefanini A. Sustainability: the role of NGOs. Bull World Health Organ 1995; 16:42-6.
25. Davies HT. Performance management using health outcomes: in search of instrumentality. J Eval Clin Pract 1998; 4:359-62.
Correspondence to
S. J. Díaz-Monsalve
Latin America Centre, Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3 5QA, UK
sdiazmonsalve@aol.com
Submitted on 09/Jun/2003
Final version resubmitted on 13/Feb/2004
Approved on 01/Mar/2004