LESSONS FROM THE FIELD
Improving health services to displaced persons in Aceh, Indonesia: a balanced scorecard
Améliorer les soins de santé destinés aux personnes déplacées à Aceh, Indonésie : un tableau de bord prospectif
Mejora de los servicios sanitarios para las personas desplazadas en Aceh, Indonesia: método de evaluación integral
Grace J ChanI,*; Kristin B ParcoII; Melva E SihombingII; Susan P TredwellII; Edward J O'RourkeI
IJohn Hopkins School of Public Health, 615 N Wolfe Street, Baltimore, MD, 21205, United States of America
IIInternational Organization for Migration, Geneva, Switzerland
ABSTRACT
PROBLEM: After the Indian Ocean tsunami in December 2004, the International Organization for Migration constructed temporary health clinics to provide medical services to survivors living in temporary accommodation centres throughout Aceh, Indonesia. Limited resources, inadequate supervision, staff turnover and lack of a health information system made it challenging to provide quality primary health services.
APPROACH: A balanced scorecard was developed and implemented in collaboration with local health clinic staff and district health officials. Performance targets were identified. Staff collected data from clinics and accommodation centres to develop 30 simple performance measures. These measures were monitored periodically and discussed at meetings with stakeholders to guide the development of health interventions.
LOCAL SETTING: Two years after the tsunami, 34 000 displaced persons continued to receive services from temporary health clinics in two districts of Aceh province. From March to December 2007, the scorecard was implemented in seven temporary health clinics.
RELEVANT CHANGES: Interventions stimulated and tracked by the scorecard showed measurable improvements in preventive medicine, child health, capacity building of clinic staff and availability of essential drugs. By enhancing communication, the scorecard also led to qualitative benefits.
LESSONS LEARNT: The balanced scorecard is a practical tool to focus attention and resources to facilitate improvement in disaster rehabilitation settings where health information infrastructure is poor. Introducing a mechanism for rapid improvement fostered communication between nongovernmental organizations, district health officials, clinic health workers and displaced persons.
RÉSUMÉ
PROBLÈME: Après le tsunami survenu en décembre 2004 dans l'océan Indien, l'Organisation Internationale pour les Migrations a construit des cliniques mobiles afin de dispenser des soins aux survivants hébergés dans des centres d'accueil temporaires dans la province d'Aceh, Indonésie. Des ressources limitées, une supervision inadéquate, le roulement du personnel et le manque d'un système d'informations sanitaires ont rendu difficile la mise en place de soins de santé primaires de qualité.
APPROCHE: Un tableau de bord prospectif a été développé et implémenté en collaboration avec le personnel local des cliniques et les responsables sanitaires du district. Les niveaux-objectifs de performance ont été identifiés. Le personnel a collecté des données auprès des cliniques et des centres d'accueil afin de concevoir 30 mesures de performance simples. Ces mesures ont fait l'objet d'un suivi périodique et de débats lors de réunions avec les parties prenantes dans un souci d'orientation du développement des interventions en matière de santé publique.
ENVIRONNEMENT LOCAL: Deux ans après le tsunami, 34 000 déplacés recevaient encore des soins prodigués par les cliniques mobiles installées temporairement dans deux districts de la province d'Aceh. De mars à décembre 2007, le tableau de bord a été mis en place dans sept cliniques mobiles.
CHANGEMENTS SIGNIFICATIFS: Les interventions stimulées et suivies par le tableau de bord ont montré des améliorations quantifiables en matière de médecine préventive, de santé infantile, de développement des capacités du personnel des cliniques et de disponibilité des médicaments essentiels. En améliorant la communication, le tableau de bord a aussi conduit à des bénéfices qualitatifs.
LEÇONS TIRÉES: Le tableau de bord prospectif est un outil pratique qui vise à concentrer l'attention et les ressources afin de faciliter les aménagements dans les sites de réhabilitation après un désastre où les infrastructures d'information sanitaire sont pauvres. Introduire un mécanisme pour une amélioration rapide a favorisé la communication entre les organisations non gouvernementales, les responsables sanitaires du district, le personnel des cliniques et les personnes déplacées.
RESUMEN
SITUACIÓN: Tras el maremoto del Océano Índico en diciembre de 2004, la Organización Internacional para las Migraciones construyó consultorios médicos provisionales para ofrecer servicios médicos a los supervivientes residentes en centros de acogida temporales por toda Aceh, Indonesia. La escasez de recursos, la supervisión ineficiente, la rotación de personal y la ausencia de un sistema de información sanitaria dificultaron en gran medida la prestación de unos servicios sanitarios primarios de calidad.
ENFOQUE: Se diseñó y aplicó un método de evaluación integral en colaboración con el personal sanitario local y los funcionarios del departamento de sanidad regional. Se identificaron los objetivos de rendimiento. El personal recopiló los datos de las clínicas y de los centros de acogida para establecer 30 medidas de actuación sencillas. Estas medidas se sometieron a un control periódico y se analizaron con los participantes para dirigir la evolución de las intervenciones sanitarias.
MARCO REGIONAL: A los dos años del maremoto, 34 000 desplazados seguían recibiendo los servicios de los consultorios médicos provisionales en dos distritos de la provincia de Aceh. Entre marzo y diciembre de 2007 se implantó el método de evaluación en siete centros de salud temporales.
CAMBIOS IMPORTANTES: Las intervenciones promovidas y llevadas a cabo por el método de evaluación acarrearon mejoras cuantificables en medicina preventiva, salud infantil, desarrollo del potencial del personal sanitario y disponibilidad de medicamentos esenciales. Al potenciar la comunicación, el método de evaluación conllevó unos beneficios cualitativos.
LECCIONES APRENDIDAS: El método integral de evaluación es una herramienta útil para fijar la atención y centrar los recursos en la ayuda para la rehabilitación de zonas catastróficas en las que las infraestructuras para la información sanitaria son deficientes. La introducción de un mecanismo de mejora rápida ha fomentado la comunicación entre las organizaciones no gubernamentales, los funcionarios sanitarios regionales, los trabajadores de los dispensarios médicos y las personas desplazadas.
Background
After the Indian Ocean tsunami in December 2004, there were 530 000 internally displaced persons in Aceh province, Indonesia. Local health care was often unavailable, since health workers were missing and facilities destroyed. The International Organization for Migration (IOM) worked with the Ministry of Health and local district health offices (DHOs) to construct temporary clinics for displaced populations.
Problem
As of November 2006, 34 000 persons still lived in temporary accommodation centres in the Aceh Barat and Nagan Raya districts of Aceh.1 Local DHOs were occupied rebuilding a permanent health infrastructure, so temporary clinics had little support.
DHOs asked IOM to assist in monitoring temporary clinic operations. Through site visits, we learnt that the clinics lacked health promotion activities and maintained inconsistent stocks of supplies. Often health workers were newly trained, isolated and unsupervised. Unreliable financial arrangements between DHOs and the government agency for tsunami rehabilitation and reconstruction disrupted planning and contributed to high turnover of clinic staff. The absence of a health information system made it difficult to monitor service delivery and outcomes.
IOM proposed a balanced scorecard to document quality issues for each clinic and to serve as a management tool for DHOs and the Ministry of Health in this setting. A balanced scorecard is a set of simple measures used to describe and improve overall performance.2 Afghanistan and the Netherlands recently adopted balanced scorecards to improve their national health systems, largely because of the tool's ability to quickly identify problems and guide action. In these countries, measures for the scorecard were derived from large national databases.3,4 Such large-scale databases do not exist in Aceh. Instead, we conducted regular onsite surveys to generate information for the balanced scorecard, which then guided rapid cycle improvement efforts.5
Balanced scorecard
Development
In September 2006, IOM staff conducted initial assessments of temporary health clinics, concentrating on four domains: (i) health worker training; (ii) facility resources; (iii) community satisfaction and outreach and; (iv) service provision. In each area, we developed simple measures. Through discussions with district health officials and clinic staff, each measure was selected based on its significance, potential for improvement and ease of data collection (Fig. 1). We hoped to develop a sustainable model to be used by DHOs and local staff.
Two local IOM staff members, a nurse and a public health specialist, collected data through interviews, observations and basic assessments such as counting supplies and determining water availability. In the accommodation centres, the team sampled every fourth household. Data were combined to create composite measures using STATA, version 9 (StataCorp, College Station, United States of America).
The scorecard was presented in an easy-to-read table with each measure in a row and each clinic in a column. Measures were highlighted on the scorecard by colour to signify whether target levels were met: green (met target), yellow (partially met target) and red (below target). For example, the drug measure was green for clinics with more than 90% of essential drugs in stock, yellow for 9050% and red for < 50% (Fig. 1). Target levels were developed from focus group meetings with provincial and district health officials using national standards.
Implementation
The balanced scorecard was piloted in February 2007. Beginning in March 2007, it was implemented every 12 months in seven clinics and their nine associated temporary accommodation centres. The project was completed in December 2007, coinciding with the relocation of displaced residents to permanent housing and the closure of centres and clinics. Over the 10-month implementation period, a total of seven scorecards were created. During each cycle, data collectors from IOM spent 4 hours per clinic on clinical assessments and affiliated resident interviews.
Findings
Observations
Several improvements in health services were observed after the scorecard was implemented. Clinic staff received training on disposal of used needles, thermometer use and equipment sterilization. To focus on preventive medicine, clinic staff initiated events such as health promotion talks and fairs. Clinic staff also revitalized the community child health programme by conducting monthly visits to temporary accommodation centres, during which they gave immunizations and tracked the growth curves of children aged less than 5 years. Clinic staff began to actively manage an inventory of drugs. Communication improved between management, staff and accommodation centre residents.
Impact
Over the course of 10 months, there was a steady decline in the number of clinics reusing needles. After training, the score for proper thermometer use was excellent across all clinics. Community satisfaction increased from 68% to 84%. The percentage of residents receiving outreach activities increased from 14% to 85%. The number of sampled children aged less than 5 years with up-to-date vaccinations increased from 6% to 40%. The availability of essential drugs increased slightly from 49% to 59%. Equally important were the unmeasured benefits from increased monitoring and improved communications.
Discussion
The balanced scorecard was an effective management tool used every 2 months for rapid improvement in a disaster rehabilitation setting. Each time the scorecard was used, we expected to see improvements in areas that were previously below standard.
Limitations
In selecting the indicators, we balanced their importance with ease of measurement and implementation using existing resources. For example, while disposal of used needles is arguably not the most important indicator, it acted as a proxy for systems indicators such as staff knowledge and training. As clinic staff became more focused on the scorecard's output, the reported data became less reliable. For example, after training on thermometer use, all patients at one clinic recorded normal temperatures of exactly 37.0 ºC. The scorecard could not directly make up for deficiencies in staff training nor did it quickly change established practice patterns, such as over-prescription of antibiotics and steroid medications. It had minimal effect on external factors such as the drug supply chain.
Lessons and accomplishments
By measuring progress frequently and presenting results graphically, the scorecard provided a collaborative, simple, evidence-based tool to focus efforts on improving health services in a low-resource environment. The implementation in Aceh highlighted the following principles.
Evidence
A data set was produced and constantly updated to provide up-to-date information on clinic activities and outcomes. As data accumulated, the scorecards gave clinic coordinators greater capacity to direct changes. At follow-up meetings, based on the most current set of measures, clinic coordinators discussed problems with DHOs and IOM and developed targeted interventions. Clinic staff could view graphs showing data on supplies and staffing levels. IOM could support its recommendations to DHOs with data. DHOs became more aware of clinic-level activities and needs, and responded with more supervisory visits.
Simplicity
The scorecard benefited from efforts to reduce the number of measures, so as not to overwhelm users with the volume and frequency of new information. Initially, monthly data collection occurred faster than the pace of interventions, so we adjusted the frequency of data collection to every 2 months to better align with reasonable expectations for change. Clinic staff and district health officials found it helpful to review a manageably small set of critical indicators. The number of essential measures could be further pared down. Rather than using STATA, the creation of measures could be done through a simpler programme.
Management
Clinic coordinators and district level supervisors used the scorecard to assess quality, prioritize resources and direct change by identifying action points with limited additional investment in resources and data infrastructure. There was a shift of resources to target problem areas and redefine staff roles and accountability. Interviews were conducted during quiet times in the clinic so as not to interfere with clinical operations. The DHO reallocated previously-budgeted but unused time for monitoring clinical operations to this new management tool. This effort triggered discussions on quality in Aceh as there was a large variation between clinics. The scorecard can be used to monitor trends across facilities and highlight excellent clinics as role models to share their knowledge with other clinics.
Capacity building
Basic skills improved among the clinic staff. The data collection process provided a structured forum for IOM to interface with the community and clinic staff, facilitating opportunities for feedback and training beyond the scope of the study. For example, after evaluating thermometer use among clinic staff, IOM workers would immediately discuss basic thermometer use. Reviews of medical records revealed an over-prescription of antibiotics with no correlation with clinical signs. This led to on-site teaching regarding appropriate antibiotic use.
Communication
The scorecard enhanced communications between accommodation centre residents, clinic staff and DHO and IOM management staff. Through the process of data collection, IOM had the opportunity to interact with and teach clinic staff and accommodation centre residents. Regular meetings to discuss the findings of the scorecard brought clinics together and promoted the dissemination of ideas. By identifying priorities, clinic staff could better focus their attention and resources.
Leadership
Leadership was a large, unmeasured factor affecting clinic performance. Clinics with enthusiastic leaders were the ones initiating and advocating for quality improvement. More emphasis could be placed on developing the role of supervisors at the district level and on training clinic leaders.
Next steps
A balanced scorecard would be relevant to primary health-care facilities attempting to implement improvements with limited resources. Although beyond the scope of this project, the lessons learnt from the period of rehabilitation may contribute to the ongoing reconstruction of the permanent health infrastructure in Aceh (Box 1). The use of a balanced scorecard may be adapted to the permanent subdistrict health clinics. These lessons may apply to other primary health-care settings where there is a need for a simple data-driven method to focus priorities and a mechanism to improve communications between management, staff and beneficiaries. 
Acknowledgements
We thank the health officials at the ministry, provincial, district and community levels, especially Hasbi Quraisy, head of DHO Nagan Raya, and Amir Hamzah, head of DHO Aceh Barat. Special thanks to the clinic staff and shelter residents, Azrul Azwar, University of Indonesia, and to our colleagues at Harvard Medical School, especially Judith Palfrey, Myron Belfer and Kate Powis.
Grace J Chan was at Harvard Medical School when this work was carried out.
Funding: Funding was provided by AmeriCares, Harvard Medical School and the Lovejoy Research Fund.
Competing interests: None declared.
References
1. Temporary settlement monitoring project: findings in the field [Report no. 4]. Jakarta: Consortium for Assistance and Recovery toward Development in Indonesia; 2006.
2. Kaplan RS, Norton DP. The balanced scorecard measures that drive performance. Harv Bus Rev 1992;70:719. PMID:10119714
3. Peters DH, Noor AA, Singh LP, Kakar FK, Hansen PM, Burnham G. A balanced scorecard for health services in Afghanistan. Bull World Health Organ 2007;85:14651. doi:10.2471/BLT.06.033746 PMID:17308736
4. ten Asbroek AH, Arah OA, Geelhoed J, Custers T, Delnoij DM, Klazinga NS. Developing a national performance indicator framework for the Dutch health system. Int J Qual Health Care 2004;16(Suppl1):i6571. doi:10.1093/intqhc/mzh020 PMID:15059989
5. Brassard M, Ritter D. Problem-solving/process-improvement model. The memory jogger II, 1st ed. Salem, NH: Goal/QPC; 1994:115.
(Submitted: 9 March 2009 – Revised version received: 1 December 2009 – Accepted: 4 December 2009 )
* Correspondence to Grace J Chan (e-mail: gchan@jhsph.edu)