POLICY AND PRACTICE
A balanced scorecard for health services in Afghanistan
Carte de pointage équilibrée pour les services sanitaires afghans
Un cuadro de mando para los servicios de salud del Afganistán
David H PetersI,1; Ayan Ahmed NoorII; Lakhwinder P SinghIII; Faizullah K KakarIV; Peter M HansenI; Gilbert BurnhamI
IDepartment of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
IIBureau for Global Health, United States Agency for International Development, Washington DC, USA
IIIIndian Institute of Health Management Research, Jaipur, India
IVMinistry of Public Health of the Islamic Republic of Afghanistan, Kabul, Afghanistan
ABSTRACT
The Ministry of Public Health (MOPH) in Afghanistan has developed a balanced scorecard (BSC) to regularly monitor the progress of its strategy to deliver a basic package of health services. Although frequently used in other health-care settings, this represents the first time that the BSC has been employed in a developing country. The BSC was designed via a collaborative process focusing on translating the vision and mission of the MOPH into 29 core indicators and benchmarks representing six different domains of health services, together with two composite measures of performance. In the absence of a routine health information system, the 2004 BSC for Afghanistan was derived from a stratified random sample of 617 health facilities, 5719 observations of patientprovider interactions, and interviews with 5597 patients, 1553 health workers, and 13 843 households. Nationally, health services were found to be reaching more of the poor than the less-poor population, and providing for more women than men, both key concerns of the government. However, serious deficiencies were found in five domains, and particularly in counselling patients, providing delivery care during childbirth, monitoring tuberculosis treatment, placing staff and equipment, and establishing functional village health councils. The BSC also identified wide variations in performance across provinces; no province performed better than the others across all domains. The innovative adaptation of the BSC in Afghanistan has provided a useful tool to summarize the
multidimensional nature of health-services performance, and is enabling managers to benchmark performance and identify strengths and weaknesses in the Afghan context.
RÉSUMÉ
Le Ministère de la santé publique d'Afghanistan a mis au point une carte de pointage équilibrée (BSC) pour surveiller régulièrement les progrès de sa stratégie de délivrance d'un ensemble de services sanitaires de base. Bien que ce système soit souvent utilisé dans le domaine de la santé, il s'agit de sa première mise en æuvre dans un pays en développement. La BSC afghane est le résultat d'un processus collaboratif visant principalement à traduire la stratégie et la mission du Ministère de la santé sous forme de vingt-neuf indicateurs et indices de référence clés, représentant six domaines différents des services de santé, et de deux mesures composites de performances associées. En l'absence de système d'information sanitaire systématique, la BSC 2004 pour l'Afghanistan a été mise au point à partir d'un échantillon stratifié et randomisé, constitué à partir de 617 établissements de santé, de 5719 observations d'interactions prestateur/patient et d'entretiens avec 5597 patients, 1553 agents de santé et 13843 ménages. A l'échelle nationale, la BSC a révélé que les services de santé atteignaient davantage les populations les plus pauvres que celles moins déshéritées et bénéficiaient plus aux femmes qu'aux hommes, deux préoccupations importantes pour les pouvoirs publics. Des insuffisances graves ont toutefois été relevées dans cinq domaines, notamment les conseils aux patients, les soins obstétricaux pendant l'accouchement, la surveillance des traitements antituberculeux et la mise en place de personnel, d'équipements et de conseils sanitaires opérationnels dans les villages. La BSC a également permis de mettre en évidence de fortes différences de performances entre les provinces, aucune d'entre elles n'obtenant des résultats systématiquement meilleurs que les autres dans tous les domaines. L'adaptation de la BSC au système de santé Afghan fournit un outil novateur et intéressant pour évaluer de manière synthétique les performances multidimensionnelles des services sanitaires et permet aux gestionnaires un « benchmarking » des performances et d'identifier les points forts et les faiblesses des services sanitaires en Afghanistan.
RESUMEN
El Ministerio de Salud Pública (MSP) del Afganistán ha elaborado un cuadro de mando (CM) para vigilar periódicamente los progresos de la estrategia que ha diseñado para suministrar un conjunto básico de servicios de salud. Aunque utilizado frecuentemente en otros entornos asistenciales, es la primera vez que este tipo de sistema se emplea en un país en desarrollo. El CM se diseñó mediante un proceso de colaboración centrado en traducir la visión y la misión del MSP en 29 indicadores y criterios de referencia básicos representativos de seis dominios de los servicios de salud, junto con dos indicadores combinados del desempeño. A falta de un sistema de información sanitaria sistemática, el CM de 2004 empleado para el Afganistán se elaboró a partir de una muestra aleatoria estratificada de 617 centros de salud, 5719 observaciones de interacciones paciente-proveedor, y entrevistas con 5597 pacientes, 1553 agentes de salud y 13 843 hogares. A nivel nacional, se observó que los servicios de salud llegaban más a la población pobre que a la población menos pobre, y a las mujeres que a los hombres, por ser esos objetivos prioritarios del Gobierno. Sin embargo, se detectaron graves deficiencias en cinco dominios, sobre todo en lo que respecta al asesoramiento a los pacientes, la prestación de asistencia al parto, la vigilancia del tratamiento de la tuberculosis, la ubicación del personal y el equipo, y el establecimiento de consejos de salud de aldea operativos. El CM permitió identificar también amplias diferencias de desempeño entre las distintas provincias; ninguna provincia funcionaba mejor que las otras en todos los dominios. La novedosa adaptación del CM realizada en el Afganistán se ha revelado como un valioso instrumento para sintetizar el carácter multidimensional del desempeño de los servicios de salud, y está permitiendo a los administradores comparar el desempeño e identificar los puntos fuertes y las deficiencias en el contexto de ese país.
Introduction
Decades of conflict in Afghanistan have left its health system in ruins. Health conditions are among the worst in the world;1 access to basic health services is poor and the country is dependent on nongovernmental organizations (NGOs) to sponsor more than 80% of health facilities across the country.2 Although a routine system for gathering health information did not exist, recent studies indicated that Afghanistan has very low levels of antenatal and delivery care, as well as low coverage of most child health services.3,4
Shortly after the fall of the Taliban regime, the Ministry of Public Health (MOPH) identified its top priority as being to "strengthen the delivery of sustainable, quality, accessible health services, especially targeted at women, through planning for, and the effective and efficient implementation of, the basic health services package".5,6 Recognizing its limitations and the importance of NGOs, the MOPH pursued a strategy to rebuild services provided by the MOPH as well as contracting with NGOs. Donor organizations agreed to support public and NGO services through different funding mechanisms, but all following the same standards for the provision of the basic package of health services (BPHS).7
The MOPH was then faced with the challenge of monitoring the NGO contracts and its own facilities in the absence of a functioning health information system, and with few models for comprehensively monitoring performance of national health systems. In the absence of a routine system to collect information on health services, the MOPH chose to initiate a programme to monitor health services through household surveys and annual surveys of health facilities, and to use a balanced scorecard (BSC) to benchmark progress. There were obstacles to developing a BSC based on surveys, including the lack of a sampling frame, insecurity, bad roads and poor communications. Although the BSC has been used to manage performance in large and complex organizations,8 including the national health system in the Netherlands,9 it has never been applied at a national level in a developing country. This paper describes how the BSC for health services in Afghanistan was created, how it is used and the first results of the BSC in 2004.
Methods
The data for the BSC are taken from the National Health Services Performance Assessment (NHSPA), an annual assessment of service provision and patient perspectives, conducted by the MOPH with independent technical support. There are three levels of facilities that provide BPHS: basic health centres, comprehensive health centres, and the outpatient and maternity wings at district hospitals. Since these facilities differ in size, staffing levels, and population served, the sampling was stratified according to the level of facility, and the results for each province weighted according to the sampling fraction of facilities. The baseline NHSPA was conducted from June to October 2004, using a stratified random sample of all facilities providing the BPHS in each of the 33 provinces of Afghanistan (a 34th province (Daykundi) was added during the survey, but no health facilities existed there in 2004) up to a maximum of 25 facilities in each province. Observation of patient care was based on a systematic sample of interactions of adults and children with the main provider, with a target of five adults and five children selected in a systematic fashion using a random starting point and a sampling interval determined by the average number of new patients seen in a day. Each of the patients observed (or the caregiver, in the case of a child) was asked for an interview after they had completed their visit, outside the facility and out of sight of the health-care provider. They were asked about their perceptions of quality of care, satisfaction with services, and other information about the care received, and their household characteristics. Similarly, a random sample of health-care providers was selected for interview at each facility, stratified according to one of three types of health-care provider (doctors, nurses, community health workers). The final sample of the NHSPA included assessments of 617 health facilities (60% of all health facilities in the country), 5719 observations of patient care, 5597 patient interviews from those who were observed, and interviews with 1553 health workers. As part of the overall evaluation scheme, interviews were also conducted with women from 13 843 randomly selected households from a random selection of communities in the catchment area of the clinics, along with 74 focus groups in the same communities.
The Afghanistan BSC was designed via a series of workshops and discussions with the MOPH, NGOs, and other development partners active in the health sector, including front-line health workers and managers. The BSC was seen by the MOPH as a tool to translate its vision, values, and strategies into a practical form that demonstrates open and rational decision-making. It was important that the BSC have a limited number of easily understood and robust indicators that represented the most important aspects of service delivery. Although the NHSPA data could be used for analysis at several levels (clinics, districts, NGO health-care providers), it was determined that the main unit of analysis and national decisions would be at the provincial level. The rationale behind this decision was that public services and the government's contracts with NGOs are organized at the provincial level, and financial performance bonuses are attached to good performance based on the BSC. It was also determined that the BSC would be used to measure progress on an annual basis, and therefore would use data obtained from the health facility-based instruments of the NHSPA. It was expected that other surveys would provide service coverage and health status indicators that could be incorporated into the BSC for evaluation purposes, even if they were not available annually.
At the design workshops, six domains were identified for incorporation into the BSC:
1. patient perspectives
2. staff perspectives
3. capacity for service provision (structural inputs)
4. service provision (technical quality)
5. financial systems
6. overall vision for the health sector.
The stakeholders then assessed an initial list of 340 potential indicators for face validity and importance, along with considerations of reliability, completeness, outlying values, and variation. The short list of indicators were then assessed as a group to ensure there was a good balance among those assessing structures, processes, and outputs.
For each indicator, upper and lower benchmarks were set to indicate levels that are achievable in Afghanistan. The upper benchmark was set at a level that is currently being achieved by the six provinces with the best performance and the lower benchmark by 27 of the 33 provinces, roughly equivalent to the upper and lower quintiles, respectively. The one exception was the indicator for the presence of functional basic medical equipment, where the upper benchmark was set at 90% to be consistent with national policy.
After several iterations with key stakeholders on indicator definition and selection, an interim BSC was submitted to the Monitoring and Evaluation Advisory Board of the MOPH, which produced a final list of indicators.
Details on the definition and analysis of each indicator are reported in official reports10 and are also available from the authors. To simplify interpretation, all but two of the indicators are calculated as percentages. For the two indicators measuring equity (of outpatient services and satisfaction with care), concentration indices were used. To construct concentration indices, the household surveys were used to determine the economic levels of people in the catchment areas of the facilities for each province (three neighbouring provinces were pooled where there were insufficient households in the sample). The economic status of the sampled patients was estimated by constructing a wealth index from the patient's reported asset ownership and comparing this to the wealth index constructed from similar items from household members in the community survey. The techniques for analysing this involve principal component analysis of the household asset data to generate standardized asset scores.11,12 These scores were then applied to the patient's asset information. Total asset scores for each patient were calculated by summing assets. Concentration indexes were calculated from an assessment of users' scores in each province according to a method developed by Kakwani and colleagues.13
Results
The full set of national results is presented in Table 1 (the results for all 33 provinces are given in the Appendix, available at http://www.who.int/bulletin). To make the BSC easier to read, all results are colour-coded according to whether their results are in the top quintile (green), bottom quintile (red), or in between (yellow). Managers are then easily able to benchmark the status of one province against others to identify strengths and weaknesses. One of the most important findings is the wide variation in each indicator across the provinces, suggesting that each province has specific areas of concern.
The results for three indicators reflecting MOPH's overall vision (domain F) are quite encouraging at the national level. In the 2004 BSC, more females than males were seen as new outpatients in 23 of the provinces (median, 55% females), although in the most extreme provinces (Nimroz and Nuristan), only 40% of the outpatients were female. The median concentration index for outpatient utilization was negative, meaning that the poor were making proportionately more outpatient visits than the less poor. However, in 14 provinces, the distribution of outpatient visits was regressive (greater than 0). The concentration index for satisfaction with care was slightly regressive at the national level and in 21 provinces, meaning that higher levels of satisfaction with services were reported among the less poor than the poor.
Turning to patient and community perspectives (domain A), the median provincial rating of patient satisfaction was relatively high (median, 83% on the satisfaction scale), ranging from 62% to 98%. The provincial ratings for patient perceptions of quality was slightly lower (median, 76%), although with a similarly wide range of responses (54% to 92%). Relatively few communities (median, 34%) had active shura-e-sehie, the community health forums that are expected to plan and monitor local health activities in each community. Across provinces, the results for active shura-e-sehie ranged from none (in Badghis and Nimroz) to as high as 84% in Kapisa.
The results for staff perspectives (domain B) also raise causes for concern and action. Overall staff satisfaction levels were relatively low (64%), and although salary payments for more than three-quarters of staff were up to date, in six provinces, less than half of workers had been paid on time.
The indicators for the capacity for service provision show many areas for improvement, notably the low number having a tuberculosis registry (median, 16%), functional laboratories (median, 18%), and having a minimum number of staff in place (median, 39%).
The results on the technical aspects of health services delivery (domain D) were generally low. Few provinces achieved the BHC target of 750 outpatient visits per month (median, 22%), and the observed quality of care was particularly problematic. Few patientprovider interactions met the benchmark of 9 minutes (median, 18%). Although most providers took a complete patient history and carried out a physical examination (median, 71%), scores on the patient counselling index were very low (median, 30%). Scores on the financial systems (domain E) were higher, with most facilities having user fee guidelines (median, 91%) and providing exemptions for poor patients (median, 85%).
The composite scores for provinces meeting the upper benchmarks were relatively low. The six provinces with the best performance achieved the upper benchmarks for at least 31% of the indicators, with the highest levels found in Uruzghan (50%) and Bamyan (35%) provinces. This suggests that provinces perform relatively equitably according to the BSC, and that each of the provinces has considerable opportunity for improvement in meeting benchmarks that other provinces in Afghanistan are achieving. As expected, provinces were more consistent in achieving the lower benchmarks. All provinces met the lower benchmark for more than half the indicators, while three provinces (Bamyan, Kabul, and Takhar) achieved all but one of the lower benchmarks.
Discussion
The BSC has proved itself to be a useful tool for the MOPH, NGOs, and other stakeholders, and has become one of the cornerstones of the government's monitoring and evaluation system. It has provided a platform for standardization of the monitoring of results across different donor, NGO, and government health-care providers, allowing MOPH to be a more useful steward of the health sector. The development and use of the BSC has become a central part of a systematic effort to build the capacity of the MOPH, with a phased transition of responsibilities from technical assistance to the government.
The BSC has helped stakeholders to focus on particular provinces, as well as on specific areas for improvement. For example, as a result of the initial findings, the MOPH identified areas where the performance was lower than expected. As a result, the MOPH, donors and implementing agencies made it a priority to improve tuberculosis care and record-keeping, health-worker training and knowledge, shura-e-sehie activities, drug availability, laboratory capacity and use of clinical guidelines. During regular review of health projects at a national level, the MOPH uses the BSC scores as a major part of discussion. NGOs are also taking the scorecard as an objective assessment and as useful for informing their decisions, and its findings are incorporated into decisions on performance bonuses and continuation of contracts.
In addition to the national results, each province receives its own scorecard and that of the other provinces (see Appendix, available at http://www.who.int/bulletin). Provincial scorecards have proven very useful for the provincial health directors, who are using the information to identify areas of weakness and for establishing performance targets. Provincial health directors are also using the BSC in their meetings with the central government, where they say they can now take up the issues related to low-performing domains with the central ministry leadership. For example, a director with particular problems in staffing feels empowered to take up the issue of non-appointment of a health management information systems officer in his province, since his performance in the area of health-management information systems was low. The BSC provides him "with the evidence" to put pressure on the central MOPH to expedite filling the vacant posts in his province (personal communication).
The BSC gives a powerful indication of how provinces and the country are doing in delivering the basic package of health services, but it also has limitations. In the case of Afghanistan, one of its biggest limitations is that the BSC has relatively little information on health-service coverage or health-status outcomes, data which are not yet available in Afghanistan on a routine basis. In other countries where more data are available, it would be easier to accommodate information on health coverage and health status in a BSC. Another limitation is that the BSC is only measured at functional health facilities, and thus does not take into account places where the BPHS is not being provided. In other countries where the presence of facilities may be more stable, this is less of a problem.
The use of an international organization to provide independent monitoring services appears expensive at first analysis, costing nearly one million US dollars per year for the first 3 years. However, the activities financed go well beyond the BSC, and include training, technical assistance, and implementation of various pilot projects. They also include the start-up costs of establishing an office in the MOPH and providing security in Afghanistan, which are used for activities beyond the BSC. When considering that these expenditures cover nearly the entire public-sector expenditures for monitoring the implementation of the BPHS, the costs are modest. The monitoring and evaluation costs represent only 2.5% of the cost of delivering the BPHS. Exceptional overhead for transportation (expensive in Afghanistan) and additional personnel are included in these costs (for cultural reasons, both female and male data-collectors are needed where a single person could do the job in another setting), as are the variable costs of collecting data for the BPHS, which are now about 300 000 US dollars per annum. The running costs are minimal compared with the costs of implementing the BPHS, suggesting that even this type of intensive data collection exercise would be more feasible in other developing countries.
The BSC itself has some limitations in its ability to provide explanations for differences in performance. There are large differences in the conditions faced by each province of Afghanistan, which may partially explain the differences in their performance, but these potential determinants are not captured in the BSC. When interpreting the provincial BSC, is important to consider that results may be affected by factors beyond the control of health managers, e.g. security, preferences of the population, poverty, climate, and access to roads. NGOs working in insecure areas, for example, may have difficulty maintaining staffing levels at facilities or conducting supervisory visits, and thus may find it more difficult to achieve high scores on certain indicators than NGOs working in more secure provinces. In each province, managers should try, firstly, to determine why they are receiving the results that they achieve, and secondly, to learn from experience in their own province and in other provinces that are doing well, using the benchmarks as a guide. For national decision-makers, it will be important to take into account contextual variables such as geography and security.
In addition to examining why a province may perform the way it does, it is also important not to rely on a single BSC indicator to rate a province's performance.
Although the particulars of the BSC are unique to Afghanistan, use of this scorecard demonstrates a way to fill an important gap in health services management that is relevant to many countries. Afghanistan is a particularly difficult environment, where rebuilding trust in institutions and delivering health services are especially vital steps to rebuilding society and improving life for its citizens. The BSC contributes to these objectives by increasing transparency in the health sector and enabling health managers and policy-makers to identify and address areas of weakness. Moreover, by rejecting approaches to health-service evaluation that rely excessively on one type of indicator at the expense of others, the MOPH has laid the groundwork for the continued development of a well-balanced health sector that reflects patient and staff perspectives, capacity for service provision, technical quality, financial systems and the vision and values of the ministry. Using the BSC during subsequent rounds of the NHSPA to benchmark performance over time will enable the MOPH and stakeholders to identify trends in health-system performance, share information on context-specific best practices and jointly address areas of weakness, thereby advancing the goals of the ministry and serving as a model approach for other countries. 
Acknowledgements
This study was funded by a contract with the Afghanistan Ministry of Public Health and the Johns Hopkins University Bloomberg School of Public Health, in collaboration with the Indian Institute of Health Management Research. Important contributions to the collection, processing, and analysis of data as well as comments on the paper were provided by team members, including Aneesa Arur, Stan Becker, Vikas Dwivedi, Anbarasi Edward, SD Gupta, Shivam Gupta, Dr Haseeb, Dr Ismail, Maureen Mayhew, Melissa Opryszko, Krishna Rao, John Safapour, NK Sharma, Laura Steinhardt, Sandhya Sundaram, Kavitha Viswanathan, Earl Wall and other staff. We also thank the 350 staff who participated in data collection in the first round of the BSC. The suggestions from the editors and two anonymous reviewers are also gratefully recognized.
Conflict of interest: None declared.
References
1. World Health Organization. Working together for health: world health report 2006. Geneva: World Health Organization; 2006.
2. Management Sciences for Health, Health and Development Services, Management Sciences for Health/Europe. Afghanistan National Health Resource Assessment. Kabul: Ministry of Health, Transitional Islamic Government of Afghanistan; 2002.
3. Bartlett LA, Mawji S, Whitehead S, Crouse C, Dalil S, Ionete D et al. Where giving birth is a forecast of death: maternal mortality in four districts of Afghanistan, 1999-2002. Lancet 2005;365:864-70.
4. Johns Hopkins University Bloomberg School of Public Health, Indian Institute of Health Management. Afghanistan Multiple Indicator Cluster Survey 2003. A re-analysis of critical health service delivery indicators: report to the Afghanistan Ministry of Public Health. Kabul: Johns Hopkins University Bloomberg School of Public Health; March 2005.
5. Ministry of Health. Interim Health Strategy 2002-2003: A strategy to lay foundations. Kabul: Islamic Transitional Government of Afghanistan, February 2003.
6. Ministry of Health. A basic package of health services for Afghanistan. Kabul: Islamic Transitional Government of Afghanistan; March 2003.
7. Palmer N, Strong L, Wali A, Sondorp E. Contracting out health services in fragile states. BMJ 2006;332:718-21.
8. Kaplan RS, Norton DS. The Balanced Scorecard. Boston: Harvard Business School Press; 1996.
9. ten Asbroek AH, Arah OA, Geelhoed J, Custers T, Delnoij DM, Klazinga NS. Developing a national performance indicator framework for the Dutch heath system. Int J Qual Health Care 2004;16(Suppl 1):i65-71.
10. Johns Hopkins University/Indian Institute of Health Management, Monitoring and Evaluation Unit. Afghanistan health sector balanced scorecard national and provincial results round 1 (2004). Kabul: Ministry of Public Health, Policy and Planning Division; June 2005.
11. Filmer D, Pritchett LH. Estimating wealth effects without expenditure data or tears: an application to educational enrollments in states of India. Demography 2001;38:115-32.
12. Gwatkin DR, Rutstein S, Johnson K, Pande RP, Wagstaff A. Socioeconomic differences in health, nutrition and population. Washington DC: World Bank; 2000.
13. Kakwani N, Wagstaff A, van Doorslaer E. Socioeconomic inequalities in health: measurement, computation and statistical inference. J Econom 1997;77:87-104.
(Submitted: 13 July 2006 Final revised version received: 3 October 2006 Accepted: 20 October 2006)
1 Correspondence to Dr Peters (email: dpeters@jhsph.edu).