POLICY AND PRACTICE
Cost and effectiveness analysis of immunization service delivery support in Andhra Pradesh, India
Analyse du coût et de l'efficacité du soutien à la prestation de services de vaccination dans l'Etat de l'Andhra Pradesh, en Inde
Análisis de costo-eficacia del apoyo a la prestación de servicios de inmunización en Andhra Pradesh, India
Chutima SuraratdechaI,1; CBS Venkata RamanaII; Satish KaipilyawarIII; JVG KrishnamurthyIII; Srilatha SivalenkaIII; Naveena AmbatipudiIII; Sanjay GandhiIII; K UmashankarIII; James CheyneIV
IPATH, 1455 NW Leary Way, Seattle, WA 98107, United States of America
IIFamily Welfare, Government of Andhra Pradesh, Hyderabad, India
IIIPATH, Hyderabad, India
IVPATH, Ferney Voltaire, France
ABSTRACT
The immunization service delivery support (ISDS) model was initiated in Andhra Pradesh, India, in November 2003 with the aim of strengthening immunization services through supportive supervision. The ISDS model involves a well-established supervision system built upon the existing health infrastructure. The objectives of this approach are to: (1) identify areas of high performance and those that need improvement, (2) assist staff in identifying and correcting wrong practices, (3) improve staff skills, (4) motivate staff, and (5) initiate corrective actions at appropriate levels through information sharing. An evaluation of cost and effectiveness of ISDS in 16 districts that participated in the programme found that the incremental cost associated with three rounds of supportive supervision visits was approximately US$ 110 630 (US$ 36 877 per round). The performance of health centre and immunization sessions was evaluated using 43- and 28-point checklists, respectively, and demonstrated significant improvement during and following the two-year implementation of ISDS. The average percentage change in health centre performance scores from baseline to the fourth round of evaluation was approximately 36%, and immunization session performance scores increased by an average of 9%. The incremental costs per additional per cent increase in average health centre performance score and per additional per cent increase in average immunization session performance score over the evaluation period were estimated to be US$ 3091 and US$ 12 760, respectively. The incremental cost-effectiveness ratios are relatively sensitive to personnel and travel costs. Integration of ISDS into the Andhra Pradesh immunization system is projected to result in a 39% potential cost savings per round of supervision visit.
RÉSUMÉ
Le modèle de soutien à la prestation de services de vaccination (ISDS) a été mis en place dans l'Andra Pradesh (Inde) en novembre 2003, dans l'objectif de renforcer les services de vaccination par une supervision de soutien. Le modèle ISDS suppose l'existence d'un système de supervision bien établi au-dessus de l'infrastructure sanitaire existante. Cette démarche a pour objectifs : (1) d'identifier les domaines où les performances sont très satisfaisantes et ceux exigeant des améliorations, (2) d'aider le personnel à identifier et à corriger les pratiques inadéquates, (3) d'améliorer les compétences du personnel, (4) de le motiver, et (5) de lancer des actions correctives à des niveaux appropriés grâce au partage d'informations. Une évaluation du coût et de l'efficacité du modèle ISDS dans 16 districts ayant participé au programme a établi que le coût différentiel pour trois tournées de visites de supervision de soutien se situait autour d'US $ 110 630 (US $ 36 877 par tournée). Les performances du centre de santé et des sessions de vaccination ont été évaluées à l'aide d'une liste de contrôle en 43 et 28 points respectivement et des améliorations notables ont été relevées au cours et à la suite des deux années de fonctionnement de l'ISDS. En moyenne, la variation en pourcentage des scores de performances entre la période de référence et la quatrième tournée d'évaluation était d'environ 36 % pour le centre de santé et d'environ 9 % pour les sessions de vaccination. Les coûts différentiels par pourcentage supplémentaire d'augmentation du score de performances moyen du centre de santé et d'une session de vaccination ont été estimés respectivement à US $ 3091 et US $ 12 760 pour la période d'évaluation. Les ratios différentiels coût/efficacité sont relativement sensibles aux coûts de main-d'uvre et de déplacement. D'après les projections, l'intégration de l'ISDS dans le système de vaccination de l'Andhra Pradesh pourrait entraîner une économie de 39 % sur les coûts d'une tournée de visites de supervision.
RESUMEN
En noviembre de 2003 se puso en marcha en Andhra Pradesh, India, un modelo de apoyo a la prestación de servicios de inmunización (APSI) con la finalidad de fortalecer ese tipo de actividades con medidas de supervisión. El modelo de APSI integra un sistema de supervisión consagrado creado a partir de la infraestructura sanitaria existente. Este sistema tiene por finalidad: (1) identificar las áreas de alta eficacia y las que necesitan mejoras, (2) ayudar al personal a identificar y corregir las prácticas incorrectas, (3) mejorar las aptitudes del personal, (4) motivar al personal, y (5) poner en marcha medidas correctivas en los niveles pertinentes mediante el intercambio de información. La evaluación del costo y la eficacia del APSI en 16 distritos que participaron en el programa reveló que el costo adicional asociado a las tres rondas de visitas de supervisión de apoyo fue de aproximadamente US$ 110 630 (US$ 36 877 por ronda). La eficacia de los centros de salud y de las sesiones de inmunización se determinó mediante listas de verificación de 43 y 28 puntos, respectivamente, que pusieron de relieve mejoras considerables durante y después de la aplicación del APSI durante dos años. El cambio porcentual medio de la eficacia de los centros de salud entre la situación basal y la cuarta ronda de evaluación fue de aproximadamente un 36%, y la eficacia de las sesiones de inmunización aumentó como promedio un 9%. El costo marginal por punto porcentual adicional de la eficacia media de los centros de salud y por punto porcentual adicional de la eficacia media de las sesiones de inmunización durante el periodo de evaluación se estimó en US$ 3091 y US$ 12 760, respectivamente. La relación entre costo marginal y eficacia es relativamente sensible a los gastos de personal y de viaje. Se calcula que la integración del APSI en el sistema de inmunización de Pradesh Andhra puede traducirse en un ahorro del 39% por ronda de visitas de supervisión.
Introduction
In November 2003, an immunization service delivery support (ISDS) model was launched in Andhra Pradesh, India, with the aim of strengthening immunization services through supportive supervision. Traditionally, supervision of immunization practices in Andhra Pradesh is nonexistent or conducted minimally in an unstructured manner. Recognizing attitude problems as well as lack of coordination and transportation as common barriers to effective supervision, the Andhra Pradesh state government decided to introduce a three-year collaborative initiative with Program for Appropriate Technology in Health (PATH) in all 23 districts covering a population of 76 million.1
Supportive supervision is one of five essential components of the WHO's Reaching Every District strategy, an operational approach to reach 90% DPT3 (three doses of vaccine against Diptheria, Pertussis and Tetanus) coverage nationally and at least 80% DPT3 coverage in every district by the year 2005, the "90/80 goal" adopted by the GAVI Alliance.2 Supportive supervision involves on-site training by supervisors, which promotes quality at all levels of the health system by strengthening relationships within the system, focuses on the identification and resolution of problems, and helps to optimize the allocation of resources. The approach also promotes continuous improvements to the quality of care by providing leadership and support and by promoting high standards, teamwork and better two-way communication.3
Different approaches to supportive supervision to strengthen health worker performance and health services have been developed and documented. For instance, an integrated health package to guide essential health service delivery was developed by the United Republic of Tanzania Ministry of Health following health sector reform in 1999. This included a plan to bring a team of supervisors to district health facilities to evaluate how services are being delivered, to provide feedback and to conduct on-site training.4 Since implementation, supervisory visits occur more frequently and have become an opportunity for health workers to solve problems and learn additional skills. In 1999, EngenderHealth adapted a Client-Oriented, Provider-Efficient Services (COPE) model, a participatory supervision approach that includes self-assessment and performance plans, for use in child health services in selected study sites in Guinea and Kenya.5 Results show that supervision was better than before, supervisors were practicing new skills and following up on identified problems, and clients were very satisfied with the overall service received. The Honduran nongovernmental organization ASHONPLAFA introduced a new supervisory system and strengthened supervisory skills while reducing operating costs to meet sustainability goals. Strengthening supervision resulted in improved service quality and increased client satisfaction without increasing cost.3,4 A study conducted in rural Mexico concluded that supportive supervision and self-assessment of physicians significantly improved doctor performance and doctor-patient communication.6
The Andhra Pradesh model
The Andhra Pradesh ISDS model is a well-established immunization supervision system and builds upon the existing health infrastructure. The objectives of implementing this model are to: (1) identify areas of high performance and those that need improvement, (2) assist staff in identifying and correcting wrong practices, (3) improve staff skills, (4) motivate staff, and (5) initiate corrective actions at appropriate levels through information sharing.
With the total combined number of urban health centres and primary health centres ranging from 70 to 80 per district with only one district immunization officer (DIO) in each district, a supervision visit can accommodate only five to ten health centres (HCs). To establish an effective supportive supervision system, the ISDS model initially used both external and government supervisors to monitor immunization practices and to ensure quality immunization service delivery. The model is a combined intervention of the external consultant mode and medical college mode (Fig. 1). Twenty-one districts were included in the programme during the first two years of implementation. The external consultant mode was coordinated by designated persons at PATH and used a team of external consultants and regional coordinating officers (RCOs), retired state or district level health officers, to implement the model in 14 districts. In each supervision round, a team consisting of six to seven members, including PATH staff, external consultants and RCOs, performed four- to five-day site visits to all urban health centres and primary health centres in each district and to regular fixed and outreach immunization sessions. In the other seven districts, supportive supervision was provided by faculty and postgraduate students of medical colleges in coordination with RCOs and DIOs. In both interventions, the supervision team was accompanied by the DIO and other district programme managers or DIO-appointed supervisors to ensure capacity strengthening and sustainability. PATH provided on-the-job training, appropriate tools, and financial and technical support to the teams. Over the course of the programme, management responsibilities were gradually handed over to trained RCOs. As assistance from external consultants and PATH staff phased out, seven RCOs were assigned responsibilities for conducting supportive supervision in 14 districts and coordinating with the medical colleges in nine districts and all DIOs (Fig. 1).
Objectives
The overall objective of this paper is to assess the cost and effectiveness of the Andhra Pradesh ISDS model over a two-year period. Information on cost and effectiveness of the intervention is critical for decision-making and planning for sustainable integration of supportive supervision into the immunization system and replication in other states of India. Specific objectives include assessing cost and effectiveness of the model during the study period and estimating financial requirements for integration and expansion.
Methods
Two rounds of supervision visits per district were conducted within a 12-month period. The period between rounds provided an opportunity for divisional/district level monthly meetings to identify areas that need to be improved. During each round of supervision, information on activity costs and key indicators of immunization performance were collected. This paper bases its analysis on outcomes at baseline and after three rounds of supervision and costs incurred during three supportive supervision visits over an 18-month period. Baseline outcome data at all HCs in all districts were collected at the same time as the first introduction of supportive supervision activities.
Measurement of cost
All costs related to ISDS activities were based on the provider's perspective, interviews with programme staff, and labour and financial reports from November 2003 (the start of the programme) to May 2005 (the end of the third round of supervision visits). All financial support from PATH, the associated costs at PATH's Hyderabad office and of the Andhra Pradesh state government and field costs during the study period to implement the model were included. Daily activity time logs and activity reports kept during programme implementation allowed for tracking of intervention costs based on the actual staff time spent on supportive supervision activities.
Total costs were presented in US dollars at 2000 prices by using a gross domestic product deflator and exchange rate (45.55 rupees = US$ 1) and included wages and benefits of the project personnel involved with supportive supervision activities, printing of checklists, meals and incidental expense allowances, accommodation, transportation and administration. The estimates represent incremental costs of delivering the programme which are additional costs to the government that would not have been incurred otherwise. Salaries and benefits of health staff at district and HC levels and costs of recruiting external consultants were excluded. Fixed costs (such as housekeeping, capital costs and utilities) were not included because they were jointly incurred by other projects, and supervision activities were conducted at HCs. Costs associated with performance evaluations were minimal and not incremental to the programme, so therefore were not included in intervention costs.
Measurement of effectiveness
Performance data were collected using checklists. User-friendly checklists, which include 43 indicators on programme management, immunization sessions, immunization supplies, cold chain maintenance, immunization biowaste disposal and adverse events following immunization, were used in evaluating HC performance. Fixed and outreach immunization sessions were randomly observed and evaluated using a 28-point checklist to assess practices related to vaccine logistics, immunization practices and community mobilization activities (checklists are available upon request from the authors).
Following each session of performance evaluation, gaps were identified and discussed immediately. HC staff were observed and trained on-site, with feedback given to HC medical officers during the visit. Each day, supervisors consolidated forms and recorded all data from checklists into a computer database. At the end of each visit, feedback and HC performance measurements were provided to district officials, and district-wise aggregate data were then made available to state and district offices. Each point on the checklist was scored and weighted according to its importance (weights are available upon request from the authors), with a total possible score of 75 points. During the two-year period, supportive supervision activities occurred in 1724 HCs and 436 immunization sessions in 21 districts.
Cost-effectiveness analysis
Very few publications address the cost-effectiveness of peer training and supervision to enhance immunization services. One study evaluated an immunization training programme in Maluku, Indonesia, and concluded that it significantly improved the practice of vaccination and the immunization coverage rate.7,8 The average costs were about US$ 0.05 per additional reported dose and US$ 0.50 per additional fully immunized child. Applying guiding principles including monitoring and supervision to the management of primary health care services in Papua New Guinea resulted in increased immunization coverage from a range of 57% to 67% between 1980 and 1982, to 89% to 94% between 1983 and 1984.9 A systematic review of literature on cost and effect of a range of interventions for expanding the coverage of immunization services suggests that peer training is one of the least costly interventions when vaccine costs are not included.10
In this paper, cost effectiveness is defined as the ratio of incremental cost to the incremental effectiveness as measured by the percentage improvement in performance scores at HCs and immunization sessions. The ratio is expressed as US dollars per percentage increase in performance scores. Between each round of supervision, additional HCs were identified and included in the programme, and some HCs were integrated with others. To allow for comparison of HC performance scores measured pre- and post-intervention, the analysis is limited to cost and outcomes associated with HCs and districts that had participated in the programme from the start until the fourth round of performance evaluation. One-way sensitivity analyses were performed to assess the robustness of cost-effectiveness ratios.
Results
Cost
Since supportive supervision activities were conducted in a phased manner, there were 1116 HCs in 16 districts (Adilabad, East Godavari, Mahabubnagar, Nizamabad and Visakhapatnam were excluded from the analysis) with complete data on performance scores available during the study period. Total associated costs of supportive supervision were approximately US$ 110 630. Table 1 presents the cost of supportive supervision by item.
Travel and personnel expenses were cost drivers of activities and added up to 96% of the total costs. Approximately 46% (US$ 50 550) of the total costs were spent on transportation, vehicles and accommodations. Personnel costs (US$ 56 192) including per diem payments accounted for about half of the total cost. The training and retraining sessions conducted by PATH for external consultants, RCOs and medical college staff amounted to an average cost of US$ 1134.20 per round. The design and printing of 20 000 copies of checklists cost US$ 0.07 per set of checklists. The total cost per round of supervision was on average US$ 36 877 and ranged from US$ 22 119 to US$ 49 898. The average programme cost was US$ 6914 per district (US $99 per HC) or US$ 2305 per district per round of supervision (US$ 33 per HC per round of supervision).
Additional external consultants and RCOs were recruited to the supervision team over the course of the programme. As a result, wages and benefits increased dramatically in the second round of supervision. In round three, four districts (Chittoor, Krishna, Kurnool and Guntur) were transferred to the medical colleges mode, resulting in lower costs.
Effectiveness
Of the 1116 participating HCs in 16 districts, the average district-level HC performance score (maximum 75 points) was 46.73 (mode 57; median 48.5) at baseline, 54.97 (mode 63; median 58) after the first supervision visit, 60.10 (mode 70; median 62) after the second supervision visit and 63.45 (mode 70; median 62) after the third supervision visit (Table 2). The Pairwise t-test shows that the intervention produced statistically significant improvements in HC performance scores during the study period at the 99% confidence level. The majority of districts show significant increases in performance scores over time.
The average percentage change in performance scores between baseline and the fourth round of performance evaluation for all 16 districts was 36% (range: 9.81216.56) and statistically significant at 0.01 level. Fig. 2 depicts average scores of each round of evaluation and percentage change in scores between rounds.
The evaluation of immunization sessions was conducted at random during each round and included only a small number of sessions. The number of sessions visited in each district during each round was therefore inadequate to perform a statistical comparison. At the aggregate level, there was a statistically significant increase in the average performance score of immunization sessions from 58.85 points at baseline to 59.71 at the second round of evaluation (1.46%; not statistically significant) and to 63.96 points at the fourth round of evaluation (8.68%; P < 0.0001), from 59.71 at the second round to 62.57 at the third round (4.79%; P = 0.0005), and from 62.57 in the third round to 63.96 in the fourth round (2.22%; P = 0.08). Fig. 3 shows the overall percentage change in performance scores and average scores.
Incremental cost-effectiveness
It costs US$ 110 630 to significantly increase the mean performance score of HCs and immunization sessions by 36% and 9%, respectively from 47 and 59 points at baseline to 63 and 64 points in the fourth evaluation. This results in a programme cost of US$ 3091 (an average of US$ 193 per district) to increase mean HC performance scores by 1%. Similarly, it costs US$ 12 760 (an average of US$ 797 per district) to increase mean immunization session performance scores by 1% during the same period of time. Table 3 calculates incremental cost-effectiveness ratios and reports sensitivity of cost-effectiveness ratios to changes in cost components and discount rates. For each cost item (wages and benefits, transport and accommodation, training) and percentage change in performance scores, the maximum and minimum values from the cost data were selected to proxy the magnitude of potential variations. Costs for wages, benefits, per diem, transportation, vehicles and accommodation had a large impact on cost-effectiveness. A reduction in compensation and travelling costs could significantly improve the incremental cost-effectiveness ratio, however it is relatively insensitive to costs associated with training.
Financing immunization service delivery support
Table 4 describes the implementation costs associated with service delivery support if the ISDS model were integrated into the immunization system and covered all 23 districts of Andhra Pradesh. The integrated ISDS model would comprise seven RCOs and nine medical institutions. During each supervision round, seven RCOs would form a team of supervisors to visit all HCs in 14 districts twice a year. In the other nine districts, ISDS activities would be performed by medical colleges. The RCOs, medical colleges and DIOs would coordinate and be jointly responsible for implementing ISDS activities upon approval of an action plan by the state office. Continuing and integrating the ISDS programme into the state immunization system in 23 districts would cost US$ 34 896 (US$ 31 370 in 2000 US$) per round of supervision. Travel expenses are the highest proportion of cost associated with ISDS. Overall, the integration represents a potential savings of 38.71%, which results primarily from replacing wages of PATH staff and consultants and PATH-expense allowances for meals and incidental expenses with government wages and allowance rates for meals and incidental expenses.
Discussion and conclusions
This paper offers the first attempt to quantitatively evaluate the supportive supervision framework. Total costs of activities in 16 districts over the evaluation period were US$ 110 630 (US$ 36 877 per round). The statistical significance of results on effectiveness suggests that performance at HCs and immunization sessions have been greatly improved since the ISDS programme was launched. Travel and personnel expenses, specifically wages, meals and incidental expense allowances for appointed PATH staff, appeared to be the major cost drivers. The involvement of PATH staff in training and other contributions in the early stage of the programme suggests that replication in other states or other settings is probable, but would require a plan for technical assistance. A phase-out of PATH support and successful integration of ISDS activities into the immunization system, however, would result in potential savings of 38.71%, which may improve cost-effectiveness provided that performance scores are maintained or improved.
The analysis between each round of supervision visits provides valuable insight into the impact of the Andhra Pradesh ISDS model. Fig. 2 and Fig. 3 illustrate that the model is significantly effective in improving performance at HCs and immunization sessions. The average performance score of HCs consistently improved, and good performance was maintained or improved, as measured at subsequent supervision visits. Statistically significant regressive performance found in Chittoor and Kurnool perhaps resulted from the change of supervisors during the transition to medical college. Although Fig. 3 demonstrates that it is more difficult to improve immunization session scores, the improvement recorded after the second supervision visit is significant. The relatively slow improvement might be due to the use of indicators that were more closely related to the practice of immunization and the behaviour of immunization officers, and the performance score was good at the time the model was launched (average scores at baseline were at 58.85). Our findings underscore that continuation of supportive supervision is critical in strengthening and enhancing performance at HCs and immunization sessions.
The limitations of this study are those attributed to the limited data and scope of the programme and our analysis. The relatively small number of immunization sessions observed may not be representative. Immunization sessions are usually conducted two days per week with fixed timing. Each supervision visit could incorporate observation of a few sessions. Although checklists were introduced in the programme to minimize subjectivity of supervisors in rating performance, this issue was not assessed. Additionally, the analysis does not consider ongoing training, which may be required to improve knowledge and strengthen supervisors' capacity.
As there are no comparators, the implication of cost-effectiveness results is limited. Our findings thus provide a basis for a comparison to be made across settings and programmes aiming to improve performance of HCs and immunization sessions. Such comparisons will generate information on how cost-effective the intervention is and how it can be further improved. Similar approaches should be further explored and applied to other health services, such as primary care and maternal and child health, where supportive supervision can play a role in improving knowledge and transferring new skills to multiple and remote sites. Appropriate models should be identified and assessed to be applicable to the local context. The analysis on cost and effectiveness can be a useful tool in planning for sustainable intervention. 
Acknowledgements
The authors would like to acknowledge the comments provided by the anonymous reviewers. We are grateful to the staff of the government of Andhra Pradesh for their support. Special thanks go to Jodi Udd and Deborah Phillips for editing and manuscript preparation.
Competing interests: None declared.
References
1. Andhra Pradesh: building a model immunization system. PATH; 2004. Available from: http://www.childrensvaccine.org/files/CVP_AP.pdf
2. Supportive supervision [presentation]. Meeting of the GAVI Implementation Task Force on Supportive Supervision, Geneva, 1517 October 2002. WHO.
3. Marquez L, Kean L. Making supervision supportive and sustainable: new approaches to old problems, Maximizing Access and Quality (MAQ) Initiative paper No. 4, 2002. Washington, DC.
4. Guidelines for implementing supportive supervision: a step-by-step guide with tools to support immunization. Seattle, WA: PATH; 2003.
5. Bradley J, Igras S, Shire A, Diallo M, Matwale E, Fofana F, et al. COPE for child health in Kenya and Guinea: an analysis of service quality. New York: EngenderHealth; 2002.
6. Kim YM, Figueroa ME, Martin A, Silva R, Acosta SF, Hurtado M, et al. Impact of supervision and self-assessment on doctor-patient communication in rural Mexico. Int J Qual Health Care 2002;14:359-67.
7.Robinson JS, Burkhalter BR, Rasmussen B, Sugiono R. Low-cost on-the-job peer training of nurses improved immunization coverage in Indonesia. Bull World Health Organ 2001;79:150-8. PMID:11242822
8. Robinson JS, Burkhalter BR, Rasmussen B, Sugiono R. Evaluation of immunizer-training-immunizer program in Maluku, Indonesia: An on-the-job peer training approach to improving the performance of health workers. BASICS II; 2001.
9. Van Zwanenberg TD, Hull C. Improving immunization: Coverage in a province in Papua New Guinea. BMJ 1988;296:1654-6. PMID:3135060
10. Pegurri E, Fox-Rushby JA, Walker D. The effects and costs of expanding the coverage of immunization services in developing countries: A systematic literature review. Vaccine 2005;23:1624-35.
(Submitted: 6 December 2006 Revised version received: 19 June 2007 Accepted: 21 June 2007 Published online: 22 January 2008)
1 Correspondence to Chutima Suraratdecha (e-mail: csuraratdecha@path.org).
doi:10.2471/BLT.06.039495