RESEARCH

The immunization data quality audit: verifying the quality and consistency of immunization monitoring systems

Contrôle de la qualité des données de vaccination : vérification de la qualité et de la cohérence des systèmes de vaccinovigilance

Auditoría de la calidad de los datos de inmunización: comprobación de la calidad y coherencia de los sistemas de monitoreo de la inmunización

O. Ronveaux^I,1; D. Rickert^II; S. Hadler^III; H. Groom^II; J. Lloyd^IV; A. Bchir^V; M. Birmingham^I

^IVaccine Assessment & Monitoring, Department of Immunization, Vaccines and Biologicals, World Health Organization, 20 Avenue Appia, CH1211 Geneva 27 Switzerland
^IIImmunization Services Division, National Immunization Program, Centers for Disease Control and Prevention, Atlanta, GA, USA
^IIIGlobal Immunization Division, National Immunization Program, Centers for Disease Control and Prevention, Atlanta, GA, USA
^IVChildren's Vaccine Program, Program for Appropriate Technology in Health, Ferney Voltaire, France
^VSecretariat for the Global Alliance for Vaccines and Immunization, Geneva, Switzerland

ABSTRACT

OBJECTIVE: To evaluate the consistency and quality of immunization monitoring systems in 27 countries during 2002–03 using standardized data quality audits (DQAs) that had been launched within the framework of the Global Alliance for Vaccines and Immunization.
METHODS: The consistency of reporting systems was estimated by determining the proportion of third doses of diphtheria–tetanus–pertussis (DTP-3) vaccine reported as being administered that could be verified by written documentation at health facilities and districts. The quality of monitoring systems was measured using quality indices for different components of the monitoring systems. These indices were applied to each level of the health service (health unit, district and national).
FINDINGS: The proportion of verified DTP-3 doses was lower than 85% in 16 countries. Difficulties in verifying the doses administered often arose at the peripheral level of the health service, usually as the result of discrepancies in information between health units and their corresponding districts or because completed recording forms were not available from health units. All countries had weaknesses in their monitoring systems; these included the inconsistent use of monitoring charts; inadequate monitoring of vaccine stocks, injection supplies and adverse events; unsafe computer practices; and poor monitoring of completeness and timeliness of reporting.
CONCLUSION: Inconsistencies in immunization data occur in many countries, hampering their ability to manage their immunization programmes. Countries should use these findings to strengthen monitoring systems so that data can reliably guide programme activities. The DQA is an innovative tool that provides a way to independently assess the quality of immunization monitoring systems at all levels of a health service and serves as a point of entry to make improvements. It provides a useful example for other global health initiatives.

Keywords: Diphtheria-tetanus-pertussis vaccine/administration and dosage; Immunization programs/statistics; Data collection/standards; Quality control (source: MeSH, NLM).

RÉSUMÉ

OBJECTIF: Évaluer la cohérence et la qualité des systèmes de vaccinovigilance dans 27 pays pendant la période 2002-2003 à travers des contrôles de la qualité des données (DQA) normalisés, lancés dans le cadre de l'Alliance mondiale pour les vaccins et la vaccination.
MÉTHODES: On a estimé la cohérence des systèmes de notification en déterminant la proportion de troisièmes doses de vaccin diphtérie-tétanos-coqueluche (DTC3) rapportées comme ayant été administrées, dont l'administration peut être confirmée par des documents écrits tenus par les installations de santé et les districts. On a mesuré la qualité des systèmes de vaccinovigilance à l'aide d'indices de qualité s'appliquant aux différents composants de ces systèmes. Ces indices ont été appliqués à chaque niveau de service de santé (niveau de la formation sanitaire ou du district ou encore niveau national).
RÉSULTATS: La proportion de doses de DTC3 vérifiées était inférieure à 85 % dans 16 pays. Les difficultés pour vérifier les doses administrées se rencontrent souvent au niveau périphérique du service de santé et résultent habituellement de divergences dans les informations détenues par les formations sanitaires et les districts correspondants ou de l'impossibilité d'obtenir, auprès de ces formations sanitaires, des formulaires d'enregistrement remplis. Les systèmes de vaccinovigilance de tous les pays présentent des points faibles : incohérences dans l'utilisation des tableaux de contrôle, surveillance insuffisante des stocks de vaccins, matériel d'injection et manifestations postvaccinales indésirables, pratiques informatiques peu sûres et suivi insuffisant de l'exhaustivité et de la promptitude des notifications.
CONCLUSION: Les données de vaccination présentent des incohérences dans de nombreux pays, ce qui compromet leurs possibilités d'utilisation pour gérer les programmes de vaccination. Il faudrait que les pays utilisent ces résultats pour renforcer leurs systèmes de vaccinovigilance, de manière à ce que les données qu'ils fournissent puissent guider de façon fiable les activités des programmes. Le DQA est un outil novateur permettant d'évaluer de manière indépendante la qualité des réseaux de vaccinovigilance à tous les niveaux de service de santé et servant de point d'entrée pour introduire des améliorations. Il fournit un exemple utile pour d'autres initiatives mondiales dans le domaine sanitaire.

Mots clés: Vaccin diphtérie-tétanos-coqueluche/administration et posologie; Programmes de vaccination/statistique; Collecte données/normes; Contrôle qualité (source: MeSH, INSERM).

RESUMEN

OBJETIVO: Evaluar la coherencia y calidad de los sistemas de monitoreo de la inmunización en 27 países durante 2002–2003, usando para ello auditorías normalizadas de la calidad de los datos emprendidas en el marco de la Alianza Global para las Vacunas y la Inmunización.
MÉTODOS: La coherencia de los sistemas de notificación se estimó determinando la proporción de terceras dosis de la vacuna contra difteria, tétanos y tos ferina (DTP-3) cuya administración previamente notificada pudo verificarse mediante documentación escrita en establecimientos y distritos de salud. La calidad de los sistemas de monitoreo se midió usando índices de calidad para distintos componentes de los sistemas de monitoreo. Estos índices se aplicaron a cada nivel del servicio de salud (puestos de salud, distrito y ámbito nacional).
RESULTADOS: La proporción de dosis de DTP-3 verificadas fue inferior al 85% en 16 países. La comprobación de las dosis administradas tropezó a menudo con dificultades en la periferia del servicio de salud, generalmente como resultado de discrepancias entre la información de los puestos de salud y la de sus correspondientes distritos, o porque los puestos de salud no aportaban formularios de registro completos. Todos los países presentaban puntos débiles en sus sistemas de monitoreo, como por ejemplo el uso incoherente de gráficos de monitoreo; una vigilancia inadecuada de las reservas de vacuna, los suministros de inyección y los eventos supuestamente atribuibles a la vacunación o la inmunización; la baja seguridad de las prácticas de computación; y un escaso monitoreo de la integralidad y puntualidad de los informes.
CONCLUSIÓN: Los datos de inmunización adolecen de incoherencia en muchos países, lo cual limita la capacidad de éstos para gestionar sus programas de inmunización. Los países deberían usar estos resultados para fortalecer los sistemas de monitoreo de modo que los datos puedan orientar de manera fiable las actividades programáticas. La auditoría de la calidad de los datos es un instrumento innovador que permite evaluar independientemente la calidad de los sistemas de monitoreo de las inmunizaciones a todos los niveles de un servicio de salud y sirve de punto de entrada para introducir mejoras. Además, constituye un valioso ejemplo para otras iniciativas de salud mundiales.

Palabras clave: Vacuna difteria-tétano-pertussis/administración y dosificación; Programas de inmunización/estadística; Recolección de datos/normas; Control de calidad (fuente: DeCS, BIREME).

Introduction

The Global Alliance for Vaccines and Immunization (GAVI) was launched in 2000, and since then it has provided annual financial support to improve childhood immunization services in 52 developing countries through a performance-based grant programme (via the Vaccine Fund). GAVI allocates investment funds to all participating countries and then provides financial rewards based on a single indicator: the reported and independently verified number of children younger than 12 months of age who have been vaccinated with all three doses of diphtheria–tetanus–pertussis vaccine (DTP-3) (1, 2). Thus, an audit was needed to verify the quality of countries' reports of the number of children immunized with DTP-3.

Most countries track the performance of their immunization programmes through hierarchical administrative monitoring systems. In a typical system, staff at local health facilities compile vaccination data from daily immunization logs or tally sheets and report these to a district health officer monthly. Ideally, staff at both the health facility and at the district level use these reports to evaluate progress in achieving immunization coverage goals. The district officer compiles the coverage data from all facilities and reports them either monthly or quarterly to the national level. National level staff use these data to assess national and district performance and to compile annual reports that are submitted to WHO and UNICEF. When efficient, accurate and timely reporting occurs at each level of the hierarchy, administrative monitoring systems provide a strong basis from which to guide planning, review progress and determine which areas need additional efforts in order to cope with low-coverage or high drop-out rates (3). However, over the past 20 years, community-based surveys have sometimes reported coverage levels that were inconsistent with those reported by the administrative monitoring system. Evaluations of these systems identified problems with data quality and validity (4–6).

To verify the consistency of national reports based on administrative monitoring systems, WHO developed an evaluation protocol, known as the immunization data quality audit (DQA), using administration of three doses of the DTP vaccine before the age of 12 months as the sentinel indicator (7). The DQA also assesses the quality, efficiency, security and usefulness of the system at each reporting level to enable practical recommendations to be made for improving the system. In 2001, an independent consortium field-tested the DQA in 8 countries (8). A revised protocol was subsequently administered in all GAVI-supported countries receiving an initial investment greater than US$ 100 000 (7). This paper presents the findings of the 2002–03 DQA effort.

Methods

The DQA process

GAVI hired two independent companies to conduct DQAs according to a recommended protocol (7), which called for on-site evaluations at each level of the system, starting at the national level. The companies were trained by WHO. A multistage sampling procedure was used that included assessments of four districts selected by a probability proportional to the reported doses of DTP-3 administered as well as assessments in six health units in each of the four sampled districts. The chosen sample size, and hence the precision of the results, was dictated by logistical and financial considerations; the sample allowed for the maximum number of structures that could be visited by two evaluation teams within a 2-week period.

The DQA measures

The DQA reviews two key performance measures. One measure is the verification factor (VF), a district-based indicator of reporting consistency. For each district, the VF is calculated for the previous year's reported activity: first the DTP-3 vaccinations are re-counted from paper records in the selected health units; these are then divided by the number of DTP-3 doses found in district file records as reported by these health units. Second, the quotient found above is multiplied by the ratio (reported DTP-3 found at district level/reported DTP-3 found at national level) to account for any reporting differences between these latter two levels as follows:

with

i = district indicator (i = 1, 2, 3, 4) and j = health unit indicator (j = 1, 2, ..., 6)

and where

x_ij = the number of re-counted DTP-3 vaccinations found in the records of the j^th health unit of the i^th district
y_ij = the number of reported DTP-3 vaccinations from the j^th health unit of the i^th district
Rd_i = at the district level, the number of all DTP-3 vaccinations reported from all health units from the i^th district to the national level
Rn_i = at the national level, the number of reported DTP-3 vaccinations reported by the i^th district.

The national VF is calculated as the weighted average of district VFs.

A VF of < 1 indicates an inability to verify all of the doses of DTP-3 reported to have been administered (overreporting). Conversely, a VF > 1 indicates that a higher number of doses were recorded as being administered at peripheral health-service levels than are reflected in the number sent to more central levels (underreporting). To characterize reporting consistency at the level of vaccine delivery results from health units were classified into three categories: consistent if the ratio (re-counted DTP-3/reported DTP-3) was > 85% and < 115%; underreported if the ratio was > 115%; and overreported if it was < 85%. Health units who had overreported were classified further depending on whether the inconsistency was primarily due to missing health unit tally sheets or logs, discrepant tally sheets or logs (e.g., health unit data on DTP-3 doses administered were available but did not match what had been reported more centrally), or a mixed problem of missing and discrepant data.

The second key measure is the Quality Index (QI), a quantitative measure of the quality of each component at each level of the monitoring system. QIs are based on questions and observations at national level (53 questions), district level (38 questions) and health-unit level (31 questions) (7). The questions and observations in the QIs are grouped into five components: recording practices, storage and reporting practices, monitoring and evaluation, denominators used at district and national levels, and system design at the national level. To assess the proficiency of recording practices, workers at the health-unit level were asked to complete national immunization cards for 20 hypothetical children.

In calculating the QI scores, 1 point is given for each question answered correctly or task observed to have been performed correctly. Scores are calculated for each level of the health service and for each of the five components, with the number of correct answers and correctly performed tasks as the numerator and the number of answers and observations as the denominator.

Data analysis

VFs for the countries and districts were calculated with 95% confidence intervals based on a t distribution with m-1 degrees of freedom (where m is the number of clusters selected). To provide composite information on the specific strengths and weaknesses of the immunization reporting systems, we aggregated responses for each QI-component question or observation for all countries, and we present overall responses for key activities for health units, districts and countries.

To identify factors associated with high consistency within a system, zero-order Pearson correlation coefficients were calculated between national-level and district-level VFs and national-level, district-level, and health-unit-level QIs. To account for the fact that VF is not a linear concept, VFs > 1 were transformed by subtracting the verification excess (greater than 1) from 1. Descriptive statistics and bivariate tests for association between classes at each level (Fisher's exact and c² tests for trend) were calculated using SAS for Windows, release 8.02 (SAS Institute, Cary, NC, USA).

Findings

During 2002–03, DQAs were completed in 27 countries: 18 in Africa, 1 in the Americas, and 8 in Asia (Table 1). Four districts were assessed in each country, but in 11 countries the number of facilities visited was less than the 24 called for in the protocol (range = 14–23), either because one or more of the four selected districts had fewer than six health units or because security or logistical constraints precluded visiting some of them. Although QIs are provided for all 27 countries, VFs could not be calculated in two countries (Nigeria and Sudan) because of inadequately reported data at the national or district level.

Reporting consistency

National verification factors ranged from 0.40 to 1.06. Nine countries had VFs suggesting consistent data (> 0.85 and < 1.15); 7 had VFs suggesting moderate overreporting (0.70–0.84) and 9 had results that suggested considerable overreporting (< 0.70) (Table 1). Most national VFs had wide confidence intervals, although confidence intervals were narrower in countries with VFs that showed they had consistent data. District VFs ranged from 0.04 to 1.43, with 42% within the high range of verification (0.85–1.15). Inaccuracies primarily occurred at peripheral levels of the health service (e.g., discrepancies between data from health units and data from districts), except in five countries where discrepancies between district and national DTP-3 data also substantially reduced the national VF. In all VF categories we observed a wide range of reported national DTP-3 coverage levels; countries with low VFs tended to have lower DTP-3 coverage than countries with high or moderate VFs (Table 1). In countries with high national VFs, 73% of health units were classified as consistent; this proportion was only 31% (range = 21–53%) in countries with low national VFs.

Among the 557 health units assessed, 53% (296) had consistent data; 7% (38) had underreported DTP-3 doses; and 40% (223) had overreported (Fig. 1). Overreporting was most often attributed to data discrepancies in 22% (120) of health units and to missing data in 14% (80). Data were most often missing as a result of failure to retain daily tally sheets for the previous year, and in two countries (Cameroon and the Lao People's Democratic Republic), DTP-3 data were missing for more than 40% of health units. Overreporting and underreporting were found in some health units in almost all countries. These discrepancies may have occurred because data from outreach sessions went missing or there may have been errors in arithmetic or transcribing.

Quality of monitoring systems

Quality Index scores varied widely at all levels, and the overall analysis showed substantial deficiencies in all countries. The mean national QI for the 27 countries was 63% (Table 2). The mean district QI was 63% (range = 15–97%), and the mean health unit QI was 58%. However, these means masked wide variation in district and health-unit performance across and within countries; individual district QI values ranged from 15% to 97%, and individual health unit ranged from 4% to 100%. In most countries, we were able to identify districts and health facilities that performed well. Both district and health-unit QIs were significantly higher in countries with high national verification factors, and values decreased systematically with decreasing VFs (Table 2). In contrast, national QIs had no significant association with national VFs. There was no correlation between QI scores at the national level and those at the district level and health-unit level.

Mean QI scores for each component and level of the health system varied from 50% to 80% (Fig. 2). Of the five component groups of questions and observations in the QIs (recording, storage and reporting, monitoring and evaluation, denominators, and system design), the weakest one at all levels was "monitoring and evaluation". The strongest one (at national and district levels) was the "appropriate use of denominators" (used to calculate coverage rates). However, aggregating the results of each component for each level of the health service hides variations in results for each question and observation within each component (Table 3).

Among specific question and observation within the "recording practices" component, when health workers were asked to complete a series of vaccination cards for 20 hypothetical children, only 59% (292/495) were observed to complete all of them correctly. We also found substantial deficiencies in the monitoring of vaccine and injection supplies; proper monitoring is required to manage vaccine supplies and correctly calculate vaccine wastage. At the health-unit level, vaccine ledger books used to monitor vaccine and injection supplies were usually not up-to-date, and wastage could be calculated for only 32% (175/538) of facilities. Vaccine stock ledgers were better maintained at district level (68% (72/105) of district facilities had well maintained ledgers), and best at the national level (74% (20/27) of national facilities had well maintained ledgers). The monitoring of injection supplies also was best at the national level (77% (20/26) of national facilities compared with 56% (51/91) of district facilities and 24% (110/456) of health units), but it was not implemented at national or district level at all in three countries (Ethiopia, Guinea, Mali).

The greatest weaknesses among questions and observations in the "storage and reporting practices" component were found in the failure to retain forms, to develop procedures for handling late reports, to have procedures for computer back-up and data processing, and to have procedures for reporting adverse events. Recording forms (registers and tally sheets from the previous year) were fully available in only 60% (348/581) of health units. Procedures for timely reporting were well established at each level, but procedures for handling late reports were available in only 45% (47/105) of districts and 11% (3/27) of national offices. Computers were utilized for data processing in all countries at the national level and in one-third of districts, but procedures for back-up and data transfer were adequate in only a minority of cases. Procedures for reporting adverse events had been established in only 19% (5/27) of countries and 31% (21/67) of districts. In contrast, procedures for filing, dating and signing reports were adequate in 60–80% of health units and districts, and in more than 80% of national offices.

At all levels, the "monitoring and evaluation" component received the lowest scores. Among the questions and observations in this component, monitoring charts, which graphically display progress in reaching coverage targets throughout the year, were displayed in only 43% (248/572) of health units, although 63% (208/330) of units could provide the estimated number of children who needed to be vaccinated in the target population. Similarly, only 48% (52/108) of districts and 26% (7/27) of national offices displayed monitoring charts. Fewer than 50% of health units (81/258) and districts (18/44) monitored drop-out rates (i.e., children who did not receive the full series of vaccinations). Monitoring of completeness and timeliness of health-unit and district reporting was variable (occurring in only 41–50% of sites). Supervisory visits had been conducted within the past 4 months in fewer than 50% of districts (27/64) and health units (66/341); standard feedback formats were used in only 50% (55/108) of districts; and there was limited analysis and discussion of information: only 61% (14/23) of countries and 23% (15/64) of districts routinely gave feedback that contained analysis and discussion.

The main issues within the "appropriate use of denominators" component was the discrepancy between district numbers and national numbers of target populations in districts (observed in 65% of the countries). All countries were found to update their denominators annually. However, 18 countries (69%) reported some district-coverage values that were higher than 100%, an indicator that data sources used for denominators may be inaccurate, that children outside of the district(s) in question were being vaccinated, or that doses administered to children who were 12 months of age or older may have been recorded as doses administered to infants.

Within the final component – "system design" – immunization reporting was integrated within comprehensive health information systems in 13 (50%) countries. In addition, the design of monthly reporting forms was usually appropriate (according to WHO recommendations), with only one country not reporting vaccinations of infants (children younger than 12 months) separately from those of older children. All countries reported tetanus toxoid immunizations for pregnant women separately.

Correlation analyses

The national verification factor showed a strong correlation with district and health-unit QIs (Pearson's correlation coefficients of 0.41 and 0.50, respectively; P < 0.001), but showed no correlation with national QIs. District VFs also correlated strongly with district and health-unit QIs (Pearson's correlation coefficients of 0.40 and 0.48, respectively; P < 0.0001) (Fig. 3). This suggests that the quality of reporting systems in districts and health units are important predictors of consistency in reporting systems in countries.

Discussion

The data quality audits conducted in 27 countries in 2002–03 provide a comprehensive evaluation of immunization monitoring systems. This is the first time a standard method has been applied to assess monitoring systems for a public health intervention in developing countries. The DQA provides a quantitative indication of reporting consistency and quality, and this can facilitate comparisons of results over time or place and motivate those who are monitoring the system to make improvements in monitoring activities (9). The DQA also provides information about which level of the health service contributes most to inaccuracies, and it diagnoses specific weaknesses in the monitoring system that, if addressed, could improve its precision, efficiency, security and usefulness.

This analysis identifies important challenges that must be faced in order to improve immunization monitoring systems. It indicates that corrective efforts must be focused at peripheral levels (districts and health units) if quality and consistency are to be improved. The DQAs reveal a particular need to improve the skills and practices of those involved in the analysis and use of data at all levels of the health service to help guide strategies to increase coverage, manage vaccine supply and monitor vaccine safety. Previous studies have shown that immunization staff in provincial and local areas have weak skills in using quantitative immunization data (10). Efforts to improve data analysis and use at the local level could in turn stimulate improvements in the accuracy of the data collected because staff may take an interest in their own data and value the opportunity to demonstrate local achievements and guide local planning. The wide variation of QIs at all levels reinforces the need to examine the information from each level independently.

We did not observe a significant association between immunization coverage levels and VFs, although data verifiability tended to be lower in countries with lower coverage. This suggests that all countries, regardless of coverage levels, may have substantive inconsistencies in their reporting systems and would benefit from systematic assessments such as the DQA.

Community-based surveys have traditionally been used to verify reported vaccine coverage and obtain point estimates of immunization coverage levels; they have also revealed substantive inaccuracies in administrative monitoring systems (11). Coverage surveys are not, however, a substitute for administrative reporting and the timely monitoring of programme effectiveness at the community level. Coverage surveys typically provide information on birth cohorts from previous years and therefore do not provide the continual flow of information needed for local programme management. Coverage surveys also vary in precision, and their data may be subject to recall bias and quality problems (12). Furthermore, they do not provide information on the quality of the monitoring system or identify the cause of inaccuracies. Therefore, administrative monitoring systems are needed to provide critical information on an ongoing basis to local staff in order to determine whether coverage targets are being met. WHO has developed a data quality self-assessment tool (DQS) for use by district-level staff to help them identify problems in their monitoring system. An abbreviated version of the DQS can be integrated into rapid assessments or supportive supervision visits at the district level and health-unit level. In addition, WHO and its partners are promoting the Reaching Every District approach (13), one component of which aims to strengthen district monitoring capacity and to promote the use of local data in planning.

The DQA has two important limitations. The first is the imprecision of the verification factor in validating the reported number of children receiving three doses of DTP. Due to the small sample size and a large variation in VFs among districts in the same country, average confidence intervals of national VFs were wide (averaging ± 30%). Greater precision in VFs could be achieved by increasing the size of the sample, but there would be a trade-off in terms of increases in cost and the time necessary to conduct the audit. A second weakness is that verification of immunization at the point of recording a dose as being administered is not addressed in the current DQA; the audit tool therefore may miss doses that were recorded but were not administered or vice versa. Community-based verification is necessary if this type of inaccuracy is suspected.

Through the DQA process, GAVI has invested in strengthening existing monitoring systems by providing both diagnostic information and incentives to countries to improve their systems. That countries have accepted this process is evidenced by the many countries that plan to improve their systems and in the substantial improvements in DQA results seen in several countries since the initial audit in 2001.

Conclusion

The DQA is thus an innovative tool that meets its two main objectives: it serves as a diagnostic tool for a country's immunization monitoring system by providing practical recommendations as to how it could be improved. It also provides critical, objective and independently collected information to the donor community on the quality and consistency of the measure that is used to determine rewards. Because of the intensive international efforts, resources and funds that are being directed towards supporting the Millennium Development Goals, there is an urgent need for accurate and timely health data to be collected on an ongoing basis (14). It is, therefore, important to improve the quality and usefulness of relatively low-cost, pre-existing monitoring systems within developing countries (15). The challenge now is to use the findings from the DQA to strengthen monitoring systems so that their data can reliably guide activities to reduce morbidity and mortality from diseases that vaccines can prevent.

Acknowledgements

We would like to thank Linda Archer who initially designed and field-tested the DQA tool, Elizabeth Zell for technical assistance in the development of the DQA, and Stacy Woodard for statistical expertise.

Funding: The Data Quality Audits were financed by the Global Alliance for Vaccines and Immunization.

Competing interests: none declared.

References

1. Wittet S. Introducing GAVI and the Global Fund for Children's Vaccines. Vaccine 2000;91:385-6.        

2. Brugha R, Starling M, Walt G. GAVI, the first steps: lessons for the Global Fund. Lancet 2002;359:435-8.        

3. World Health Organization. Immunization in practice: a practical resource guide for health workers. 2004 update. Geneva: WHO; 2004. WHO document WHO/IVB/04.06.        

4. Onta SR, Sabroe S, Hansen EH. The quality of immunization data from routine primary health care reports: a case from Nepal. Health Policy and Planning 1998;13:131-9.        

5. World Health Organization. WHO vaccine-preventable diseases: monitoring system. Geneva: WHO; 2003. WHO document WHO/V&B/03.20.        

6. Murray CJL, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination coverage in 45 countries. Lancet 2003;362:1022-7.        

7. World Health Organization. The immunization data quality audit (DQA) procedure. Geneva: WHO; 2003. WHO document WHO/V&B/03.19.        

8. LATH Consortium. Immunization data quality audit evaluation report: final report from Deloitte Touche Tohmatsu Emerging Markets, Euro Health Group, and Liverpool Associates in Tropical Health. Geneva: WHO; 2001.        

9. Weeks RM, Svetlana F, Noorgoul S, Valentina G. Improving the monitoring of immunization services in Kyrgyzstan. Health Policy and Planning 2000;15:279-86.        

10. Loevinsohn BP. Data utilization and analytical skills among mid-level health programme managers in a developing country. International Journal of Epidemiology 1993;23:194-200.        

11. Henderson RH, Sundaresan T. Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bulletin of the World Health Organization 1982;60:253-60.        

12. Boerma JT, Sommerfelt AE. Demographic and health surveys (DHS): contributions and limitations. World Health Statistics Quarterly 1993;46:222-6.        

13. WHO Regional Office for Africa. Building capacity to reach every district with immunization services in the African region: status and perspectives. Afro EPI-Newsletter, Vaccine Preventable Diseases Bulletin 2003;33:1-3. Also available from: http://www.afro.who.int/ddc/vpd/bulletins/2003/mar2003.pdf        

14. Lee JW. Global health improvement and WHO: shaping the future. Lancet 2003;362:2083-8.        

15. United Nations Development Programme. Millennium Development Goals. Available from: http://www.undp.org/mdg/        

(Submitted: 28 May 2004 – Final revised version received: 16 December 2004 – Accepted: 14 January 2005)

1 Correspondence should be sent to Dr Ronveaux at this address (email: ronveauxo@who.int).