POLICY AND PRACTICE
The costs, effects and cost-effectiveness of strategies to increase coverage of routine immunizations in low- and middle-income countries: systematic review of the grey literature
Coûts, effets et rapport coût-efficacité des stratégies destinées à augmenter la couverture des vaccinations de routine dans les pays à faible revenu et à revenu intermédiaire : revue systématique de la littérature grise
Costos, efectos y costoeficacia de las estrategias de aumento de la cobertura de la inmunización sistemática en los países de ingresos bajos y medianos: examen sistemático de la literatura gris
Katherine BattI; J.A. Fox-RushbyII,1; Marianela Castillo-RiquelmeIII
IResearch Assistant, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, England
IISenior Lecturer, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, England (email: julia.fox-rushby@lshtm.ac.uk)
IIIResearch Fellow, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, England
ABSTRACT
Evidence-based reviews of published literature can be subject to several biases. Grey literature, however, can be of poor quality and expensive to access. Effective search strategies also vary by topic and are rarely known in advance. This paper complements a systematic review of the published literature on the costs and effects of expanding immunization services in developing countries. The quality of data on the effectiveness and cost-effectiveness of strategies to increase immunization coverage is shown to be similar across literatures, but the quality of information on costing is much lower in the grey literature. After excluding poorer quality studies from this review we found the quantity of available evidence almost doubled, particularly for more complex health-system interventions and cost or cost-effectiveness analyses. Interventions in the grey literature are more up to date and cover a different geographical spread. Consequently the conclusions of the published and grey literatures differ, although the number of papers is still too low to account for differences across types of interventions. We recommend that in future researchers consider using non-English keywords in their searches.
Keywords: Immunization/economics; Immunization programs/economics; Costs and cost analysis; Cost-benefit analysis; Publications/classification/standards; Information storage and retrieval/methods; Comparative study; Review literature (source: MeSH, NLM).
RÉSUMÉ
Les revues portant sur les articles publiés peuvent être sujettes à plusieurs biais. En revanche, la littérature grise peut être de qualité médiocre et d'accès coûteux. Les stratégies de recherche efficaces diffèrent aussi selon le sujet et sont rarement connues à l'avance. Le présent article complète une revue systématique des articles publiés sur les coûts et les effets d'une extension des services de vaccination dans les pays en développement. La qualité des données sur l'efficacité et le rapport coût-efficacité des stratégies destinées à augmenter la couverture vaccinale est similaire qu'il s'agisse d'articles publiés ou non, mais la qualité de l'information sur les coûts est nettement plus faible dans les articles non publiés. Après exclusion des études de qualité médiocre, nous avons trouvé que la quantité de données disponibles avait pratiquement doublé, notamment en ce qui concerne les interventions complexes sur les systèmes de santé et les analyses de coût ou de coût-efficacité. Les interventions décrites dans la littérature grise sont plus récentes et ont une couverture géographique différente par rapport aux études publiées. Par conséquent, les conclusions des deux types d'articles diffèrent, bien que leur nombre soit encore trop faible pour rendre compte des différences entre les divers types d'interventions. Nous recommandons aux chercheurs de penser à inclure des mots-clés non anglais dans leurs futures recherches documentaires.
Mots clés: Immunisation/économie; Programmes de vaccination/économie Coût et analyse coût; Analyse coût bénéfice; Publication/classification/normes; Stockage et recherche information/méthodes; Etude comparative; Revue de la littérature (source: MeSH, INSERM).
RESUMEN
En las revisiones basadas en la evidencia de la literatura publicada pueden influir sesgos de diverso tipo. La literatura gris, por otro lado, puede ser de calidad deficiente y difícil acceso. Además, la eficacia de las estrategias de búsqueda depende del tema y es difícilmente previsible. Este artículo complementa una revisión sistemática de la literatura publicada sobre los costos y los efectos de ampliar los servicios de inmunización en los países en desarrollo. Se muestra que la calidad de los datos sobre la eficacia y la costoeficacia de las estrategias de aumento de la cobertura vacunal es similar en los dos tipos de literatura, pero la calidad de la información sobre el cálculo de costos es mucho menor en la literatura gris. Tras excluir de esta revisión los estudios de peor calidad, la evidencia disponible casi se duplicó, en particular para las intervenciones de sistemas de salud o los análisis de costos o costoeficacia más complejos. Las intervenciones halladas en la literatura gris están más actualizadas y tienen una cobertura geográfica distinta. En consecuencia, las conclusiones de los artículos publicados y de la literatura gris difieren, pero el número de artículos es todavía demasiado bajo para poder fundamentar las diferencias entre los distintos tipos de intervención. Recomendamos que en el futuro los investigadores consideren la posibilidad de usar en sus búsquedas palabras clave no inglesas.
Palabras clave: Inmunización/economía; Programas de inmunización/economía; Costos y análisis de costo; Análisis de costo-beneficio; Publicaciones/clasificación/normas; Almacenamiento y recuperación de la información/métodos; Estudio comparativo; Literatura de revisión (fuente: DeCS, BIREME).
Introduction
Immunization against the six diseases included in the traditional Expanded Programme on Immunization (EPI) schedule (poliomyelitis, diphtheria, pertussis, tetanus, measles and tuberculosis) is one of the most cost-effective ways to improve child health (1). However many countries have not achieved the expected coverage levels (2), and even those reaching 80% experience much variation between districts. For example, in India in 199293, 71.0% of the richest children were protected from measles whereas only 22.8% of the poorest 20% of children received such protection (3, 4). Therefore, arguments to increase efficiency and equity can drive policy on increasing coverage rates.
A recent review of published literature (5) suggested that the most effective strategies to increase coverage were community health workers and door-to-door canvassing, and the least costly were peer training and door-to-door canvassing. However, the conclusion was drawn cautiously because there was little evidence (60 papers); the quality of evidence was poor (only 29 interventions once the lowest quality papers were excluded); the literature was old (24/29 interventions were evaluated in the 1980s); and all interventions had not been evaluated. Therefore, the authors recommended searching the grey literature to increase the evidence base.
Grey literature has been defined as "a range of published and unpublished material which is not normally identifiable through conventional methods of bibliographic control" (6). Searching the grey literature can broaden the evidence base, overcome publication bias (7, 8), and highlight ongoing research. However, searches of grey literature are labour intensive, and efficient strategies vary by topic (9, 10). Because this literature may not be peer reviewed there is also debate over its potential value.
This paper outlines the methods used to search, select and review grey literature on the cost, effectiveness and cost-effectiveness of strategies to improve immunization coverage. The results document both the quality of the literature and the substantive evidence found. The discussion compares the quality, quantity and nature of evidence in the grey literature and the published literature and reflects on the most effective approaches to searching the grey literature on this topic.
Methods
Search strategy
The search strategy focused on literature up to May 2003 and included:
hand searches in institutional documentation centres within WHO, Geneva; UNICEF, New York; the Pan American Health Organization (PAHO), Washington, DC; and the US Agency for International Development (USAID) Experience Clearinghouse, Washington, DC;
interviews with 28 international experts (from The World Bank, UNICEF, Basic Support for Institutionalizing Child Survival, stage II [BASICS II], the US National Institutes of Health, PAHO, WHO country offices and EPI programme managers) who helped identify documents and work in progress;
a review of databases using the mix of free text and MESH terms employed by Pegurri et al. (5). The databases reviewed were the System for Information on Grey Literature in Europe (SIGLE), Eldis, Health Technology Assessment (HTA) at the Centre for Review and Dissemination at the University of York, Popline, CAB abstracts and regional WHO databases;
a three-way search of the Internet including free text searches in Google using the same keywords and "similar pages"; a search of conference proceedings, such as the American Public Health Association and the CHANGE Project; a search of web pages of international organizations, bilateral agencies, nongovernmental organizations (NGOs), consultancy firms and universities involved in the delivery, funding or evaluation of immunization services (such as the Global Alliance for Vaccines and Immunization, the UK Department for International Development, BASICS II, the Program for Appropriate Technology in Health (PATH), Tearfund and the Johns Hopkins School of Public Health).
Selection methods
The selection criteria mirrored those of Pegurri et al. (5, 11) (Appendix A, web version only, available at: http://www.who.int/bulletin) with one difference: because the grey literature arose in a wide variety of non-standard formats (e.g. country evaluations, PowerPoint presentations), the criteria that had been used to exclude letters and other materials were relaxed. In some cases, different sources of information were eligible for inclusion when combined together.
Nearly 2000 titles were identified, and this led to a review of 238 abstracts. Of these, 88 documents appeared eligible for review using the inclusion criteria, although a more detailed check (1-page review) reduced the eligible documents to 34. Table 1 outlines the findings from the different search and selection strategies. Of those documents selected for inclusion in our review, 22 originated from interviews with experts and searches of libraries and document centres in the United States; eight came from searches of electronic sources; one came from WHO; and three from other sources.
Review methods
The 34 papers selected are listed in Appendix B. Each document was reviewed using Pegurri et al.'s data extraction form (available on request from the corresponding author). This form included questions about background information, descriptive data on the intervention or alternatives to it and the context, methods of evaluation used, results and generalizability. The extracted information also formed the basis for the quality assessment and scoring of the study's internal and external validity, using the same criteria as Pegurri et al. (5).
Findings
The 34 documents selected for inclusion represent 26 different interventions. Of these studies, 24 presented data on effectiveness; 15 presented data on cost; and 5 presented data on cost-effectiveness. All 24 effectiveness studies adopted an ecological design using a single time series (n = 13), multiple time series (n = 10) or a multi-group design in one time period (n = 1).
Two cost analyses did not present effectiveness data and were thus "new" studies, increasing the total number of new studies to 26. Thirteen effectiveness studies also included cost data; of these, three included cost-effectiveness data. Of the remaining two cost-effectiveness studies, one study's effectiveness data was rejected although its cost data were accepted, and another had only its cost and cost-effectiveness data reviewed here because the effectiveness had already been reviewed by Pegurri et al. (5).
Of the interventions, 11 evaluated effectiveness only; 10 evaluated costs and effectiveness separately; 3 studied effectiveness, costs and cost-effectiveness; and 2 assessed costs and cost-effectiveness.
Evaluation of the quality of evidence
Studies about effectiveness and costs had more limitations when compared with cost-effectiveness analyses (Table 2, web version only, available at: http://www.who.int/bulletin). Following a review of the quality of studies, we found that of the 24 effectiveness analyses, 15 had fewer than 4 limitations; of the 15 cost analyses, 6 had fewer than 8 limitations; and of the 5 cost-effectiveness analyses, 3 were taken forward to be considered. Altogether, 24 interventions are considered in this paper.
Background and description of interventions
Most interventions were evaluated in Africa (n = 11, 9 of which were evaluated in west Africa); the second most common geographical area was Central and South America (n = 7) followed by the Middle East (n = 3), south-east Asia (n = 2) and the Pacific (n = 1). Most interventions (15/24) were evaluated as a national approach; all cost analyses fell into this category. Only effectiveness analyses focused on a subnational level. Evaluations took place during the 1980s (n = 10) and 1990s (n = 10) and 11 had taken place since 2000. Three documents were undated.
Most interventions targeted demand and supply, and this was true for all evaluations of costing (Table 3, web version only, available at: http://www.who.int/bulletin). Supply interventions focused on endowing providers with increased responsibility and accountability by changing payments to performance-based systems and contracting with NGOs at varying levels of responsibility to provide services to a designated health-care population. Demand-side approaches were aimed at educating communities and health workers, improving awareness of missed opportunities, and reducing the opportunity costs to families (shortening waiting times).
Interventions that addressed both supply and demand fell into two broad categories: mass campaigns and overall system activities. Mass campaigns were generally organized from a central level, relying on recent census data to determine population numbers and "divide" the country or region into areas for individual vaccinators or teams. This was followed by recruitment, training and mobilization of a vaccination force along with mobilization of vaccination supplies. Distribution often involved other ministries, the armed forces and the private sector. To encourage people to attend vaccination days, communication strategies were launched using the press, radio and television.
The reorganization of systems in different evaluations often shared similar characteristics, e.g. improvement of logistical and/or technological systems and the enhancement of training in conjunction with improving vaccination distribution. Changes were creative, with some focus on problem solving and participative decision-making. They included overhauls of political and administrative boundaries and adoption of a myriad of district-specific strategies.
Costs, effects and cost-effectiveness of strategies
Fig. 1 shows that most interventions increased the coverage of fully vaccinated children (FVC). From an average baseline coverage of 44% (standard deviation (SD) = 21, range = 1784) the average percentage point increase was 20 (SD = 19, range from 8 to +55). Coverage fell following two interventions: by 0.2 percentage points in the Philippines after introduction of the Integrated Supervisory Checklist and by 7.5 percentage points in Guatemala after introduction of the National Plan of Action. The interventions with the highest impact on full coverage (>30 percentage point increase from baseline) were health education campaigns, changes in provider or payer model, and mass campaigns.
Table 4 shows the average cost of the interventions. We found only one direct comparison of average costs per fully vaccinated child between a mass campaign and routine programme (Cameroon); the cost per dose delivered per fully vaccinated child during the mass campaign was 49 times higher than the cost through routine services (dependent on financial/opportunity costing).
Table 5 shows that the relative weight of cost inputs varied among interventions. Personnel were usually the most expensive line item; their cost varied between 253%. Transport costs account for a larger share in mass campaigns when compared with strategies that reorganize health systems (418% in mass campaigns and 1% in reorganizations), although the proportion of costs for transport in mass campaigns varied by country. While the proportion of costs for vaccines differs dramatically (558%), much variation appears to relate to differences in financial or opportunity costing. Thus differences in donations and the country where the vaccine is produced significantly affected costs and resource profiles. The proportion of costs attributed to buildings, possibly a more permanent need in an overall reorganization of the health system, was significantly higher in Benin and Guinea than in countries adopting mass campaigns.
All interventions reported positive costeffectiveness ratios with a range of expected costs, mostly based on best-case and worst-case scenarios. Judging from the range of total costs among studies (US$ 5 million to US$ 3900 million) the size of the programmes appears to differ substantially. Most estimates of average incremental cost fall below US$ 50.00 per additional fully vaccinated child. However, it should be noted that indicators of effectiveness differ, e.g. the study from the Dominican Republic presented the number of doses distributed whereas the Indonesian intervention estimated US$ 1400.00 per death averted (to facilitate comparison with other health programmes).
Generalizability of results
The descriptions of context were judged to be partial in 70% of cases and inadequate in 18%. Consideration of external validity was inadequate in 24% of cases and partial in 35% because few compared their results with other findings or conducted a sensitivity analysis. While 70% of the studies reported reasons for low baseline coverage rates, more than half failed to give any background information on current EPI operations, such as implementation year or EPI delivery strategies prior to the intervention. This, in combination with the paucity of information on the functioning of the health system (which was supplied by only six studies), renders it difficult to assess what demands the interventions placed on existing systems or whether the interventions can be replicated. Finally, 11 studies provided no information on the prevalence or incidence of the disease evaluated and even fewer provided information on the burden of disease avoided.
Discussion
Comparing the quality of the grey literature and published literature
One of the concerns with the grey literature is that it is not subject to peer review and may therefore be of lower quality and have unreliable findings. To counter this potential difficulty, we used the same criteria as Pegurri et al. (5) in selecting published literature. While 63% of papers on effectiveness from the grey literature met the criteria for full review for this study, only 57% from the published literature met the criteria. In contrast, 40% of papers from the grey literature with cost data met the criteria compared with 80% from the published literature. Clearly, cost data is significantly less robust in the grey literature.
The grey literature tended to be better at specifying the time frame for analysis; defining study populations; being more transparent in sources of cost data; identifying alternative interventions for comparison; and considering sustainability, possibly because analyses were more often financial than economic. The published literature tended to be better at covering categories of costs more comprehensively, undertaking economic costing rather than financial costing, providing more transparent explanations of the valuation approaches, conducting sensitivity analyses, and clarifying the study sample. The designated end-audience may provide some justification for the differences: some grey literature appeared as brochures designed to market immunization programmes.
Comparing the focus and findings
Searching the grey literature almost doubled the amount of evidence available by adding 24 interventions to the original 29 (5). The nature and balance of evidence also changed, with proportionally more papers being cost analyses or cost-effectiveness analyses and a higher proportion focusing on west Africa and the Middle East. The evidence in the grey literature was more recent and tended to focus on national rather than subnational levels. This reflects the production of several national plans and reviews of country programmes, which formed an important body of grey literature. The grey literature reflected more of a focus on operational research and finance rather than the broader issues of economics and policy-making examined in the published literature.
The results from the grey and published literature were combined and compared in a number of ways. Paired-sample t tests using data from the combined sample showed that there was a statistically significant improvement in coverage of FVC from baseline to post-intervention (t = 2.78, P = 0.08) and a statistically significant negative relationship between the percentage point change in coverage and final coverage levels (t = 12.8, P < 0.001). However, there were no statistically significant differences between published and grey literature in terms of baseline or final coverage levels or changes in coverage.
Fig. 2 shows that from baseline coverage of 39%, the interventions reviewed have provided a 0.62 percentage point rise in coverage of FVC per percentage point of baseline coverage. However, two cautionary points must be considered. First, there is more variability in values around the 1540% levels for baseline coverage, suggesting this interpretation is less reliable at lower levels of baseline coverage. Second, the interpretation treats each observation equally whereas the data points reflect a wide range in sample size. It would be possible for future analysts to account for sample size in a meta-analysis however many studies did not provide this data.
Evaluations that passed our quality review covered a different mix of interventions. The grey literature had fewer supply-side interventions (8% versus 41%) and demand-side interventions (13% versus 24%) but a larger proportion evaluated interventions targeted at both demand-side and supply-side factors (79% versus 34%). This complexity might be related to difficulties in attributing change in coverage to a system-wide intervention and may explain why fewer of these papers have been published. Alternatively, it may reflect a lag in publishing results (12).
We ran a series of analyses to discover whether the addition of the grey literature to the published literature changed conclusions about the relative effectiveness of interventions that focused on demand, supply or supply-led plus demand-led interventions. In the combined sample of literature, the mean baseline coverage of FVC was 36% for demand-led interventions, 40% for supply-led, and 38% for supply-led plus demand-led interventions. Following the intervention, coverage was 64% for demand-led interventions, 70% for supply-led, and 57% for supply-led plus demand-led interventions. T tests (assuming unequal variances) showed no statistically significant differences between these broad categories of intervention at baseline or in terms of percentage increase in coverage.
Tentative suggestions (based on the three interventions with the largest increases in coverage of FVC) about the types of interventions shown to be more effective or less costly were different in the grey literature and the published literature. While the published literature suggested community health workers and door-to-door campaigns would be effective, the grey literature suggested campaigns to educate health workers or mass campaigns as well as alternative models for providers or payers would be effective.
A major gap in both literatures is the lack of head-to-head comparisons of the cost of alternative interventions used to increase coverage, e.g. comparisons of health education campaigns with mass vaccination campaigns. Better quality evaluations of introducing performance-based pay and alternative service delivery agreements between ministries of health and NGOs are also needed, as is more consideration of the generalizability of findings.
Reflections on the search process for the grey literature
Most of the 68 documents reviewed initially were found after interviewing experts and searching institutional document stores. Further documents may be available within country offices, and future searches could access such evidence given sufficient resources. Internet searches that used the Google search engine also produced several documents. However, the search of databases of grey literature was almost entirely unproductive. The relationship between the quality of papers and the original source served to emphasize the importance of meeting experts, hand searching document stores in key organizations and using the Internet. Given the cost of visiting libraries in other countries, organizations could dramatically increase the size and nature of evidence available by placing more evaluation documents on the Internet.
Advances in technology and changes in publishing seem to have improved access and simplified the process of locating grey literature. However, the explosion of documents available for review (13) does not appear to have improved cataloguing. We spent much time judging the appropriateness of references because, for example, the absence of abstracts meant that more papers were reviewed than was necessary when compared with the published literature.
Potential problems with our approach
We are aware that the literature we reviewed largely derives from internationally focused organizations and covers little work funded by national governments. This was a function of our search strategy combined with the necessity of searching within a restricted budget. Nationally-funded work is likely to be best searched for within countries, and future project evaluations should include such material. The dearth of materials and the desire for evidence meant that papers arriving late for the review of published literature by Pegurri et al. (5) were included in this review because they had valuable information. Therefore, not all the materials gathered are strictly grey literature.
The grey literature selected included a few papers from francophone Africa (n = 3) and a few on Latin American countries in Spanish (n = 3); a further two papers in French and five in Spanish were also considered. This lack of non-English language papers occurred despite our previous search of WHO regional databases including LILACS (a database of articles in the health sciences from Latin America and the Caribbean), African Index Medicus (AIM), Index Medicus for WHO South-East Asia Region (IMSEAR) and Index Medicus for WHO Eastern Mediterranean Region (IMEMR) and may reflect the fact that we interviewed people whose first language was not always English. The preponderance of English-language documents in our analysis is most likely to have been caused by the fact that our searching was limited to English keywords owing to resource limitations and being a group of researchers who work largely in English. Had we chosen Spanish, French or Portuguese terms in addition to the English terms, we may have been able to draw on a wider range of papers. Future researchers are encouraged to find out what impact limiting a search to English language keywords might have had. In searching the grey literature it is particularly important to consider that relevant operational research will be fed into national policy in local languages and that English is the official language of business in only a minority of countries.
Finally, we accept that interpretational discrepancies may have occurred in comparing the grey and published literature even though one of the authors was involved in both reviews.
Conclusion
This review of the grey literature on immunization provided new insights into the performance of interventions designed to increase coverage of routine immunizations. The evidence from the grey literature was more up to date, reflected more complex interventions aimed at health systems and better represented west Africa and the Middle East. Although the quality of the evidence on effectiveness and cost-effectiveness data was largely comparable with that in the published literature, data on cost tended to be more unreliable. Decisions about which coverage strategies to use would be helped considerably by an increase in the quantity and quality of costing data alongside evaluations of effectiveness. It is also important for researchers to consider the external validity of their research results and present disaggregated data to facilitate more appropriate transfer of knowledge across settings. 
Acknowledgements
During the study all authors were members of the Health Economics and Financing Programme, which is supported by the UK Department for International Development through the London School of Hygiene and Tropical Medicine.Funding: We gratefully acknowledge the World Health Organization, Geneva, for funding granted under project reference I8/181/1024. This work is part of a larger project funded by the UK Department of International Development (Ref. R7842).
Conflicts of interest: none declared.
References
1. The World Bank. World Development Report 1993: investing in health. Oxford: Oxford University Press; 1993.
2. Cutts FT, Olivé J-M. Vaccination programmes in developing countries. In: Plotkin SA, Orenstein WA, editors. Vaccines. 3rd ed. Philadelphia: W.B. Saunders; 1998. p. 1047-73.
3. Gwatkin DR, Rutstein S, Johnson K, Pande RP, Wagstaff A for the HNP/Poverty Thematic Group. Socio-economic differences in health, nutrition and population in Côte d'Ivoire. Washington (DC): World Bank, Health Population and Nutrition Group; 2000.
4. Gwatkin DR, Rutstein S, Johnson K, Pande RP, Wagstaff A for the HNP/Poverty Thematic Group. Socio-economic differences in health, nutrition and population in India. Washington (DC): World Bank, Health Population and Nutrition Group; 2000.
5. Pegurri E, Fox-Rushby J, Walker D. The effects and costs of expanding coverage of immunization services in developing countries: a systematic literature review. Vaccine 2004. In press 2004.
6. Hart C. Doing a literature search: a comprehensive guide for the social sciences. London: Sage; 2001.
7. McAuley L, Pham B, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000;356:1228-31.
8. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study, 2003. Available from: http://www.ncchta.org
9. Cook, A, Finlay I, Edwards AG, Hood K, Higginson IJ, Goodwin DM, et al. Efficiency of searching the grey literature in palliative care. Journal of Pain and Symptom Management 2001;22:797-801.
10. Mathews E, Edwards AG, Barker J, Bloor M, Covey J, Hood K, et al. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libraries Review 1999;16:112-20.
11. Pegurri E, Fox-Rushby J, Walker D. The effectiveness and efficiency of methods to improve coverage of routine immunization programmes. Geneva: World Health Organization; 2002.
12. Misakian AL, Bero LA. Publication bias and research on passive smoking: comparison of published and unpublished studies. JAMA 1998;280:250-3.
13. Helmer D. Grey literature. In: Etext on Health Technology Assessment (HTA), 2002. Available from http://www.nlm.nih.gov/nichsr/ehta/chapter10.html
(Submitted: 13 January 2004 Final revised version received: 20 May 2004 Accepted: 23 June 2004)
1 Correspondence should be sent to this author.
Appendix A
Inclusion criteria for studies from the grey literature
Appendix B
Papers selected for inclusion in the review
Pan American Health Organization. Achieving equitable immunization coverage: the Bolivia Project. EPI Newsletter: Expanded Program on Immunization in the Americas 2002;24:4-6.
Akalay O. The 1987 National Vaccination Campaign in Morocco. Casablanca (Morocco): UNICEF; 1987. p. 1-36.
US Agency for International Development/Basic Support for Institutionalizing Child Survival, stage II. Augmenter la couverture vaccinale en améliorant l'utilisation des données par le monitorage [Increasing vaccination coverage by increasing the use of monitoring data]. Washington (DC): USAID/BASICS II; (PowerPoint presentation, slides 1-15).
Bhushan I, Keller S, Schwartz B. Achieving the twin objectives of efficiency and equity: contracting health services in Cambodia. Manila: Asian Development Bank; March 2002 (ERD Policy Brief Series, p. 1-10).
Pan American Health Organization. Bolivia benefits from PAHO/World Bank partnership. EPI Newsletter: Expanded Program on Immunization in the Americas 1999;21:1-3.
Pan American Health Organization. Bolivia/PAHO/World Bank partnership supports second generation EPI. EPI Newsletter: Expanded Program on Immunization in the Americas 2001;23:4-5.
Brenzel L, Claquin P, Mclellan I, Stansfield S. Rapid assessment: Senegal's accelerated immunization phase. Arlington (VA): Resources of Child Health Projects, Jon Snow Inc; December 1987 (Evaluation Publication No. 5, p. 3-41).
Anonymous. Child survival: accelerated immunization project. (A mid-term evaluation of Expanded Programme of Immunization. USAID, Pan American Health Organization, Inter-American Development Bank and Rotary Club.) August 1989, Grant No. CAR-0005-G-00-6971-00: p. i-153.
Claquin P, Diouf F, Floury B, Garenne M. Rapid assessment: rapport d'évaluation de la couverture vaccinale des enfants de 12 à 23 mois en République de Sénégal au 01/07/1987 [Report on the evaluation of the vaccination coverage of children aged 12-23 months in the Repulic of Senegal]. Senegal: UNICEF; 1987. p. 43-66.
Diene SM, Shimp L, Wansi E, Ouedraogo Y, Harris N, Rossi E, et al. How Madagascar is improving family health: using an integrated community-based approach Madagascar: Ministry of Health, Madagascar Fianarantosa, Antananarivo District Health Teams; 2003. Chapters 2 and 6.
Diene SM, Shimp L, Wansi E, Ouedraogo Y, Harris N, Rossi E, et al. Improving family health using an integrated community based approach: Madagascar case study. Summary report. Madagascar: Ministry of Health, Madagascar Fianarantosa, Antananarivo District Health Teams; 2003. p. 1-30.
Eichler R, Auxila P, Pollock P. Output-based health care: paying for performance in Haiti. Washington (DC): World Bank Group, Private Sector and Infrastructure Network; 2001 (No. 236, p. 1-4).
Eichler R, Auxila P, Pollock P. Performance-based payment to improve the impact of health services: evidence from Haiti. World Bank Institute Online Journal 2001. Available from: http://www.worldbank.org/wbi/healthflagship/oj_haiti.pdf
Ekunwe EO. Expanding immunization coverage through improved clinic procedures. World health forum 1984;5:361-3.
UNICEF. Follow-up assessment of house-to-house vaccination in the Dominican Republic 15-27 May 1988. New York: UNICEF; 1988. p. 1-27 (Evaluation Publication No. 6).
UNICEF. Immunizing more children. New Delhi: UNICEF; 1984. p. 1-66.
UNICEF. Improvement of EPI management and performance in districts with low coverage: methodology and lessons learned, Zimbabwe 1993-1995. Harare: UNICEF; 1996. p. 1-11.
Anonymous. Improvement of the vaccination coverage and reduction of the drop-out rate. Basic Support for Institutionalizing Child Survival, stage II (BASICS II): unpublished document, accessed through country representative, 5 pages.
Karefa-Smart J, Patte C, Brenzel L, Johnson T, Jabr I. Rapid assessment: Cameroon's National Vaccination Campaign. New York: UNICEF; March 1987. p. 3-30 (Evaluation Publication No. 4).
La Forgia J, Danel I. Large scale contracting out of basic health services for the poor in Guatemala. Washington (DC): World Bank; March 2002 (PowerPoint presentation, slides 1-18).
Leavell R, Sumanto S, Feldon K. Child survival X: improving immunization coverage and village health post (posyandu) implementation. San Diego (CA): Project Concern International; 1997. p. 1-54.
Loevinsohn BP, Guerrero ET, Gregorio SP. Improving primary health care through systematic supervision: a controlled field trial. Health Policy and Planning 1995;10:144-53.
Meza JL, Castillo O, Echeverría D, Johnson T. Rapid assessment: house to house immunization. The Dominican Republic experience. Panama City: UNICEF Regional Office for Latin America and the Caribbean; 1986. p. 1-17, annexes I-XII.
Naimoli JF, Gbekley A, Bamaze T, Parker K, Bryce J, Naimoli G. Working with communities to increase the use of health services: an experience from Togo, West Africa. International Quarterly of Community Health Education 1993;14:257-71.
UNICEF. Overview of Immunization Plus activities in Sénégal: summary of routine EPI activities in the country. Senegal: UNICEF.
Quality Assurance Project. Guatemala: child survival and reproductive health services in the Guatemalan highlands. Bethesda (MD): Center for Human Services; 2001. p. iv-20 (HRN-00-96-90013-00).
Rabelahasa E, Diallo A, Cisse M, Joyal A. Big sister/brother is watching you: increasing child vaccination coverage in rural Guinea, presentation at 129th Annual Meeting of the American Public Health Association, October 2001, slides 1-16.
UNICEF. Rapid assessment: Turkish national immunization campaign of 1985. New York: UNICEF; 1986. p. 3-74 (Evaluation Publication No. 3).
Anonymous. Sénégal: background information Immunization Plus. New York: UNICEF. p. 1-4.
Shojaezadeh D. A study of the most effective health education strategy to increase acceptance of immunisation in Iran [thesis]. Liverpool: Liverpool University; 1990.
Six C, Laender F. Evaluation of the immunization program in Nicaragua. Managua: REACH, consultants for USAID/Nicaragua; 1993. p. 1-41 (Project No. 936-5982).
Soucat A, Levy-Bruhl D, De Bethune X, Gbedonou P, Lamarque J, Bangouta O et al. Affordability, cost-effectiveness and efficiency of primary health care: the Bamako Initiative experience in Benin and Guinea. International Journal of Health Planning and Management 1997;12 Suppl:S81-108.
Anonymous. Urban accelerated strategy in the Philippines: UNICEF's urban examples. New York: UNICEF; 1990. p. 9-12.
UNICEF. Urban PHC programme in Addis Ababa, Ethiopia: a model for Third World cities. UNICEF's urban examples. In: Universal child immunization: reaching the urban poor. Washington (DC): UNICEF; 1990. p. 18-9.