INVESTIGACIÓN ORIGINAL ORIGINAL RESEARCH
Costo-eficacia de la mejora de la atención hospitalaria pediátrica en Nicaragua
Edward I. BroughtonI,*; Ivonne GomezI; Oscar NuñezI; Yudy WongI
IUniversity Research Co., LLC/United States Agency for International Development, Health Care Improvement Project, Bethesda, Maryland, United States of America
IIUse of a shared learning approach among a large number of quality improvement teams who work together toward the same aim to rapidly achieve significant improvements in the processes, quality, and efficiency of services (5)
OBJECTIVE: To determine the costs and cost-effectiveness of an intervention to improve quality of care for children with diarrhea or pneumonia in 14 hospitals in Nicaragua, based on expenditure data and impact measures.
METHODS: Hospital length of stay (LOS) and deaths were abstracted from a random sample of 1294 clinical records completed at seven of the 14 participating hospitals before the intervention (2003) and 1505 records completed after two years of intervention implementation ("post-intervention"; 2006). Disability-adjusted life years (DALYs) were derived from outcome data. Hospitalization costs were calculated based on hospital and Ministry of Health records and private sector data. Intervention costs came from project accounting records. Decision-tree analysis was used to calculate incremental cost-effectiveness.
RESULTS: Average LOS decreased from 3.87 and 4.23 days pre-intervention to 3.55 and 3.94 days post-intervention for diarrhea (P = 0.078) and pneumonia (P = 0.055), respectively. Case fatalities decreased from 45/10 000 and 34/10 000 pre-intervention to 30/10 000 and 27/10 000 post-intervention for diarrhea (P = 0.062) and pneumonia (P = 0.37), respectively. Average total hospitalization and antibiotic costs for both diagnoses were US$ 451 (95% credibility interval [CI]: US$ 419-US$ 482) pre-intervention and US$ 437 (95% CI: US$ 402-US$ 464) post-intervention. The intervention was cost-saving in terms of DALYs (95% CI: -US$ 522- US$ 32 per DALY averted) and cost US$ 21 per hospital day averted (95% CI: -US$ 45- US$ 204).
CONCLUSIONS: After two years of intervention implementation, LOS and deaths for diarrhea decreased, along with LOS for pneumonia, with no increase in hospitalization costs. If these changes were entirely attributable to the intervention, it would be cost-saving.
Key words: Hospital costs; hospitals, pediatric; cost-benefit analysis; diarrhea; pneumonia; delivery of health care; Nicaragua.
OBJETIVO: Determinar el costo y la eficacia en función del costo de una intervención para mejorar la calidad de la atención de niños con diarrea o neumonía en 14 hospitales de Nicaragua, sobre la base de la información sobre gastos y la medición de las repercusiones.
MÉTODOS: Se compilaron datos sobre la duración de la hospitalización y la mortalidad de una muestra aleatoria de 1 294 historias clínicas compiladas en 7 de los 14 hospitales participantes antes de la intervención (2003) y 1 505 historias clínicas compiladas después de dos años de ejecución de la intervención ("postintervención", 2006). Los años de vida ajustados en función de la discapacidad (AVAD) se obtuvieron de los resultados asistenciales. Se calcularon los costos de hospitalización según los registros de los hospitales y del Ministerio de Salud, y datos del sector privado. Los costos de la intervención se obtuvieron de los registros contables del proyecto. Para calcular la relación costo-eficacia incremental se usó un análisis de árbol de decisiones.
RESULTADOS: La duración promedio de la hospitalización disminuyó de 3,87 y 4,23 días antes de la intervención a 3,55 y 3,94 días después de la intervención para la diarrea (P = 0,078) y la neumonía (P = 0,055), respectivamente. La letalidad disminuyó de 45/10 000 y 34/10 000 antes de la intervención a 30/10 000 y 27/10 000 después de la intervención para la diarrea (P = 0,062) y la neumonía (P = 0,37), respectivamente. Los costos totales promedio de la hospitalización y de los antibióticos para ambos diagnósticos fueron de US$ 451 (intervalo de confianza [IC] de 95%: US$ 419 a US$ 482) antes de la intervención y US$ 437 (IC 95%: US$ 402-US$ 464) después. La intervención representó un ahorro de costos en cuanto a los AVAD (IC 95%: -US$ 522 a US$ 32 por cada AVAD evitado) y costó US$ 21 por cada día de hospitalización evitado (IC 95%: -US$ 45 a US$2 04).
CONCLUSIONES: Después de dos años de ejecución de la intervención, la duración de la hospitalización y la mortalidad por diarrea disminuyeron, junto con la duración de la hospitalización para la neumonía, sin un aumento de los costos de hospitalización. En caso de que estos cambios fueran totalmente atribuibles a la intervención, esta representaría un ahorro de costos.
Palabras clave: Costos de hospital; hospitales pediátricos; análisis costo-beneficio; diarrea; neumonía; prestación de atención de salud; Nicaragua.
Pneumonia and diarrhea are responsible for the highest burden of morbidity and mortality among children in Nicaragua (1, 2). Significant gaps have been found between the clinical management of children in referral hospitals with these two diagnoses and that recommended in the World Health Organization (WHO)/UNICEF Integrated Management of Childhood Illness (IMCI) strategy for low-income countries in general (3) and Nicaragua in particular (4). To reduce these gaps, Nicaragua's Ministry of Health (MINSA) worked with the U.S. Agency for International Development (USAID), through its Quality Assurance Project (QAP), and UNICEF, to design and implement a hospital-based intervention based on the quality improvement (QI) collaborative approach.2
Known as the Pediatric Hospital Improvement (PHI) Collaborative, the intervention was launched in 2004 at 14 regional hospitals and focused on children admitted to the pediatric ward with severe cases of diarrhea or pneumonia. Intervention objectives, identified during a 6-month evaluation conducted at six hospitals in 2003, were designed to address various quality-of-care problems, including:
Lack of a standardized diagnosis and classification system based on explicit clinical criteria, standard definitions for identifying patients with severe dehydration, and uniform guidelines for treating those patients;
Inappropriate prescription of antibiotics and other medications;
No specified room for close, constant monitoring of critically ill children;
Poor recognition of danger signs by clinical staff;
Inadequate triage system for patients seen in the emergency room;
Inadequate written guidelines such as clinical algorithms and treatment protocols to direct medical and nursing staff to evidence-based care;
Poor transportation system for transferring critically ill children to referral facilities; and
Poor ability to identify the nutritional status of patients.
The QI strategy included eight components: 1) introducing the WHO/UNICEF referral care manual (6) to standardize the clinical approach to treatment of children with severe diarrhea or pneumonia at the hospital level; 2) defining quality indicators to measure compliance with standards and identify performance gaps; 3) forming QI teams to promote clinical practice changes that would lead to improved quality of care; 4) establishing mechanisms for prioritizing the care of children who come to emergency rooms; 5) changing hospital organization so critically ill children are concentrated in a single hospital environment to allow closer and continuous monitoring; 6) developing skills in medical and nursing staff to identify, manage, and monitor for danger signs; 7) improving adherence to standardized prescribing practices for antibiotics; and 8) involving local authorities in supporting quality activities.
While routine monitoring of quality indicators by PHI Collaborative hospital teams showed that the intervention was associated with a decline in case fatality rates for diarrhea and pneumonia as well as an increase in compliance with clinical protocols (including the appropriate use of antibiotics) and reduced hospital stays, its costs and cost-effectiveness were not known. To fill this gap, the authors of the current study used data on the intervention's expenditures and various measures of its impact to determine its overall cost and economic efficiency. This information could help Nicaraguan decision-makers determine if nationwide implementation of this type of intervention would be affordable and cost-effective compared to other strategies to improve care.
MATERIALS AND METHODS
The study was based on two years of intervention implementation (2004- 2005). Pre-/post-intervention evaluations were conducted to determine intervention impact. The research team examined diarrhea and pneumonia inpatient medical records and admissions registers completed at participating hospitals 1) before the intervention (in 2003) ("preintervention"), and 2) after two years of its implementation (in 2006) ("postintervention"). Expenditure data were obtained from hospital and MINSA records and private sector calculations (for hospitalization costs) and UNICEF and USAID/QAP accounting records (for intervention costs). Miscellaneous training costs (e.g., transportation, food, and housing) were based on author estimates. Impact measures and expenditures were defined according to the criteria of MINSA and USAID/QAP.
The 14 hospitals participating in the PHI Collaborative did not begin the intervention simultaneously. Therefore, to ensure the consistency of the data, the study sample was limited to hospitals where QI teams had implemented improvement cycles and monitored the indicators of interest for at least two years (i.e., since the intervention's launch in 2004). The seven qualifying hospitals were Chinandega, Madriz, Estelí, Boaco, Granada, Masaya, and La Trinidad. The authors randomly selected hospital records from all patient charts available for review for both the pre- and post-intervention evaluations. For the pre-intervention evaluations, 647 clinical records were selected for both diarrhea and pneumonia. For the post-intervention evaluations, 750 records were selected for diarrhea and 755 records for pneumonia. A total of 2 799 records were examined out of the 14 833 case records (9 938 diarrhea and 4 895 pneumonia) available at the seven hospitals included in the study. Sampling was stratified by hospital. A sample of 90-110 records was taken from each hospital. The sampling number was determined according to USAID/QAP monitoring requirements. Cases of severe diarrhea and pneumonia were identified based on IMCI definitions (7). Patients with a simultaneous diagnosis of diarrhea and pneumonia were excluded from the sample.
Total costs were calculated retrospectively based on data from 1) hospital records, MINSA, and private sector calculations (for hospitalization costs) and 2) UNICEF and USAID/QAP accounting records (for intervention costs). Hospitalization costs included the salaries of the doctors, nurses, and ancillary staff required to staff the pediatric wards, calculated according to MINSA salary rates; hospital bed costs, including food, water, medical supplies, electricity, and other utilities, calculated according to rates established in private hospitals; the amount paid for antibiotics prescribed and consumed during hospitalization, calculated according to the MINSA price list of basic medicines; and the amount paid for laboratory blood tests for all patients (pneumonia and diarrhea), a chest radiograph for each pneumonia patient, and a fecal laboratory test for each diarrhea patient, according to MINSA records. For items that are not accounted for in Nicaraguan public hospitals (e.g., clinical line items), estimated costs were based on those established for the private sector (which are considered a reasonable approximation to costs in the public sector). All costs were recorded in Nicaraguan córdobas, converted to U.S. dollars using the January 2006 exchange rate (C$17 per US$1) (8) and adjusted to 2010 by applying an annual inflation rate of 3%.
The three measures of the QI intervention's impact on cases of diarrhea or pneumonia were 1) hospital length of stay (LOS), 2) disability-adjusted life years or DALYs (years of potential life lost due to premature mortality and years of productive life lost due to disability), and 3) deaths. Data on LOS and mortality were collected from the patient records sampled in the participating hospitals. DALYs for acute illness were estimated using disability weights assigned to severe cases of pneumonia and diarrhea, obtained from Murray and Lopez (9), and assuming average age of onset was 2 years. Age weighting, a discount rate of 3% per annum, and a life expectancy of 73 (10) were applied to DALYs for mortality calculations according to a standard methodology that used similar economic analyses without regard to gender (11, 12) (Table 1). The total number of patients admitted for each of the two diagnoses was obtained from hospital admission registries.
Decision-tree analysis was used to compare the QI intervention strategy (improved compliance with clinical guidelines) to "business-as-usual" (preintervention clinical management of diarrhea and pneumonia case) in a single iterative model (Figure 1). The incremental cost-effectiveness ratio (ICER), defined as the incremental change in cost divided by the incremental change in effectiveness, was calculated by comparing indicators at the seven participating hospitals pre- and post-intervention. In terms of DALYs, the ICER represented the cost per additional DALY averted ($/DALY) associated with the QI strategy. The lower the ICER value, the better the relative cost-effectiveness of the care delivered post-intervention compared to that provided pre-intervention. A positive ICER value indicates that an additional expenditure is required to achieve better health outcomes, whereas a negative ICER value means the intervention is associated with better health outcomes and a decrease in overall cost and is thus designated as "cost-saving." The equation used to calculate the ICER (incremental cost/average number of incremental DALYs) is shown below.
Because the outcome data used in the current study were collected from a random sample of cases from seven different hospitals, the sampling distribution associated with the ICER point estimate (PE) (the average of multiple ICER point estimates, representing estimated hospitalization and antibiotic costs, across multiple data sets) needed to be taken into account to determine a credibility interval (CI). The authors used Monte Carlo simulations in which repeated sampling from the distributions of all input variables was used to calculate a probability distribution for the ICERs. This allowed for calculation of a 95% CI.
One-way probabilistic sensitivity analysis was used to rank the variables by their relative importance to the costeffectiveness result (in terms of DALYs) by using each of the variables in turn and performing Monte Carlo simulations to determine CIs based on the degree of uncertainty for each specific variable. Appropriate probability distributions had to be assigned to the input variables as some were not statistically significant at the alpha = 0.05 level. These distributions were determined based on the study data (Table 1).
Average LOS decreased from 3.87 days pre-intervention to 3.55 days postintervention for diarrhea cases (P = 0.078) and from 4.23 days pre-intervention to 3.94 days post-intervention for pneumonia (P = 0.055). The case fatality ratio decreased from 45 per 10 000 preintervention to 30 per 10 000 postintervention for diarrhea (P = 0.062) and from 34 per 10 000 pre-intervention to 27 per 10 000 post-intervention for pneumonia (P = 0.37) (Table 1).
Intervention costs are listed in Table 2. Technical assistance, which accounts for almost 25% of the total cost of the QI intervention, included all work done by USAID/QAP personnel to implement the eight strategy components. It did not include the time cost of MINSA administrators or hospital officials because their roles in these activities were considered part of their normal work duties and therefore did not incur additional cost to MINSA. The time cost of hospital personnel attending the training sessions was included, even though this did not incur any additional direct cost to MINSA, because attendance in the sessions took these health workers away from their regular duties, and whether or not relief staff were brought in to compensate for this absence it was important to account for this potential loss in productivity. Training sessions consumed the greatest proportion of total expenditures (43.2%).
Costs per day for hospitalization (excluding antibiotics) remained unchanged (pre- versus post-intervention) at $93.30 and $99.12 per day for diarrhea and pneumonia cases respectively. The cost of antibiotics per day increased from $8.80 to $11.80 for diarrhea and from $13.70 to $14.00 for pneumonia. The average total cost of hospitalization and antibiotic medication for both diagnoses was $451 pre-intervention and $437 post-intervention.
In terms of DALYs, the QI intervention was cost-saving compared to "business-as-usual" (95% CI: -$522-$32 per DALY averted; Table 3). When the variability of the result is plotted on a cost-effectiveness acceptability curve, the intervention is 100% certain to be cost-effective when the health care funder is willing to pay $200 or more per DALY averted (Figure 2). In terms of hospital days, the intervention cost $21 (per hospital day averted) (95% CI: -$45-$204).
One-way probabilistic sensitivity analysis is shown in Table 4. The larger the credibility interval, the greater the uncertainty associated with the variable and therefore the greater its importance to the cost-effectiveness result. LOS variables had the greatest effect on the result, followed by case fatality ratios and cost of antibiotics. Uncertainty in the cost of hospitalization had a small impact on the result, while uncertainty in the cost of the QI intervention and the ratio of diarrhea to pneumonia cases had almost no effect on the result.
This study found close to statistically significant decreases in LOS (for diarrhea and pneumonia) and in deaths (for diarrhea) that were associated with the QI intervention. Calculation of cost-effectiveness in terms of DALYs showed that the intervention was cost-saving, indicating a high likelihood that the QI intervention improves health outcomes and decreases costs to the hospital system. This result explains the decrease in case fatality rates for both diarrhea and pneumonia that was observed following the QI intervention.
In terms of LOS, the ICER was $21, indicating that decreasing the average LOS would most likely result in a small additional cost. The fact that the ICER was positive in terms of LOS but negative (i.e., cost-saving) in terms of DALYs can be attributed to the scope of the DALYs metric (which considers deaths, and acute illness) versus that of the LOS metric (which treats all discharged patients as one and the same regardless of the outcome). The LOS ICER is presented to show the intervention's relative cost-effectiveness in financial terms regardless of whether or not it meets the criteria for DALY calculations.
Even when considering the point on the cost-effectiveness acceptability curve at which there is 100% certainty of costeffectiveness (a willingness to pay of $200 per DALY averted), the ICER is highly favorable. WHO defines a health intervention as very cost-effective if its ICER is less than the GDP per capita of the population it serves (13). As Nicaragua's per capita GDP is $2 900, the PHI Collaborative intervention would be considered very cost-effective (14). Comparing the efficiency of the QI strategy used in this intervention to strategies used in other health interventions in Nicaragua or other Central American countries is difficult given the paucity of economic evaluations of health programs from the region (15). To the best of the authors' knowledge, no other cost-effectiveness studies have been conducted to evaluate interventions for pediatric diarrhea or pneumonia cases. The PHI Collaborative intervention appears to be significantly more cost-effective than cervical cancer screening in Nicaragua, reported to cost $3 700 per DALY averted (16), and compares favorably with rotavirus vaccination for diarrheal disease prevention in both Honduras, reported to cost $269 per DALY averted (17), and Northeastern Brazil, based on another economic analysis that showed its effects on costs for diarrhea were not cost-saving (18).
Diarrhea and pneumonia are the two most common causes of infant death worldwide, accounting for 40% of all mortality in children under 5 years old (19, 20), and remain significant public health problems in Nicaragua, where the infant mortality rate is 27/1 000 (21). Any strategies that help address the high burden of disease resulting from these two conditions are likely to have a significant impact on the overall effectiveness of the health system.
Money saved due to the slight decrease in average LOS that occurred over the course of the intervention was partially offset by the increase in expenditure for antibiotics that resulted from improved compliance with clinical guidelines. Before the intervention, prescription practices for antibiotics in Nicaragua were reported to be generally poor, leading to suboptimal outcomes and an increase in the risk of resistance (22). Suboptimal practices included the prescription of cheaper antibiotics that in many cases were not recommended. The intervention addressed this issue by standardizing prescribing practices and facilitating the use of more appropriate (and often more expensive) antibiotics for diarrhea and pneumonia. Therefore, while this part of the intervention achieved the desired result of improved compliance with prescribing standards, it also led to an increase in the average cost per patient for antibiotics.
Sensitivity analysis showed that the uncertainty in LOS input variables had the greatest impact on the cost-effectiveness result. Given that the differences between pre- and post-intervention LOS were not statistically significant, achieving a more accurate estimate of these inputs by using a larger sample size would have yielded a more precise result.
By definition, collaborative improvement is a temporary intervention designed to 1) address specific problems in the quality of services and 2) institutionalize the changes made in care processes to ensure sustainability of the results. Once this occurs, QI teams either move onto another set of problems or disband. If that definition holds true, favorable results would continue to be achieved upon completion of the PHI Collaborative intervention at the targeted facilities, without further intervention and expense. If this were the case, the QI intervention strategy would prove more cost-effective when considered over a longer time period.
While no other interventions were implemented to address the quality of diarrhea and pneumonia care in Nicaraguan hospitals either before or during the PHI Collaborative intervention, the lack of a control group in this study design leaves open the possibility that the changes detected in the pre-/post-intervention comparisons could have been due to factors other than the intervention. Further studies using a control group or an analogous design would help determine how much improvement was attributable specifically to the PHI Collaborative initiative.
Other than the start-date of their respective intervention activities, there were no substantive differences between the seven hospitals selected for the study sample and the remaining seven that were not. Nevertheless, the possibility of selection bias would have been reduced and the case for generalizability improved if all 14 hospitals (or a random sample from all 14 hospitals) had been studied.
This study did not consider the possible downstream benefits to either the health system or other departments in the participating hospitals of having better trained and more highly engaged and motivated clinical staff as a result of the QI intervention. According to the literature, QI collaboratives have produced downstream benefits such as improved provider engagement with their work, and increased trust among providers (23); better staff relationships; and the extension of QI approaches to other areas within the facilities (24). The higher-quality clinical care for diarrhea and pneumonia resulting from the PHI Collaborative intervention may also have led to a decrease in morbidity and mortality from other causes, but these outcomes were not measured. If these benefits did occur and had been quantified and included in the analysis, the cost-effectiveness of the intervention would be higher. The intervention's effect on the care of those diagnosed with malnutrition (without pneumonia or diarrhea) was not considered either, even though part of the intervention addressed quality of care in this area. In addition, the authors did not consider factors related to cost-effectiveness from the societal perspective, such as potential cost savings for families caring for children with diarrhea and pneumonia after their discharge from the hospital. Once again, given the decreased burden of disease associated with the intervention, the cost to families and caregivers would have most likely decreased, improving overall cost-effectiveness.
Recommendation and conclusion
Due to the improved patient outcomes resulting from the intervention, overall hospital expenditures for the care of children with diarrhea and pneumonia would not increase if MINSA expanded the intervention to other pediatric hospitals in Nicaragua that were not part of the initial implementation. The $45 124 cost of the intervention, when divided by the number of children admitted to hospital pediatric wards with either diarrhea or pneumonia, is equivalent to about $3 per child served. This is less than 0.7% of the average cost of hospitalization for either of the two conditions. The decrease in the overall burden of disease associated with the intervention indicates that it may lead to a substantial net benefit to the health system. Assuming that the cost of implementing the QI intervention in other pediatric facilities is approximately the same as it was in the initial seven facilities, the $3-per-child investment may produce substantial cost-savings per DALY averted. Given the potential for large cost-savings, implementation of the QI intervention by MINSA throughout the country seems prudent. However, further research is needed to establish whether or not the improvements seen were completely attributable to the intervention before the authors can unreservedly recommend its adoption to all hospitals in this setting.
Acknowledgments. The authors thank the staff and administrators of the hospitals for their participation in this study, and the Nicaraguan Ministry of Health (MINSA) for their support and assistance. This work was supported by the American people through the United States Agency for International Development (USAID) and its Quality Assurance Project (QAP) (contract no. GHN-I-01-00003-00) and Health Care Improvement (HCI) Project (contract no. GHN-I-01-07-00003-00), implemented by University Research Co., LLC (URC).
1. World Health Organization. Disease and injury country estimates: burden of disease. Deaths estimates for 2008 by cause for WHO Member States [Internet]. Geneva: WHO; 2009. Available from: http://www.who.int/healthinfo/global_burden_disease/estimates_country/en/index.html. Accessed 8 March 2011.
2. Becker-Dreps S, Paniagua M, Dominik R, Cao H, Shah NK, Morgan DR, et al. Changes in childhood diarrhea incidence in Nicaragua following 3 years of universal infant rotavirus immunization. Pediatr Infect Dis J. 2011;30(3):243-7.
3. Duke T. Child survival and IMCI: in need of sustained global support. Lancet. 2009; 374(9687):361-2.
4. USAID Health Care Improvement Project. Improving pediatric care in Nicaragua. Bethesda, MD: University Research Co., LLC; 2008.
5. Nembhard IM. Learning and improving in quality improvement collaboratives: which collaborative features do participants value most? Health Serv Res. 2009;44(2 Pt 1): 359-78.
6. World Health Organization; UNICEF. Integrated Management of Childhood Illness chart booklet. Geneva: WHO; 2008.
7. World Health Organization; UNICEF. Model IMCI handbook: integrated management of childhood illness. Geneva: WHO; 2005.
8. Exchangerateusd.com. 2006 historical exchange rates [Internet]. Available from: http://www.exchangerateusd.com/historical currency/2006. Accessed 15 September 2010.
9. Murray C, Lopez A. The global burden of disease. Cambridge, MA: Harvard University Press; 1996.
10. World Health Organization. Global health observatory data repository: country statistics. Nicaragua [database on the Internet]. Geneva: WHO; 2009. Available from: http://apps.who.int/ghodata/?vid=14800&theme=country. Accessed 8 March 2011.
11. Kim SY, Goldie SJ, Salomon JA. Cost- effectiveness of Rotavirus vaccination in Vietnam. BMC Public Health. 2009;9(1):29.
12. De la Hoz F, Alvis N, Narváez J, Cediel N, Gamboa O, Velandia M. Potential epidemiological and economical impact of two rotavirus vaccines in Colombia. Vaccine. 2010; 28(22):3856-64.
13. World Health Organization. CHOosing Interventions that are Cost Effective (WHOCHOICE). Threshold values for intervention cost-effectiveness by region [table on the Internet]. Geneva: WHO; 2010. Available from: http://www.who.int/choice/costs/CER_levels/en/index.html. Accessed 23 August 2010.
14. World Bank. World Development Indicators database. Gross national income per capita, 2009: Atlas method and PPP [Internet]. Washington: World Bank; 2010. Available from: http://siteresources.worldbank.org/DATASTATISTICS/Resources/GNIPC.pdf. Accessed 10 August 2010.
15. Iglesias CP, Drummond MF, Rovira J; NEVALT Project Group. Health-care decisionmaking processes in Latin America: problems and prospects for the use of economic evaluation. Int J Technol Assess Health Care. 2005;21(1):1-14.
16. Goldie SJ, Diaz M, Constenla D, Alvis N, Andrus JK, Kim SY. Mathematical models of cervical cancer prevention in Latin America and the Caribbean. Vaccine. 2008;26 Suppl 11: L59-72.
17. Rheingans RD, Constenla D, Antil L, Innis BL, Breuer T. Potential cost-effectiveness of vaccination for rotavirus gastroenteritis in eight Latin American and Caribbean countries. Rev Panam Salud Publica. 2007;21(4):205-16.
18. Centenari C, Gurgel RQ, Bohland AK, Oliveira DM, Faragher B, Cuevas LE. Rotavirus vaccination in northeast Brazil: a laudable intervention, but can it lead to cost-savings? Vaccine. 2010;28(25):4162-8.
19. Vilchez S, Reyes D, Paniagua M, Bucardo F, Möllby R, Weintraub A. Prevalence of diarrhoeagenic Escherichia coli in children from León, Nicaragua. J Med Microbiol. 2009;58(Pt 5):630-7.
20. Ministerio de Salud (NI). Sala de situación. Boletín epidemiológico. May 3. Managua: MINSA; 2011. Available from: http://www.minsa.gob.ni/index.php?option=com_content&view=article&id=31:sala-desituacion&catid=6:direccion-general-vigilancia-de-la-salud-publica&Itemid=40. Accessed 15 September 2011.
21. UNICEF. At a glance: Nicaragua [Internet]. New York: UNICEF; c2010. Available from: http://www.unicef.org/infobycountry/nicaragua.html. Accessed 26 August 2010.
22. den Engelsen C, van der Werf C, Matute AJ, Delgado E, Schurink CA, Hoepelman AI. Infectious diseases and the use of antibiotics in outpatients at the emergency department of the University Hospital of León, Nicaragua. Int J Infect Dis. 2009;13(3):349-54.
23. Ayers LR, Beyea SC, Godfrey MM, Harper DC, Nelson EC, Batalden PB. Quality improvement learning collaboratives. Qual Manag Health Care. 2005;14(4):234-47.
24. Bradley JE, Mayfield MV, Mehta MP, Rukonge A. Participatory evaluation of reproductive health care quality in developing countries. Soc Sci Med. 2002;55(2):269-82.
Manuscript received on 10 October 2010.
Revised version accepted for publication on 13 May 2011.