LETTERS

Making systematic reviews more useful for policy-makers

Russell L. Gruen^I,II,1; Peter S. Morris^I; Elizabeth L. McDonald^I; Ross S. Bailie^I,III

^IMenzies School of Health Research, Institute of Advanced Studies, Charles Darwin University, PO Box 41096, Casuarina, Northern Territory, 0811 Australia (email: russell.gruen@menzies.edu.au)
^IIFaculty of Health Sciences, University of Queensland, Herston, Australia
^IIINorthern Territory Clinical School, Flinders University, Darwin, Northern Territory, Australia

Editor – The Mexico Summit has highlighted the need to improve evidence for informed policy-making about health concerns in low- and middle-income countries (LMICs). We agree with Volmink et al. who, in a recent article published in the Bulletin, asserted that systematic reviews provide the most useful type of evidence for determining the effectiveness of health care interventions (1). Other authors have focused on how policy-makers can interpret systematic reviews (2), and have called for more reviews of health problems in LMICs (1, 3).

However, as highlighted by Langer et al. in another recent Bulletin article, the evidence in most systematic reviews originates from high-income countries (4). This is likely to continue to be the case for the foreseeable future. Evidence-based policy-making in LMICs would benefit if the applicability of reviews conducted elsewhere could be maximized.

Use of health-care interventions in LMICs often depends on factors such as the resources available, the organization of health services, cultural norms, and the physical environment. Considering such contextual factors can enhance the generalizability of systematic reviews and their usefulness to policy-makers in diverse settings. This process is analogous to that of improving the internal validity of systematic reviews by considering inter-study variations in participants, study design, and analytic methods (5). A structured approach for dealing with contextual variation can be either extrinsic or intrinsic to the review process.

An extrinsic approach requires reviewers to give some context-dependent guidance for applying a review's findings. Although the reviewers may not have an in-depth knowledge of all circumstances, they often have considerable expertise in the intervention and alternatives. Furthermore, reviews often provide an opportunity to evaluate studies conducted in a variety of settings. Generalizability can be tackled by considering the following questions:

1. Relative importance of the health problem: Do the occurrence and severity of the health problem vary significantly between settings and how might this affect the intervention's potential benefit to the population?

2. Relevance of outcome measures: Are there different outcome measures that might be more or less meaningful in different settings?

3. Practicality of the intervention: What factors, if any, might significantly affect the feasibility of the intervention in different settings?

4. Appropriateness of the intervention: Are there other interventions for achieving the stated goal that might be more appropriate in some settings?

5. Cost-effectiveness of the intervention: Are the costs and benefits of the intervention likely to differ significantly across settings?

Addressing these questions may take as little as one or two paragraphs. Using the example of the effect of streptokinase for the treatment of myocardial infarction, such a paragraph could read: "Acute myocardial infarction is an important problem in all populations and cultures. Streptokinase is an intervention that can be effectively provided in settings where medications can be given intravenously. Alternative treatments do exist, some of which are considerably cheaper (e.g. aspirin), while others are considerably more expensive (e.g. tissue plasminogen activating factor, angioplasty, etc). Other reviews have compared the effectiveness of some alternatives with that of streptokinase. The relative beneficial effect of streptokinase compared to no treatment is likely to be consistent across different risk groups in a range of populations. The cost of streptokinase (around US$ 30 per dose) may vary in different countries. While it is considered a cheap intervention in developed countries, it is relatively expensive in developing countries."

Alternatively, in an intrinsic approach, important contextual dimensions are addressed through planned stratification or subgroup analyses. We recently used this approach in a Cochrane Review of the effectiveness of specialist outreach clinics compared with hospital outpatient clinics (6). A total of 73 studies were identified reporting outreach to nearby urban clinics, such as in the United Kingdom, or to rural communities, including remote populations in Africa, Latin America, Australia and Canada. In some studies outreach was targeted at general populations and in others at disadvantaged subgroups such as indigenous communities or homeless people. The nature of the intervention, the types of outcomes measured, and the effectiveness of outreach all varied according to the setting and population, and we stratified studies accordingly. For example, outreach to urban clinics had minimal effect on access, but could improve health outcomes when combined with education sessions or joint consultations. Outreach to rural and disadvantaged populations led to improved access. We also identified a paucity of high-quality comparative studies in rural and disadvantaged settings, where outreach has potentially the most to offer.

Evidence-based public policy-making is a laudable goal in all settings, but relies on making the most of available evidence. Authors, as well as users, can work towards this goal by producing systematic reviews that are widely applicable.

Competing interests: none declared.

References

1. Volmink J, Siegfried N, Robertson K, Gülmezoglu AM. Research synthesis and dissemination as a bridge to knowledge management: the Cochrane Collaboration. Bulletin of the World Health Organization 2004;82:778-83.

2. Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet 2004;364:1615-21.

3. Horton R. Medical journals: evidence of bias against the diseases of poverty. Lancet 2003;361:712-3.

4. Langer A, Diaz-Olavarrieta C, Berdichevsky K. Why is research from developing countries underrepresented in international health literature, and what can be done about it? Bulletin of the World Health Organization 2004;82:802-3.

5. Egger M, Davey Smith G, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Publishing Group; 2001.

6. Gruen RL, Weeramanthri TS, Bailie RS, Knight SE. Specialist outreach clinics in primary care and rural hospital settings (Cochrane Review). In: The Cochrane Library Issue 3, 2004. Chichester: John Wiley & Sons, Ltd; 2004.

1 Correspondence should be sent to Associate Professor Russell Gruen at this address.