EDITORIALS

Fair tests of health-care policies and treatments: a request for help from readers

Andrew D Oxman^{I, *}; Iain Chalmers^II

^INorwegian Knowledge Centre for Health Services, Oslo, Norway
^IIJames Lind Library, Oxford, England

Health-care policies impact on peoples' lives. For example, a policy decision not to have publicly funded health insurance with universal coverage limits peoples' choices to what they can afford. Those who make policy decisions are ethically and politically bound to make decisions that are in the interests of the people whom they serve. Evaluating the effects of policies is important because this is the only way of knowing the extent to which policies are doing more good than harm.

We would appreciate assistance from Bulletin readers to address the question: what is a fair test of a health-care policy? There are three ways in which you can help us:

1. Provide examples of randomized evaluations of health-care policies.

There are many ways in which health-care policies can be evaluated. The advantage of randomized evaluations is that they help to ensure that like is compared with like. Recent examples of randomized trials of policies relevant to reducing neonatal mortality include evaluations of community-based behaviour change management,¹ home care versus community care,² and community participation with women's groups.³ Examples of randomized trials of pharmaceutical policies include evaluations of the impacts of supervision on stock management and adherence to treatment guidelines, trials of strategies for improving adherence to prescribing guidelines and a trial of reimbursement restriction.^4–6 An inventory of randomized evaluations such as these would make it easier for policymakers to find and use this evidence. Other benefits would include illustrating: (i) the principles of fair tests of health-care policies, (ii) the feasibility of randomized evaluations of health-care policies, (iii) practical strategies for addressing the challenges of randomized evaluations of health-care policies, and (iv) limitations of randomized evaluations of health-care policies.

Because these evaluations may be difficult to identify, we would be grateful if Bulletin readers could send us examples of randomized evaluations of health-care policies, particularly if they have not been published in journals, or have been reported in languages other than English. Specifically, we welcome examples of cluster randomized trials of arrangements for delivering, financing or governing health-care services, and of public health interventions.

2. Provide examples of compelling evidence from non-randomized evaluations of health-care policies.

There may be practical reasons and other arguments for using non-randomized comparisons and sometimes these provide compelling evidence, particularly when there are dramatic effects of policies. Non-randomized comparisons include interrupted time-series analyses, controlled before–after studies, uncontrolled before–after studies and cross-sectional studies. We would be pleased if Bulletin readers could help us identify good examples of individual non-randomized evaluations of the impacts of health-care policies, or systematic reviews of them, including a brief comment on what makes the results of these evaluations compelling.

3. Provide early examples of treatment evaluations.

Evaluations of preventive, therapeutic and rehabilitative treatments inform health-care policy decisions, as well as clinicians' and individuals' decisions about health care. An article in the news section of this issue of the Bulletin introduces readers to the James Lind Library (available at: www.jameslindlibrary.org) – a multilingual web site explaining and illustrating the features of fair tests of treatments,^7,8 with explanatory essays in Arabic, Chinese, English, French, Portuguese, Russian and Spanish. These essays explain the need to: (i) make unbiased treatment comparisons, by minimizing differences between the people compared and in assessing treatment outcomes, including unanticipated outcomes; (ii) take account of the effects of the play of chance, when interpreting unbiased comparisons; and (iii) review systematically all the relevant evidence from fair tests, minimizing the effects of biased reporting and selection of evidence, and using meta-analysis to reduce the effects of the play of chance, when appropriate and possible.

To illustrate these principles and their application, the James Lind Library draws on an archive of hundreds of records, from ancient to modern times. The authors of these records come from many countries, and although the majority of their contributions are written in English, many of them are in other languages.

We would be grateful if readers would draw our attention to additional records eligible for inclusion in the James Lind Library if you know of examples outlining principles of fair tests of treatments and the need for comparisons published before the end of the 19th century, or examples of studies using alternation, random allocation or blinded outcome assessment published before 1950.

If you are aware of examples relevant to any of the three categories described above, please send us copies of them, identifying the key passages and providing a translation if the text is not in English, by post, facsimile or e-mail to: Bulletin of the World Health Organization Project, c/o James Lind Initiative, Summertown Pavilion, Middle Way, Oxford OX2 7LG, England. Fax: +44 1865 516 311; e-mail: feedback@jameslindlibrary.org. Your help will be acknowledged explicitly unless you instruct us otherwise.

References

Available at: http://www.who.int/bulletin/volumes/87/6/09.066787/en/index.html

References

1. Kumar V, Mohanty S, Kumar A, Misra RP, Santosham S, Awasthi S, et al. Effect of community-based behaviour change management on neonatal mortality in Shivgarh, Uttar Pradesh, India: a cluster-randomised controlled trial. Lancet 2008;372:1151-62. PMID:18926277 doi:10.1016/S0140-6736(08)61483-X        

2. Baqui AH, El-Arifeen S, Darmstadt GL, Ahmed S, Williams EK, Seraji HR, et al. Effect of community-based newborn-care intervention package implemented through two service-delivery strategies in Sylhet district, Bangladesh: a cluster-randomised controlled trial. Lancet 2008;371:1936-44. PMID:18539225 doi:10.1016/S0140-6736(08)60835-1        

3. Manandhar DS, Osrin D, Shrestha BP, Mesko N, Morrison J, Tumbahangphe KM. Effect of a participatory intervention with women's groups on birth outcomes in Nepal: cluster-randomised controlled trial. Lancet 2004;364:970-9. PMID:15364188 doi:10.1016/S0140-6736(04)17021-9        

4. Trap B, Todd CH, Moore H, Laing R. The impact of supervision on stock management and adherence to treatment guidelines: a randomized controlled trial. Health Policy Plan 2001;16:273-80. PMID:11527868 doi:10.1093/heapol/16.3.273        

5. Rx for change. Ottawa: Canadian Agency for Drugs and Technologies in Health;2009. Available from: http://www.cadth.ca/index.php/en/compus/optimal-ther-resources/interventions [accessed on 23 April 2009]          .

6. Schneeweiss S, Maclure M, Carleton B, Glynn RJ, Avorn J. Clinical and economic consequences of a reimbursement restriction of nebulised respiratory therapy in adults: direct comparison of randomised and observational evaluations. BMJ 2004;328:560-7. PMID:14982865 doi:10.1136/bmj.38020.698194.F6        

7. The James Lind Library Editorial Team. Chalmers I, Milne I, Tröhler U, Vandenbroucke J, Morabia A, Tait G, Dukan E. The James Lind Library: explaining and illustrating the evolution of fair tests of medical treatments. J R Coll Physicians Edinb. 2008;38:259-64. PMID:19227602        

8. Senior K. Unique, multilingual resource on testing health-care treatments. Bull World Health Organ 2009;87:412-413.         

* Correspondence to Andrew D Oxman (e-mail: oxman@online.no).