POLICY AND PRACTICE
THEME PAPERS
Bridging the implementation gap between knowledge and action for health
Combler le fossé entre les connaissances théoriques et leur mise en pratique dans le domaine sanitaire
Cerrar la brecha de aplicación entre los conocimientos y la acción sanitaria
Andy HainesI,1; Shyama KuruvillaII; Matthias BorchertIII
IDean, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, England (email: andy.haines@lshtm.ac.uk)
IIDoctoral Student, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, England
IIIClinical Lecturer, Department of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, London, England
ABSTRACT
There is widespread evidence of failure to implement health interventions that have been demonstrated to be cost-effective by high-quality research; this failure affects both high-income and low-income countries. Low-income countries face additional challenges to using research evidence including: the weakness of their health systems, the lack of professional regulation and a lack of access to evidence. There is a need to strengthen institutions and mechanisms that can more systematically promote interactions between researchers, policy-makers and other stakeholders who can influence the uptake of research findings. The concept of public engagement with health research requires a public that is both informed and active. Even when systematic reviews are available further work is needed to translate their findings into guidelines or messages that are understandable to patients and health professionals. Many of the commonly used approaches for keeping health professionals' knowledge up-to-date appear to have small or inconsistent effects. The evidence-base is more extensive for interventions directed towards professionals, such as education, reminders or feedback, than for those directed at organizations or patients. The effect of interventions varies according to the setting and the behaviour that is targeted. Case studies in low-income settings suggest that some strategies can result in increased coverage of evidence-based interventions, but there is a lack of evidence from systematic reviews of rigorous research. Given the potential for near-term improvements in health, finding more effective ways of promoting the uptake of evidence-based interventions should be a priority for researchers, practitioners and policy-makers.
Keywords: Health services research/utilization; Health plan implementation/methods; Policy making; Evidence-based medicine; Information management; Health personnel; Practice guidelines; Review literature; Developing countries (source: MeSH, NLM).
RÉSUMÉ
L'échec de la mise en œuvre d'interventions de santé dont le bon rapport coût-efficacité a été démontré par des travaux de haut niveau est amplement démontré. Ce phénomène touche aussi bien les pays à revenu élevé que les pays à faible revenu. Ces derniers doivent en outre faire face à des difficultés supplémentaires lorsqu'il s'agit de traduire les résultats de la recherche dans la pratique : faiblesse des systèmes de santé, absence de régulation des professions médicales et manque d'accès à l'information. Il est nécessaire de renforcer les institutions et les mécanismes capables de promouvoir plus systématiquement les interactions entre les chercheurs, les responsables politiques et autres partenaires susceptibles d'influer sur l'utilisation des résultats de la recherche. Le concept d'engagement du public en faveur de la recherche en santé implique que ledit public soit à la fois informé et actif. Même lorsqu'il existe des revues systématiques des travaux de recherche, il faut encore en traduire les résultats en directives ou en messages compréhensibles pour les patients et les professionnels de santé. Nombre des approches couramment utilisées pour que les professionnels de santé puissent mettre à jour leurs connaissances semblent n'avoir que peu d'effet. On a en revanche davantage de preuves de l'efficacité des interventions axées sur les professionnels, comme l'éducation, les aide-mémoire ou le retour d'information, que de celles qui visent les organisations ou les patients. L'effet des interventions varie selon le contexte et le comportement visé. D'après des études de cas dans des contextes de faible revenu, certaines stratégies pourraient se traduire par une meilleure couverture des interventions reposant sur des bases factuelles, mais on manque d'informations tirées de revues systématiques de travaux de recherche bien conduits. Etant donné le potentiel d'amélioration de la santé à brève échéance, la découverte de moyens plus efficaces de promotion des interventions reposant sur des bases factuelles devrait être une priorité pour les chercheurs, les praticiens et les responsables politiques.
Mots clés: Recherche en santé publique/utilisation; Mise en œuvre plan sanitaire/méthodes; Choix d'une politique; Médecine factuelle; Gestion information; Personnel sanitaire; Ligne directrice; pratique médicale; Revue de la littérature; Pays en développement (source: MeSH, INSERM).
RESUMEN
Está ampliamente demostrada la incapacidad para ejecutar muchas intervenciones sanitarias cuya costoeficacia ha sido avalada por investigaciones de alta calidad; este fracaso afecta tanto a los países de ingresos bajos como a los de ingresos altos. Los primeros tienen que superar además otros problemas a la hora de usar los datos de investigación, entre los que cabe citar la precariedad de sus sistemas de salud, la falta de regulación profesional y la falta de acceso a los datos relevantes. Hay que fortalecer las instituciones y los mecanismos que permiten promover más sistemáticamente la interacción entre los investigadores, los formuladores de políticas y otros interesados directos que pueden influir en la aplicación de los resultados de investigación. La idea de lograr la participación de la población en las investigaciones sanitarias exige un público documentado y al mismo tiempo activo. Incluso cuando existen revisiones sistemáticas, se requieren trabajos ulteriores para traducir sus resultados en directrices o mensajes que sean comprensibles para los pacientes y los profesionales de la salud. Muchos de los enfoques comúnmente adoptados para mantener actualizados los conocimientos de los profesionales de la salud parecen tener efectos escasos o desiguales. La evidencia disponible es mayor para las intervenciones dirigidas a los profesionales, como la educación, los recordatorios o la retroalimentación, que para las dirigidas a las organizaciones o los pacientes. El efecto de las intervenciones depende de las circunstancias y del comportamiento considerado. Los estudios de casos realizados en entornos de ingresos bajos indican que algunas estrategias pueden traducirse en una mayor cobertura de las intervenciones basadas en la evidencia, pero faltan datos sólidos aportados por revisiones sistemáticas de investigaciones rigurosas. Considerando el potencial existente para introducir mejoras a corto plazo en la salud, la búsqueda de fórmulas más eficaces para promover la aplicación de intervenciones basadas en la evidencia debe ser una prioridad de los investigadores, los profesionales de la salud y los formuladores de políticas.
Palabras clave: Investigación sobre servicios de salud/utilización; Implementación de plan de salud/métodos; Formulación de políticas; Medicina basada en evidencia; Gerencia de la información; Personal de salud; Pautas prácticas; Literatura de revisión; Países en desarrollo (fuente: DeCS, BIREME).
Introduction
There is widespread evidence of failure to implement interventions that have been shown to be cost-effective by high-quality research (1, 2). Studies in both the United States of America and Europe have shown that around 3050% of patients fail to receive clinical interventions that are justified according to the best scientific evidence (3, 4). A recent study looking at primary care in the United Kingdom showed that only around 40% of patients received care that adhered to guideline recommendations for prescribing for four common conditions (5). A review of the uptake of interventions to improve child survival, which was based on data from 42 low-income countries, showed that there was wide variation in the proportion of children receiving the interventions; the proportions ranged from around 90% overall for breastfeeding for a duration of 611 months to 2% for the use of insecticide-treated materials, and the proportion was even lower for other interventions, such as zinc supplementation (6). The authors suggested that around 60% of the 9.7 million deaths among children in the countries studied could be prevented by the use of effective and affordable interventions.
There are many examples of ineffective or inefficient treatments being widely used, thus wasting limited resources and placing additional burdens on overstretched health services. Recently discussed examples are the widespread use of unnecessary injections (7) or caesarean sections that are not clinically indicated (8). In these cases there is a significant risk of harm to patients as well as the waste of limited resources.
While the uptake of research findings can occur through a range of formal and informal processes, effective implementation requires the use of systematic and strategic approaches. This article gives an overview of the effective approaches used to encourage the uptake of research findings for three main groups: policy-makers, the public and health-service providers. It also outlines the additional challenges to implementation in low-income countries and the need for more investment in evaluating implementation strategies.
Role of systematic reviews in providing evidence for research implementation
Where possible this article relies on conclusions arising from systematic reviews. In particular it draws on reviews of "educational, behavioural, financial, organizational and regulatory interventions designed to improve health professional practice and the organization of health care services" published by the Cochrane topic group for Effective Practice and Organisation of Care (EPOC) (9). Some types of interventions have not been studied or are impossible or difficult to study in randomized trials. Therefore studies using other designs are eligible for inclusion in Cochrane/EPOC reviews although they must meet strict quality criteria. Although most systematic reviews deal with strategies to change clinical practice, some reviews cover the use of research evidence by policy-makers, patients and the wider public.
Discussion
The role of policy-makers
Policy-makers play an important role in influencing whether and to what degree research findings influence health services and public health, however it is worth heeding the caution "evidence based policy: proceed with care" (10). For instance, while there may be extensive research on the effectiveness of health-care interventions, there is often less evidence on their cost-effectiveness, implementation, cultural appropriateness and effects on health inequalities, all of which are important considerations for policy-making (10, 11). In some countries there is growing pressure to demonstrate that government funds spent on research and development have tangible benefits for patients and public health (12), but other incentives may drive policy-makers. Elected politicians may feel that their priority is to satisfy the perceived demands of their constituencies, for example by building a hospital catering particularly for the urban elite. In these circumstances research evidence that demonstrates better ways of using resources may be viewed as an irrelevance or even an irritant. The private health-care sector, where for example many unnecessary injections may take place (7), is often outside the control of government policy-makers in the absence of appropriate regulatory mechanisms.
Research findings may influence policy through direct "rational" processes or in other ways (13). Formal and informal interactions between researchers and policy-makers can influence both research and policy processes. Policy-makers may use research tactically or symbolically by commissioning new research to postpone taking decisions or by selectively using evidence to justify decisions already made. Research findings may also gradually percolate into the language and concepts used in policy-making.
A systematic review on the use of evidence by policy-makers concluded that they consider direct interaction with researchers to be the most influential facilitator of research uptake for policy (14). However these direct interactions could be selective or limited in terms of the issues and evidence considered, particularly in light of the volumes of research published every year. Institutions and mechanisms need to be strengthened so that evidence can be more systematically evaluated in the context of policy constraints and interactions between researchers and policy-makers can be more focused (1517). While most studies have been conducted in high-income countries, similar factors appeared to influence policy-makers' use of research in low- and middle-income countries (15). Key processes during which interactions between research and policy should be considered include (15, 16):
setting priorities for and commissioning of research
carrying out the research
synthesizing the evidence
setting policy agendas
formulating policies
implementing policies
evaluating the impact of policies.
The role of professional communicators and "knowledge brokers" (18), who translate and disseminate research findings in an accessible and useful form to policy-makers and other stakeholders requires evaluation.
Use of research evidence by patients and the public
The increased access to information by the public, although conducive to an increased uptake of effective interventions, does not always predict use (19). Information that is not congruent with existing cultural values may not be assimilated, and better informed patients and the public may be more critical or wary of certain health interventions (20, 21).
The public interacts with health research in a variety of ways, for instance by influencing research agendas, contributing to research processes both as participants and by offering lay expertise, as consumers of research-informed health services and products, and by being involved in community and advocacy activities. Enhancing public participation is increasingly seen as a method of promoting and uptake of health research (20, 21). Mechanisms to facilitate a more systematic exchange of knowledge among the public, researchers, health practitioners and policy-makers show potential in promoting the relevance and uptake of health research (1820). An example of these mechanisms would be ensuring that educational curricula at different levels reflect the current synthesis of research evidence (22).
The media can also influence the behaviour of policy-makers, health service providers, industry and the general public (20, 23). Social marketing and direct-to-consumer advertising through various media are widely used to inform the public and influence their opinion on matters related to health interventions (24, 25). The impact of such advertising on health may be positive or negative depending on the nature of the product.
Interaction with health-care providers is still the main means through which the public learns about and uses health interventions (26). Training both health practitioners and consumers how to communicate effectively and helping the public to develop health literacy skills may enhance the utilization of health interventions. Interactive media, including decision aids, can deliver both quantitative data on outcomes and qualitative data on the experience of those outcomes in a personalized way that gives lay people a clear perception of the treatment options open to them and the consequences of each option (27). Tools have also been developed to assist in appraising information materials for patients for accuracy and other characteristics (28).
Mechanisms aimed at increasing the public's engagement with health research and implementation processes have been used in several countries (20, 21, 25); these mechanisms include citizens' juries, public consultations and research review boards. However, public participation in health systems may fulfil a number of purposes, such as helping the system to increase its responsiveness to users' requirements, and these may not always be related to the implementation of research findings.
Service providers
Given the immense volume of research conducted for example, there are more than 400 000 trials in the Cochrane Controlled Trials Register (29) it is not surprising that few practitioners have the time or skills to search for and appraise the relevant evidence. While systematic reviews help practitioners in that they appraise and summarize findings, further work is needed to translate them into guidelines or messages that are understandable to patients and health professionals.
The path from generating evidence to applying it with individual patients or populations has a number of steps (30) (Fig. 1). Even when good quality evidence exists in a form that can be used by health professionals, there are many barriers to uptake (31) (Box 1). These barriers may vary between the different steps on the pathway, the clinical context and the institutional and political setting, and the nature of the research finding. For example, a lack of resources to purchase medicines constitutes a barrier in low-income countries but not in most high-income settings. These barriers can interact in complex ways to retard the implementation of research. For example, those findings that depend for their implementation on the presence of well trained health professionals or advanced technology will face more obstacles than those that depend solely on an intervention that can be delivered even where the health system is rudimentary, such as vaccination campaigns.
Many of the commonly used approaches to keeping health professionals up to date appear to have small or inconsistent effects. For example, merely circulating guidelines or other documentation to health professionals has only a small effect on practice, and traditional educational approaches, such as didactic courses or conferences, give mixed results (32, 33).
An overview of reviews of intervention studies to change practices described the effects of 17 strategies (Table 1) (34). The number of studies addressing each strategy ranged from 298. The evidence-base is more extensive for professionally oriented interventions, such as education, reminders or feedback, than for those directed at organizations or patients. Most interventions had some effect, but the effects varied according to the setting and the targeted behaviour. For example, educational outreach visits seemed particularly useful for influencing prescribing and prevention activities; reminders were especially useful for prevention; and computerized decision support was useful for both drug dosing and prevention activities.
Clinical guidelines have had a chequered history; many early versions were not evidence based, leading to unnecessary and sometimes harmful activities, as well as the loss of credibility of the guideline-development process. Guidance on various aspects of developing guidelines has been published to ensure that the level of evidence used to justify a recommendation is made explicit and that provision is made for regular updating (35).
If guidelines are to be useful in practice, they need to be accompanied by an active dissemination strategy. A review of factors influencing adherence to the recommendations of guidelines found that more complex guidelines were less likely to be followed (36). Other studies have suggested that better adherence is associated with the following facilitating factors (3739):
the guideline addresses an acute rather than a chronic condition
the quality of the supporting evidence is good
the guideline requires few new skills and little organizational change
the guideline is compatible with existing values and
the guideline gives a concrete description of the performance that is advocated.
However, these factors account for less than 20% of the variation in adherence. In general, median absolute improvements in performance are modest, ranging from 14% for reminders to 6% in cluster randomized trials of multifaceted interventions involving educational outreach; however there is considerable variation between studies in the magnitude of effects (40) These overall modest effects may suggest that either barriers are inaccurately characterized or that strategies to overcome them are often ineffective. A review of 235 trials did not support the proposition that combined interventions addressing specific barriers to change are more effective than single interventions. It is difficult to draw generalizable conclusions about appropriate combinations because of the large number of different combinations tested and methodological and reporting weaknesses (40).
Decision analysis combines probabilistic information about the effects of interventions with utilities or values that reflect to what degree a patient (or policy-maker) is prepared to accept any adverse effects against the advantages of a given course of action. It has been used widely in policy and practice to guide the decisions of individual patients, to inform the decisions faced by those who commission services and to influence research design and priority setting (41, 42).
Implementation strategies in low-income and middle-income countries
There is a relative dearth of primary research on health problems that particularly afflict poor communities, and a compelling case has been made for increased investment in such research (43). This deficit is paralleled by a lack of evidence of effective implementation strategies in low-income settings.
Low- and middle-income countries face additional challenges to using research evidence when compared with high-income nations. These challenges include: the weakness of health systems, the lack of professional regulation and opportunities for continuing professional development, the lack of access to research evidence, and the effects of unregulated commercial interests which may, for example, market inappropriate drugs (44). Some national governments have acted to promote research-led practice, for example, Chile has developed a health technology assessment programme; Thailand has implemented evidence-based hospital accreditation; South Africa has given support to systematic reviews; and the Philippines has funded the development of evidence-based guidelines (44).
The EPOC database was scrutinized by MB to establish evidence on the organization of maternal health services in developing countries as well as on strategies to promote the uptake of research findings. Of 51 topics that are possibly relevant to maternal and perinatal health for instance "audit and feedback", "fee-for-service systems of payment", "substitution of doctors by nurses in primary care" reviews have been published for only 14 topics (27%) based on a total of 180 studies, while for 13 topics protocols for future reviews are available (25%); the remaining 24 topics have not yet been addressed. Of all 180 studies, only 11 were conducted in the maternal and perinatal health field (6%). Only 14 studies (8%) came from low- or middle-income countries, and they sometimes originated in unrepresentative settings, e.g. large hospitals in China, Hong Kong SAR and Thailand. No maternal and perinatal health study from a developing country has been included in any of the EPOC reviews.
Likewise, few of the studies on approaches to changing professional behaviour had been conducted in low-income countries. For example, of 18 studies in the systematic review on educational outreach visits, only two were from low-income countries; only four of 32 studies included in the systematic review on continuing education meetings and workshops came from low-income countries; and only 1 of 21 studies on mass media interventions came from a low-income country. Table 2 summarizes the results of studies from low-income countries. There is evidence that educational outreach (two studies from Indonesia) and mixed group discussions with prescribers and patients (one study from Indonesia) are effective in improving drug prescribing and dispensing. A study from Zambia showed a small positive impact of continuing education meetings on case management, including rational drug use.
Some case studies in low-income settings suggest that strategies such as outreach from existing facilities, social marketing, supportive supervision and application of the principles of quality assurance can result in increased coverage of evidence-based interventions (45, 46). One example is the use of social marketing to increase the use of insecticide-treated nets for malaria control among children living in rural areas of the United Republic of Tanzania (47).
The potential for near-term improvements in health to be gained by scaling-up effective interventions is illustrated by the experience of the United Republic of Tanzania's Ministry of Health and its Essential Health Interventions Project (48). Evidence linking decentralized health expenditures to cost-effective interventions and targeting the diseases with the highest burdens was generated, packaged and communicated to local planners using a tool to assist them in mapping their budget and expenditure. Preliminary results suggest that supporting decentralized budgeting in this way could result in rapid changes in expenditure patterns with concomitant improvements in health outcomes in the districts concerned (D. de Savigny, personal communication, 2004). This emphasizes the importance of strengthening management capacity at the peripheral level of health systems. In Uganda, wireless technology has been harnessed to provide access to health information, through the launch of the pilot project of the Uganda Health Information Network, using 200 hand-held computers communicating via battery operated units with a base station in Kampala (49).
These innovations show promise but require further evaluation. The cost-effectiveness of organizational, behavioural, technological, and financial interventions is likely to depend on the health problem being addressed and also on the socio-political context. This means that a number of rigorous evaluation studies in different low-income countries may be needed. To strengthen external validity, evaluations of effectiveness should include investigations of modifiers of contextual effects: qualitative approaches may be needed to improve understanding of how the interventions work or why they do not. Interventions to be tested need refinement through preparatory research so that the likelihood of effectiveness is maximized. Economic evaluation is required to assess the operational costs of the intervention and its consequences in terms of savings or extra expenditures for providers and users of health-care services. In some cases, the cost of implementation may negate the savings theoretically gained from promoting what are thought to be cost-saving interventions (50).
The lack of rigorous evaluations of implementation strategies, particularly in low-income countries, reflects in part the low priority accorded to health and systems research. Many stakeholders involved in implementing public health interventions do not appear to perceive investment in rigorous evaluation to be a priority: they believe they know what should be done, and their main priority is to put their beliefs into practice. In doing so the opportunity to generate robust evidence about how to change policy and practice is lost.
Conclusions
The range of resources and skills required to operationalize effective communication and implementation strategies is rarely available within a single organization, and research institutions may be ill-equipped to undertake these activities effectively on their own. Facilitating interactions between researchers and policy-makers may increase the uptake of research findings and lead to research agendas that more accurately reflect the priorities of policy-makers. A range of specific approaches may be used to promote the use of research evidence. Selecting the best approach for the topic and the context requires an understanding of why current patterns of policy and practice exist, and the strategy selected must be consistent with the local context and the behaviour to be targeted. For example, educational outreach visits may be particularly useful in influencing prescribing and prevention activities. Strategies to communicate research findings to patients and the wider public can have an impact on an individual's decision-making and response to public health messages.
Given the potential for near-term improvements in health resulting from the utilization of research findings for common causes of death in low-income countries, finding cost-effective ways of promoting the uptake of evidence-based interventions should be a priority for researchers, practitioners and policy-makers alike. 
Acknowledgements
We thank Don de Savigny and Jeremy Grimshaw for helpful comments and for providing background material. The work leading to MB's contribution to this paper was sponsored by Johns Hopkins University.
Conflicts of interest: none declared.
References
1. Haines A, Jones R. Implementing findings of research. BMJ 1994; 308:1488-92.
2. Bodenheimer T. The American Health Care system: the movement for improved quality in health care. New England Journal of Medicine 1999;340:488-92.
3. Schuster M, McGlynn E, Brook R. How good is the quality of medical care in the United States? Milbank Quarterly 1998;76:517-63.
4. Grol R. Successes and failures in the implementation of evidence based guidelines in clinical practice. Medical Care 2001;39 Suppl 2:46-54.
5. Freemantle N, Nazareth I, Eccles M, Wood J, Haines A. A randomised controlled trial of the effect of educational outreach by community pharmacists on prescribing in UK general practice. British Journal of General Practice 2002;52:290-5.
6. Jones G, Stekettee RW, Black RE, Bhutta ZA, Morris SS and the Bellagio Child Survival Study Group. How many child deaths can we prevent this year? Lancet 2003;362:6-71.
7. Hutin YJF, Hauri AM, Armstrong GL. Use of injections in healthcare settings worldwide, 2000: literature review and regional estimates. BMJ 2003;327:1075-8.
8. Johanson R, Newburn M, Macfarlane A. Has the medicalisation of childbirth gone too far? BMJ 2002;324:892-5.
9. The Cochrane Collaboration. Effective Practice and Organisation of Care Group. Available from: http://www.epoc.uottawa.ca/
10. Black N. Evidence based policy: proceed with care. BMJ 2001;323:275-9.
11. Macintyre S, Chalmers I, Horton R, Smith R. Using evidence to inform health policy: case study. BMJ 2001;322:222-5.
12. Haines A, Donald A. Making Better Use of Research Findings. BMJ 1998;317:72-5.
13. Weiss C. The many meanings of research utilization. Public Administration Review 1979;39:426-31.
14. Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers' perceptions of their use of evidence: a systematic review. Journal of Health Services Research and Policy 2002;7:239-44.
15. Hanney S, Gonzalez-Block M, Buxton M, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Research Policy and Systems [Online journal] 2003;1:2. Available from: http://www.health-policy-systems.com/content/1/1/2
16. Lavis JN, Robertson D, Woodside J, McLeod CB, Abelson J. How can research organizations more effectively transfer research knowledge to decision-makers? Milbank Quarterly 2003;81:221-48.
17. Denis JL, Lomas J. Convergent evolution: the academic and policy roots of collaborative research. Journal of Health Services Research and Policy 2003;8 Suppl 2:1-6.
18. Canadian Health Services Research Foundation/Fondation canadienne de la recherche sur les services de santé. The theory and practice of knowledge brokering in Canada's health system, 2003. Available from: http://www.chsrf.ca/brokering/pdf/Theory_and_Practice_e.pdf
19. Pleasant A, Kuruvilla S, Zarcadoolas C, Shanahan J, Lewenstein B. A Framework for assessing public engagement with health research: report to WHO, Department of Research Policy and Cooperation. Geneva: World Health Organization; 2003.
20. Irwin A. Constructing the scientific citizen: science and democracy in the biosciences. Public Understanding of Science 2001;10:1-18.
21. Dierkes M, von Grote C, editors. Between understanding and trust: the public, science and technology. Amsterdam: Harwood Academic; 2000.
22. Fields H, Powell P. Public understanding of science vs public understanding of research. Public Understanding of Science 2001;10:421-6
23. Grilli R, Ramsay C, Minozzi S. Mass media interventions: effects on health services utilisation. Issue 2. Oxford: Update Software; 2004 (Cochrane Review).
24. Nutbeam D. Evaluating health promotion progress, problems, and solutions. Health Promotion International 1998;13:27-44.
25. Piotrow PT, Kincaid DL, Rimon II JG, Rinehart W. Health communication, lessons from family planning and reproductive health. Westport (CT): Praeger; 1997.
26. Zarcadoolas C, Pleasant A, Greer D. Elaborating a definition of health literacy: a commentary. Journal of Health Communication 2003;8 Suppl 1:119-20.
27. O'Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner M, et al. Decision aids for people facing health treatment or screening decisions. Issue 3. Oxford: Update Software; 2003. (Cochrane Review.)
28. Charnock D, Shepperd S, Needham G, Gam R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health 1999;53:105-11.
29. Cochrane Controlled Trials Register. 2nd ed. Oxford: Update Software; 2004.
30. Haynes RB, Haines A. Barriers and bridges to evidence based practice. BMJ 1998;317:273-6.
31. Haines A, Rogers S. Integrating research evidence into practice. In: Silagy C, Haines A, editors. Evidence-based practice in primary care. 2nd ed. London: BMJ Publishing Group; 2001. p. 157-74.
32. Grimshaw J, Eccles M, Walker A, Thomas R. Changing physicians' behaviour: what works and thoughts on getting more things to work. Journal of Continuing Education in the Health Professions 2002;22:237-43.
33. Davis DA, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conference, workshops, rounds, and other traditional continuing education activities change physician behaviour or health care outcomes? JAMA 1999; 282: 867-74.
34. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patient care. Lancet 2003:362;1225-30.
35. The AGREE collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Quality and Safety in Health Care 2003;12:18-23.
36. Grilli R, Lomas J. Evaluating the message: the relationship between compliance rate and the subject of a practice guideline. Medical Care 1994;32:202-13.
37. Grol R, Dalhuijsen J, Thomas S, In't Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guideline in general practice: observational study. BMJ 1998;317:858-61.
38. Foy R, MacLennan G, Grimshaw J, Penney G, Campbell M, Grol R. Attributes of clinical recommendations that influence change in practice following audit and feedback. Journal of Clinical Epidemiology 2002;55:717-22.
39. Burgers J, Grol R, Zaat J, Spies T, Van der Bij A, Mokkink H. Characteristics of effective clinical guidelines for general practice. British Journal of General Practice 2003;53:15-9.
40. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technology Assessment 2004;8:1-72.
41. Kramer MS, Haas J, Kelly A. Maternal anthropometry-based screening and pregnancy outcome: a decision analysis. Tropical Medicine and International Health 1998;3:447-53.
42. Bell JC, Rose DN, Sacks HS. Tuberculosis preventive therapy for HIV-infected people in sub-Saharan Africa is cost-effective. AIDS 1999;13:1549-56.
43. Global Forum for Health Research. The 10/90 gap in health research 2000. Geneva: Global Forum for Health Research; 2000.
44. Garner P, Smith H, Kale R, Dickson R, Dans T, Salinas R. Implementing research findings in developing countries. In: Haines A, Donald A, editors. Getting research findings into practice. 2nd ed. London: BMJ Publishing Group; 2001.
45. Bryce J, el Arifeen S, Pariyo G, Lanata CF, Gwatkin D, Habicht J-P and the multi-country evaluation of IMCI Study Group. Reducing child mortality: can public health deliver? Lancet 2003;362;159-64.
46. Marquez L, Kean L. Making supervision supportive and sustainable: new approaches to old problems. Population Reports 2002;4 Suppl:1-27 (Maximising Quality and Access Initiative paper No. 4).
47. Armstrong Schellenberg JA, Abdulla S, Nathan R, Mukasa O, Marchant TJ, Kikumbih N, et al. Effect of large-scale social marketing of insecticide-treated nets on child survival in rural Tanzania. Lancet 2001;357:1241-7.
48. International Development Research Centre. Tanzania essential health interventions project. Available from: http://www.idrc.ca/tehip
49. Uganda Health Information Network. Satellife. Available from: http://pda.healthnet.org
50. Mason J, Freemantle N, Nazareth I, Eccles M, Haines A, Drummond M. When is it cost-effective to change the behaviour of health professionals? JAMA 2001;86:2988-92.
(Submitted: 1 December 2003 Final revised version received: 26 April 2004 Accepted: 28 April 2004)
1 Correspondence should be sent to this author.
References (Table 2)
51. Angunawela II, Diwan VK, Tomson G. Experimental evaluation of the effects of drug information on antibiotic prescribing: a study in outpatient care in an area of Sri Lanka. International Journal of Epidemiology 1991;20:558-64.
52. Bexell A, Lwando E, von Hofsten B, Tembo S, Eriksson B, Diwan VK. Improving drug use through continuing education: a randomized controlled trial in Zambia. Journal of Clinical Epidemiology 1996;49:355-7.
53. Hadiyono JE, Suryawati S, Danu SS, Sunartono, Santoso B. Interactional group discussion: results of a controlled trial using a behavioral intervention to reduce the use of injections in public health facilities. Social Science and Medicine 1996;42:1177-83.
54. Westphal MF, Taddei JA, Venancio SI, Bogus CM. Breastfeeding training for health professionals and resultant institutional changes. Bulletin of the World Health Organization 1995;73:461-8.
55. Ross-Degnan D, Soumerai SB, Goel PK, Bates J, Makhulo J, Dondi N, et al. The impact of face-to-face educational outreach on diarrhoea treatment in pharmacies. Health Policy and Planning 1996;11:308-18.
56. Santoso B, Suryawti S, Prawaitasari JE. Small group intervention vs formal seminar for improving appropriate drug use. Social Science and Medicine 1996;42:1163-8.
57. Brasca A, Pigliacampo J, Pollastri E, Hidalgo A, Ramadan A, Miranda J. An awareness campaign on the diagnosis and prevention of colon and rectal cancer: planning and evaluation of its results. Acta Gastroenterologica Latinoamericana 1987;17:325-9.
58. French GL, Cheng AF, Wong SL, Donnan S. Repeated prevalence surveys for monitoring effectiveness of hospital infection control. Lancet 1989;2:1021-3.
59. Hong SW, Ching TY, Fung JP, Seto WL. The employment of ward opinion leaders for continuing education in the hospital. Medical Teacher 1990;12:209-17.
60. Seto WH, Ching TY, Yuen KY, Chu YB, Seto WL. The enhancement of infection control in-service education by ward opinion leaders. American Journal of Infection Control 1991;19:86-91.