Policy and Practice

Information systems: the key to evidence-based health practice

Roberto J. Rodrigues ¹

Increasing prominence is being given to the use of best current evidence in clinical practice and health services and programme management decision-making. The role of information in evidence-based practice (EBP) is discussed, together with questions of how advanced information systems and technology (IS&T) can contribute to the establishment of a broader perspective for EBP. The author examines the development, validation and use of a variety of sources of evidence and knowledge that go beyond the well-established paradigm of research, clinical trials, and systematic literature review. Opportunities and challenges in the implementation and use of IS&T and knowledge management tools are examined for six application areas: reference databases, contextual data, clinical data repositories, administrative data repositories, decision support software, and Internet-based interactive health information and communication. Computerized and telecommunications applications that support EBP follow a hierarchy in which systems, tasks and complexity range from reference retrieval and the processing of relatively routine transactions, to complex ‘‘data mining’’ and rule-driven decision support systems.

Keywords: Information systems, trends; information technology; Internet; management information systems; decision support techniques.

The evidence-based practice movement

Most clinical practice is based on limited evidence, mostly in the form of textbook information, obsolete premises, untrustworthy research or case studies, partial or unendorsed reviews, and anecdotal or personal clinical experience. Proven therapies backed by ample evidence are underutilized for lack of knowledge or grasp of available evidence and, often, clinicians do not believe that results observed in clinical trials can be directly translated into clinical practice (1). A major contribution to the call for a more rational approach to clinical decision-making was the growth in clinical, administrative, and outcomes research studies, and the increasing interest in the determinants of health care costs. Evidence-based practice (EBP), which gained impetus in the early 1990s, replaces the traditional model of ‘‘medicine by authority’’, and employs a more discriminating scientific approach that avoids the use of unendorsed and unsystematic information (2, 3). These developments have made it mandatory for health care professionals and managers to be aware of, critically appraise, and systematically make use of peer-reviewed published data (4).

The EBP movement is an effort to teach direct care professionals to evaluate research evidence and apply it to clinical activities. It requires that the results of primary research be compiled in a methodical way and made accessible to those involved in the decision-making process (5). It enhances knowledge by building on clinical expertise and verifiable facts about disease mechanisms. Ultimately, EBP attempts to formalize the processes and principles that the best clinicians have perfected over time (3). Failure to use evidence systematically results in a considerable time lag before effective therapies come into use and ineffective ones are withdrawn. The introduction of evidence-based clinical practice has had a positive effect on medicine and pharmacy, especially in areas involving drug therapy. Concepts of evidence-based practice have been increasingly adopted by physicians’ training programmes that emphasize problem-solving and information search skills (6–8).

How information systems support evidence-based practice

Information systems can capture, transform and maintain data at three levels: raw data, processed data, and knowledge (9). If the raw data is valid, then the processed data, or ‘‘information’’, can be considered as equivalent to evidence. Knowledge is information (evidence) in context. It can be seen as an intellectual construct of a higher order, in which evidence from various fields and sources is linked, validated, and correlated to established scientific truths, thus becoming a generally accepted body of wisdom.

Information and knowledge management is at the heart of the health care worker’s professional, intellectual and practical activities. To put in proper perspective the idea of information as the essential component of appropriate decision-making, consider the costly mistakes that stem from incomplete information: of the roughly one trillion dollars spent on health care in the USA in 1998, an estimated 25% was on unnecessary or avoidable care, redundant tests, and excessive administrative costs.

Most of the literature on EBP emphasizes the use of formally published technical and scientific material (7, 10–12). From this perspective, evidence is understood as ‘‘research evidence’’, i.e., peer-reviewed experimental primary literature or explicitly derived sources such as systematic reviews of a large number of accredited publications and clinical trials (13–17). Although the experimental primary literature and systematic reviews are the driving forces of EBP, it is clear that these are not the only sources of evidence. The applicability of all the available ‘‘best evidence’’ to individual patient care in a particular management environment requires other information as well, which may be of a contextual or local nature. The integration and exchange of clinical and administrative best practice information among health professionals outside the restricted scope of the technical and scientific literature has been shown to be a significant factor in appropriate decision-making (18).

From a broader perspective, evidence-based decisions must also include sources of information such as those derived from unendorsed sites and from the application of ‘‘knowledge engineering’’ technologies to clinical and administrative repositories of health data and other resources. Information systems and, particularly, emerging Internet-based interactive health communication technologies and expert systems, offer great potential for accessing evidence-based knowledge on demand. This includes the possibility of creating clearinghouses for public domain information materials, tools, and resources, and an enhanced ability to provide widespread dissemination and immediate updating of content or functions (19).

As shown in Fig. 1, information systems and technology (IS&T) contributions to evidence-based practice are accomplished through six application areas:

• reference databases (A)

• contextual and case-specific information (B)

• clinical data repositories (C)

• administrative data repositories (D)

• decision support software (E)

• Internet-based interactive health information (F).

Reference databases

Data obtained from basic investigation and clinical observation are made available with other relevant information of the biomedical literature (books and periodicals), the formal publications of clinical trials and reviews, and the records of current research. Jointly, they represent a body of biomedical knowledge that can be used to produce clinical guidelines, protocols of care, and support decision-making (7, 10–17).

Reference databases containing formal scientific and technical literature ‘‘hard’’ data are certainly very useful in a research environment but less so in day-today practice in a non-academic environment. What most clinicians want are fast, up-to-date, and structured concise responses to focused queries about health conditions. Conventional reference databases are excellent for finding good studies with details of successful and unsuccessful health interventions, but their use is difficult and time-consuming. Searches of the formal biomedical literature in any subject bring up a large number of references that are hard to sort and appraise and are not displayed in the format or with the coherence that can provide the relevant, orderly, and concise answers required by practical physicians.

Randomized clinical trials and systematic reviews of peer-reviewed primary research and trials, on the other hand, provide robust, coherent, and systematic evidence about the effectiveness of health interventions. The Cochrane Collaboration Library (20, 21), the product of an attempt to overcome the limitations of reference databases, reviews current research findings and informal clinical experience by presenting knowledge about well-defined domains in a structured way that, generally, responds to the expectations and needs of health practitioners. It was established as part of an international effort to facilitate the preparation, maintenance, and dissemination of systematic reviews of the results of health care interventions. The Cochrane Collaboration Library goes a long way towards meeting the criteria for an ideal tool EBP tool (11).

Librarians play an important function in the expansion and support of EBP because of their role in identifying and retrieving appropriate literature from a variety of sources and provide indispensable help to managers and clinicians in indexing and searching for information on diagnosis, etiology, therapy and prognosis. In addition, librarians have an important role in the conduct of systematic reviews and meta-analyses, and providing support for the development of clinical practice guidelines (13). In most circumstances, due to the volume of published material, the complexity of literature search, and the need to cross-link references, the traditional methods used in literature searches are inadequate and automated tools have been developed to facilitate the technical tasks involved (22–25). The positive impact of the incorporation of critically reviewed literature sources in evidence-based decisions in clinical medicine has been documented by the measurable improvement in the quality of care and outcomes (26).

Contextual and case-specific information

Historically and pragmatically, on a regular basis medical and nursing practitioners make use of unreviewed and unvalidated sources of ‘‘soft’’ data and evidence as the basis for their decision-making; they can be categorized as contextual and case-specific sources. Contextual information refers to environmental, socioeconomic, and epidemiological factors for a particular site and time. Case-specific information is the accumulated data on current and past encounters with the health care system; often voluminous, it is represented by the clinical and administrative data registered in the individual medical and business (insurance, patient accounts, etc.) transactional records.

Environmental, socioeconomic, anthropological, and epidemiological contextual evidence is of great significance in clinical decision-making and indeed even anecdotal evidence may be found to be the most decisive factor in the selection of a course of action (27). Generally short-lived, limited in geographic scope, and not subject to a rigorous systematic review followed by formal literature publications, the main value of this type of evidence is specificity and timely availability to the decision-maker. Such ‘‘soft’’ data, although it may only be valid for a very specific place and a very short time, can be extremely valuable for decision-makers. Weekly reporting on the incidence of communicable diseases, for instance, illustrates the importance of contextual information as evidence in support of clinical diagnosis and identification of emerging diseases. The enhancement of data definitions and quality control in collecting and processing morbidity and mortality data, and advances made in the reporting and communication on the incidence and prevalence of communicable diseases and environmental health risk factors, have established the importance of epidemiological and environmental evidence in clinical decision-making. The US Centers for Disease Control and Prevention publications and Internet site, and the epidemiological information dissemination centres maintained by national, state, and municipal health authorities in most countries help to meet the unquestionable need for this kind of information. From this contextual perspective, the ‘‘hard’’ evidence of peer-reviewed material may be questioned and critically appraised in terms of its scientific, social, cultural and local applicability. This approach counterbalances the one-sidedness of knowledge in contemporary medicine, in which practitioners are trained to maintain high standards of critical consciousness in scientific methodological domains but not in the broader sociocultural domain (28, 29).

The best scientific research is useless as evidence if not applied in the context of case-specific information. Case-specific use of evidence is not new: ‘‘Life is short, and art long; the crisis fleeting; experience perilous, and decision difficult’’, commented Hippocrates of Cos (30) about 2400 years ago, in regard to clinical judgement. He was a keen observer of clinical signs and symptoms and the long-lasting wisdom captured in his works reflects the concern for the careful inspection of the sick, for the systematic observation and recording of the sequence of phenomena as their condition evolved, the contribution of environmental factors, and the interventions that proved to be conducive to a favourable outcome. Unquestionably, he taught and practised evidence-based health interventions, and recognized the complexity of determining cause–effect relationships, the hazards of clinical trials (‘‘experience perilous’’), and all the difficulties involved in appropriate case-specific decision-making. Case-specific evidence, recorded in the patient’s clinical documentation, must be always taken into consideration, and its use optimized (18, 31).

Clinical data repositories

Data fragmentation, lack of structure, incompatible terminologies, the separation of clinical from financial and administrative data, and the break-up of patient data over time and space all prevent the use of the wealth of data already stored in non-automated medical and administrative records. Moreover, standardization is a prerequisite for the upstream consolidation of individual patient clinical and administrative data. Record integration at the individual level also requires a unique identifier and uniform clinical encounter data sets and classification terminologies, which are in most cases still being designed and tested.

The possibility of capturing data at the point of care and the emergence of computer-based structured health records (electronic medical record, EMR) open up a whole new spectrum of options for improving access to individual and collective patient data residing in data repositories, independently of the site of care (32). Data-related standards, however, are still a major issue in operating the EMR, although in a number of organizations most of the standardization issues have been dealt with despite major problems related to the categorization of clinical interventions and codification of health-related conditions. A growing effort is being made to reach consensus on the remaining national and international issues to be resolved.

Data warehousing technology makes it possible for any authorized user to merge and access digital data from various sources. When used in conjunction with knowledge engineering techniques and products, the integration of digitized clinical, administrative, and financial data can be advantageously used in decision-making about treatment and other health interventions (32). From a population perspective, collections of structured records can be maintained in clinical and administrative data repositories and selectively extracted to assist in the construction of clinical guidelines and in the definition of administrative procedures. New software called ‘‘intelligent agents’’ makes it possible to extract selectively and manipulate data from large repositories (33). The use of clinical and administrative data repositories is becoming a viable and affordable method of supporting EBP, although there are still a number of problems, the most significant being: data capture, consolidation, and reliability; standards; and confidentiality issues.

Evidence-based decision-making is also changing management practices, gaining acceptance in areas such as purchasing, contracting and resource allocation, by taking advantage of cumulative regional databases of past activities to support administrative decisions. A number of support information tools now exist to manage human, material, and financial resources, selection of products and suppliers, and the like.

The security and privacy issues regarding access to and control of data are particularly serious. The transformation of data on paper, today stored in different physical sites, into logically integrated digital data, easily retrieved through a variety of access points, and the accessibility of data on identifiable individuals, creates the possibility of misuse and fraud. Although some of the possible uses of detailed patient data, such as population-based studies on the efficacy of alternative interventions, need not involve the identification of individual sources, many others do require such identification. On the other hand, tasks such as reimbursement, use of composite individual records, or auditing for fraud and abuse, require access to patient-specific data outside the care setting. From a technical perspective alone, the merging of patient data from diverse sources necessitates identification with its many attendant security and confidentiality issues.

‘‘Push’’ techniques, whereby health managers use patient-specific information to identify candidates for potential interventions and reach-out health programmes, even though considered in most circumstances justifiable from the public health standpoint, can meet with strong opposition from civil society. An example is the controversy, in the USA in 1998, over chain pharmacies that disclosed patient-specific data to third parties (34).

Decision support software

Clinical decision software applications are designed to support the analysis of patient data and to automate aspects of clinical decision-making that can be expressed as explicit and reproducible rules that are built and maintained by a database of guidelines. The implications of the mainstream introduction and use of clinical decision-support software are many and far-reaching. It raises questions such as the following:

– Does the amount of automated ‘‘clinical thinking’’ done by the software application constitute the ‘‘practice of medicine’’ with all its ethical and legal implications?

– Is the expert software to be considered a medical device, subject to regulation?

– To what extent does the software or its documentation allow clinicians to examine independently the underlying logic involved and evaluate the validity of the automated conclusions?

– What is the role of such products in critical clinical decisions?

– Who will take the blame for errors?

Clearly there is a vast array of problems with expert systems that have not yet been faced. They are important if only because the health information technology industry is investing millions of dollars in developing such applications. A large number of policy, legislation, and regulation issues in this area are in urgent need of attention (35).

Internet-based interactive health information

The growing area of interactive health communication (IHC) is defined as ‘‘the interaction of an individual consumer, patient, care-giver, or professional, with or through an electronic device or communication technology to access or transmit health information or to receive guidance and support on a health-related issue’’ (36). Advances in telecommunications, computer applications, and user interfaces are changing the nature of interaction between health professionals, the population, and the ubiquitous World Wide Web (Internet) global communication network. The positive health impact of these technologies is seen in informed decision-making, the promotion of healthy behaviour, the facilitation of information exchange between peers, self-care, and the management of the demand for health services.

‘‘Old’’ media, including radio, television, and print have effectively contributed to knowledge dissemination and health promotion. IHC applications expand those functions and create new alternatives by changing the very nature of health communications and health practice. They are accelerating the empowerment of providers and patients, and making it easier for consolidated entities to operate within increasingly large and complex organizations. The new opportunities offered by Internet-based IHC dramatically expand possibilities in areas such as the following (36):

– access to health information and support on demand in any site via a plain telephone line or any variety of wireless communication options;

– opportunity for interactions among users of electronic messaging;

– capacity for wide dissemination and for keeping contents and functions up to date;

– opportunity to tailor information to the specific needs of individuals or user groups;

– multimedia resources, combining text, audio, and visuals;

– opportunity for users to remain anonymous, thus facilitating access to sensitive information and frank discussions about health status, behavioural risks, and fears and uncertainties.

The main health concern about the widespread use of interactive Web-based technologies is the volume of unendorsed, non-validated, misleading, fraudulent, and potentially harmful health information available on the Internet (37). Obviously, the general public has an enormous interest in obtaining health information; according to one study, of the 50 million adult users of Internet in the USA in 1997 and 1998, over 17 million searched it for health information. This number was expected to double in the following two years (38).

IS&T and knowledge management challenges

How to approach the utilization of clinical and management health information is still a complex, chaotic, and controversial subject. It is not surprising that many expectations regarding the contribution of health information systems to clinical practice have not been fulfilled. The use of patient medical records and encounter data as sources of evidence requires the redesign of clinical records, methods to capture and organize data, computerization of records, and methods for data searching and interpretation. Many investigators have attempted to extract meaningful data from textual material recorded in clinical encounters, but the handling of medical record text is fraught with serious difficulties because of the idiosyncrasies of medical records. An example of a solution to the management of clinical documentation is HEALNet, a project aimed at extracting information from medical text in support of EBP, using statistical concept representation and the construction of a core information retrieval engine (39).

Most of the innovative development and use of informatics products for health will be carried out by profit-making organizations. While the private sector is promoting the rapid development of new health information technologies, the public sector has been mostly passive and may even set up roadblocks, intentionally or otherwise, to the spread of technology. Questions of privacy, software regulation, and ethical and legal aspects of telecommunication in health care may be the main sources of conflict (34–40). The industry, on the other hand, is prone to misjudge the reality and the factual needs of the health sector, which may result in failed projects and unsuitable products.

Although information technology has the potential to sharpen the focus of health care systems on patients’ needs and preferences, this will only be achieved by a concerted effort on the part of professionals, governments and industry to solve the problems we have enumerated (41–43). Health reform ‘‘industrialization’’ involves statistical process control, operations research and re-engineering techniques, line-employee (direct care provider) performance measurement, benchmarking, and outcomes measurement. New techniques also being used include programme budgeting and marginal analysis studies, an economic evaluation combining practice data on the ‘‘before period’’, and literature data to model the ‘‘after period’’ (44).

These initiatives require vast quantities of health care data and a wealth of data-processing and analytical intelligence. The ultimate goal of health care industrialization is to provide every clinical situation, no matter how unique, with automated digital support that will generate quantitative predictors, optimized (evidence-based) decision-making, and maximized benefits in the form of lower costs, improved quality or a combination of both.

The computerized applications to support such objectives must follow a hierarchy in which systems tasks range in complexity from reference retrieval and the processing of transactions to more complex decision support systems. The functions required by EBP in management and clinical service delivery involve the whole range of the spectrum of applications. They include static and dynamic modelling routines driven by rules, normative knowledge databases, and vast stores of clinical and administrative data. Their outputs consist of quantitative comparisons of outcomes associated with alternative managerial and medical decisions.

Dissemination and adoption of evidence

The dissemination and adoption of evidence and guidelines present special problems. Segmentation of the target audience is necessary for effective dissemination of knowledge. A study in the Netherlands, conducted among family physicians to evaluate the adoption of evidence-based practice guidelines, indicated that for some practitioners it was desirable to make evidence quickly available through publications; for others it was more effective to spread the guidelines through local networks; for others still, a more active, personal approach was needed (45).

It has been suggested that the most important change required today in the teaching of medicine is to foster the aptitude of future professionals for information management. Training in this area has been considered a major component in the implementation of EBP, and evidence-based reasoning is an increasingly important area in medical school curricula and continuing medical education (7).

Getting evidence into practice requires resources and methods (46–48). To use evidence-based decision-making one must have organizations that have mechanisms for finding and appraising evidence and professionals who are skilled in searching, evaluating, storing, and using information and knowledge. In addition to the need for economic and organizational discipline, true evidenced-based reform in health care calls for full participation of organizations and professionals in a still incipient health information revolution — a key symptom of the need for transformation in the health information infrastructure is the virtual lack of systematic and uniform operations-measurement techniques, tools, and quality control data in most health organizations. These were not needed in the previous public cost-plus or private fee-for-service models of care but are now indispensable for the practice of evidence-based care.

Reviewing the progress of evidence-based practice, it appears that what works best, at least in some environments, is the adoption of collaborative benchmarking and practice-based assistance (49). The factors reviewed in this article represent major challenges for medical informatics and health managers and professionals. They make a case for significant public investment in technology development and deployment through collaborative research efforts involving industry, academic centres, research centres, the health establishment, and government.

Résumé

Systèmes d’information : conditions requises pour une pratique médicale fondée sur l’expérience clinique

On privilégie de plus en plus le recours aux meilleures données du moment, qu’il s’agisse de la pratique clinique, des services de santé, ou de la prise de décision en matière de gestion des programmes. Cet article présente une analyse des questions concernant le rôle de l’information dans la pratique fondée sur l’expérience, un processus qui consiste à traduire des problèmes cliniques en interrogations et à systématiquement situer, évaluer et employer des sources validées, une recherche et des examens critiques en rapport avec le problème, avant de prendre des décisions. L’élaboration, la validation et l’utilisation de toutes sortes de sources de données et de connaissances, qui vont bien au-delà du modèle bien connu de la recherche, des essais cliniques et de l’analyse systématique de la littérature, y sont examinées. L’apport des systèmes d’information et des technologies de pointe à ces pratiques fondées sur l’expérience s’articule autour de six domaines d’application : bases de données de référence, informations contextuelles et spécifiques de cas, archives de données cliniques, archives de données administratives, logiciels d’aide à la décision et information sanitaire interactive à partir de l’Internet. Les applications dans le domaine de l’informatique et dans celui des télécommunications, renforçant les pratiques fondées sur l’expérience, obéissent à une hiérarchie dans laquelle les tâches des systèmes s’échelonnent par ordre de complexité croissante, depuis la recherche de référence et le traitement d’opérations relativement courantes jusqu’à des systèmes complexes d’aide à la décision basés sur l’exploration en profondeur des données et gouvernés par des règles. A l’analyse des résultats concernant : la façon dont les pratiques fondées sur l’expérience progressent, les méthodes de mise en œuvre les plus efficaces et les moyens permettant de surmonter les obstacles à l’utilisation étendue des directives fondées sur l’expérience, il apparaît qu’une recherche comparative concertée des meilleures méthodes et qu’une aide fondée sur la pratique donnent les meilleurs résultats, du moins dans certains environnements cliniques. Ce sont là des facteurs qui posent de gros problèmes en informatique médicale et qui plaident en faveur d’un investissement public important dans le développement de la technologie, par l’entremise d’un effort de recherche concerté associant l’industrie, les centres universitaires, les centres de recherche, les instances sanitaires et les pouvoirs publics.

Resumen

Sistemas de información: factores facilitadores de las prácticas de salud basadas en la evidencia

Se impone cada vez más la necesidad de utilizar la mejor evidencia disponible para adoptar decisiones en materia de práctica clínica, servicios de salud y gestión de programas. En este artículo se revisan diversas cuestiones relacionadas con el papel de la información en la práctica basada en la evidencia (PBE), que se caracteriza por la traducción de los problemas clínicos en preguntas y la localización, evaluación y empleo sistemáticos de fuentes de información validadas, investigaciones y revisiones críticas relacionadas con el problema como base para adoptar decisiones. Se analizan el desarrollo, la validación y el uso de una serie de fuentes de pruebas científicas y conocimientos que trascienden el afianzado paradigma basado en la sucesión de investigaciones, ensayos clínicos y revisiones sistemáticas de la literatura. La aportación de los sistemas de información y la tecnología (SIT) avanzados a la PBE se materializa en seis ámbitos de aplicación: bases de datos de referencia, información contextual y específica sobre casos, depósitos de datos clínicos, depósitos de datos administrativos, software decisional, e información sanitaria interactiva basada en Internet. Las aplicaciones computarizadas y de telecomunicaciones que sostienen la PBE se organizan en una jerarquía de tareas de distinta complejidad, desde los sistemas de recuperación de referencias y el procesamiento de transacciones relativamente rutinarias hasta los más sofisticados mecanismos de explotación de datos (data mining) y los sistemas de soporte de la adopción de decisiones basados en reglas. Analizando cómo está progresando la práctica basada en la evidencia, cuáles son los métodos de aplicación que funcionan mejor, y qué opciones pueden emplearse para superar los obstáculos al uso generalizado de directrices basadas en la evidencia, cabe deducir que la mejor alternativa, al menos en algunos contextos de práctica, es la adopción de criterios de comparación desarrollados en colaboración y de formas de asistencia basadas en la práctica. Estos factores constituyen importantes desafíos para la informática médica, y exigen inversiones públicas considerables en el desarrollo de tecnología mediante investigaciones en colaboración en las que deben participar la industria, centros universitarios, centros de investigación, el establishment médico y el Estado.

References

1. McMurray JJ. Failure to practise evidence-based medicine: why do physicians not treat patients with heart failure with angiotensin-converting enzyme inhibitors? European Heart Journal, 1998, 19 (Suppl L): 15–21.        

2. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. Journal of the American Medical Association, 1992, 268: 2420–2425.        

3. Sackett DL et al. Evidence-based medicine: what it is and what it isn’t. British Medical Journal, 1995, 312: 71–72.        

4. Berkvits M. From practice to research: the case for criticism in an age of evidence. Social Science and Medicine, 1998, 47: 1539–1545.        

5. Rosenberg W, Donald A. Evidence-based medicine: an approach to clinical problem-solving. British Medical Journal, 1995, 310: 1122–1126.        

6. Kenney AF, Hill JE, Mcrary CL. Introducing evidence-based medicine into a community family medicine residency. Journal of the Mississippi State Medical Association, 1998, 39: 441–443.        

7. Michaud GC et al. The introduction of evidence-based medicine as a component of daily practice. Bulletin of the Medical Library Association, 1996, 84: 478–481.        

8. Shin JH, Haynes RB, Johnston ME. Effect of problem-based, self directed undergraduate education on life-long learning. Canadian Medical Association Journal, 1993, 148: 969–976.        

9. Zvárova J. On the medical informatics structure. International Journal of Medical Informatics, 1997, 44: 75–81.        

10. McKibbon KA. Evidence-based practice. Bulletin of the Medical Library Association, 1998, 86: 396–401.        

11. Odunsi KO, Cooke IE, Olive DL. Evidence-based medicine: how do we find the evidence? Journal of the American Association of Gynecologic Laparoscopists, 1998, 5: 313–318.        

12. Lindberg DA, Humphreys BL. Medical informatics. Journal of the American Medical Association, 1997, 277: 1870– 1872.        

13. Marshall JG. The impact of hospital library on clinical decision making: the Rochester study. Bulletin of the Medical Library Association, 1992, 80: 169–178 .        

14. Johnston ME et al. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Annals of Internal Medicine, 1994, 120: 135–142.        

15. Haynes RB et al. Online access to MEDLINE in clinical settings. A study of use and usefulness. Annals of Internal Medicine, 1990, 112: 78–84.        

16. Jadad AR et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. Journal of the American Medical Association, 1998, 280: 278–280.        

17. Jadad AR et al. Systematc reviews and meta-analyses on the treatment of asthma: critical evaluation. British Medical Journal, 2000, 320: 537–540.        

18. Dickinson E. Clinical effectiveness for health care quality improvement. Journal of Quality in Clinical Practice, 1998, 18: 37–46.        

19. Eng TR, Gustafson DH (eds). Wired for health and well-being: the emergence of interactive health communication. Science Panel on Interactive Communication and Health, Office of Disease Prevention and Health Promotion, US Department of Health and Human Services. Washington, DC, US Printing Office, April 1999.        

20. Becker L. The Cochrane Collaboration. Valuable resource for family physicians. Canadian Family Physician, 1997, 43: 403–404, 412–414.        

21. Silagy C, Lancaster T. The Cochrane Collaboration in Primary Care: an international resource for evidence-based practice of family medicine. Family Medicine, 1995, 27: 302–305.        

22. Langton KB et al. A Clinical Informatics Network (CLINT) to support the practice of evidence-based health care. In: Proceedings of the AMIA Annual Fall Symposium, 1996: 428–432.        

23. Hayward RS et al. GAP: a computer-assisted design tool for the development and analysis of evidence-based automated questionnaires. Medinfo, 1995, Part 2: 934–937.        

24. Langton KB et al. A critical appraisal of the literature on the effects of computer-based clinical decision support systems on clinician performance and patient outcomes. In: Proceedings of the Annual Symposium on Computer Applications in Medical Care, 1992, 626–630.        

25. Nicoll LH. Don’t forget GratefulMed when conducting literature searches. ANNA Journal, 1992, 19: 521.        

26. Michaud G et al. Are therapeutic decisions supported by evidence from health care research? Archives of Internal Medicine, 1998, 158: 1665–1668.        

27. Enkin MW, Jadad AR. Using anecdotal information in evidence-based health care: heresy or necessity? Annals of Oncology, 1998, 9: 963–966.        

28. Berkvits M. From practice to research: the case for criticism in an age of evidence. Social Science and Medicine, 1998, 47: 1539–1545.        

29. Beyea SC, Nicoll LH. Dispelling the myth that research answers all questions. AORN Journal, 1998, 68: 1044–1047.        

30. Hippocrates (400 BC). Aphorisms; On ancient medicine; and The book of prognosis. Translated by Francis Adams. In The Internet Classics Archive, available online at http://classics.mit.edu//Hippocrates/aphorisms.html        

31. Beyea SC, Nicoll LH. When an adverse sentinel event is the cause for action. AORN Journal, 1999, 70: 703–704.        

32. Rodrigues RJ. Telemedicine and the transformation of healthcare practice in the information age. In: Speaker’s Book of the International Telecommunication Union Telecom Americas 2000 Telecom Development Symposium, Session TDS.2, Rio de Janeiro, April 10–15, 2000, 91–105.        

33. Silverman BG. The role of Web agents in medical knowledge management. MD Computing, 1998, 15: 1–10.        

34. Prescription sales, privacy fears: CVS, giant, share customer records with drug marketing firm. The Washington Post, 15 February 1998: A1.        

35. Moran DW. Health information policy: on preparing for the next war. Health Affairs, 1998, 17: 9–22.        

36. Robinson TN et al. An evidence-based approach to interactive health communication: a challenge to medicine in the information age. Journal of the American Medical Association, 1998, 280: 1264–1269.        

37. Health Summit Working Group. Criteria for assessing the quality of health information on the Internet — policy paper. Virginia, Health Information Technology Institute, Mitretek Systems, 1998. Available online at http://hitiweb.mitretek. org/hswg        

38. Fitzgibbons SM, Lee R. The health.net Industry: the convergence of healthcare and the Internet. Industry Report, 8 January 1999. Hambrecht & Quist Institutional Research, 1999.        

39. Kagolovsky Y et al. Towards improved information retrieval from medical sources. International Journal of Medical Informatics, 1998, 51: 181–195.        

40. Rodrigues RJ. Ethical and legal issues in interactive health communications: a call for international cooperation (Editorial). Journal of Medical Internet Research 2: 2000. Available online at http://www.symposion.com/jmir/index.htm        

41. Kendall DB, Levine SR. Pursuing the promise of an informationage health care system. Health Affairs, 1998, 17: 41– 43.        

42. Pan American Health Organization. Information systems and information technology in health – challenges and solutions for Latin America and the Caribbean. Washington, DC, Pan American Health Organizaion, 1998.        

43. Jadad AR et al. The Internet and evidence-based decision-making: a needed synergy for efficient knowledge management in health care. Canadian Medical Association Journal, 2000, 162: 362–365.        

44. Scott A, Currie N, Donaldson C. Evaluating innovation in general practice: a pragmatic framework using programme budgeting and marginal analysis. Family Practice, 1998, 15: 216–222.        

45. Grol R et al. Dissemination of guidelines: which sources do physicians use in order to be informed? International Journal of Quality Health Care, 1998, 10: 135–140.        

46. Moehr JR. Informatics in the service of health, a look to the future. Methods of Information in Medicine, 1998, 37: 165–170.        

47. Godlee F. Getting evidence into practice. Needs the right resources and the right organization. British Medical Journal, 1998, 317: 6.        

48. Scally G, Donaldson LJ. The NHS’s 50th anniversary. Clinical governance and the drive for quality improvement in the new NHS in England. British Medical Journal, 1998, 317: 61–65.        

49. Kaegi L. Health services researchers zero in on quality, policy, and practice in a changing health care system: report of the 15th Annual Meeting of the Association for Health Services Research. Joint Commission Journal on Quality Improvement, 1998, 24: 658–671.        

¹ Regional Adviser in Health Services Information Technology, Pan American Health Organization/World Health Organization Regional Office for the Americas, 525 23rd Street NW, Washington, DC 20037, USA.

Ref. No. 99-0229