ORIGINAL ARTICLES AND REVIEWS
Information retrieval for the Cochrane systematic reviews: the case of breast cancer surgery
Gaetana CognettiI; Laura GrossiII; Antonio LuconI; Renata SoliminiIII
IIstituto Nazionale Tumori Regina Elena, Rome, Italy
IIAUSL Romagna, UO Qualità Ricerca Organizzativa e Innovazione, Rimini, Italy
IIIDipartimento del Farmaco, Istituto Superiore di Sanità, Rome, Italy
ABSTRACT
Introduction. Systematic reviews are fundamental sources of knowledge on the state-of-the-art interventions for various clinical problems. One of the essential components in carrying out a systematic review is that of developing a comprehensive literature search.
Materials and methods. Three Cochrane systematic reviews published in 2012 were retrieved using the MeSH descriptor breast neoplasms/surgery, and analyzed with respect to the information sources used and the search strategies adopted. In March 2014, an update of one of the reviews retrieved was also considered in the study.
Results. The number of databases queried for each review ranged between three and seven. All the reviews reported the search strategies adopted, however some only partially. All the reviews explicitly claimed that the searches applied no language restriction although sources such as the free database Lilacs (in Spanish and Portuguese) was not consulted.
Conclusion. To improve the quality it is necessary to apply standards in carrying out systematic reviews (as laid down in the MECIR project). To meet these standards concerning literature searching, professional information retrieval specialist staff should be involved. The peer review committee in charge of evaluating the publication of a systematic review should also include specialists in information retrieval for assessing the quality of the literature search.
Key words: Cochrane systematic reviews, breast neoplasm surgery, information retrieval, quality of reporting, search strategies
Healthcare providers, consumers, researchers, and policy makers are inundated with unmanageable amounts of information, including evidence from healthcare research. It is unlikely that all will have the time, skills and resources to find, appraise and interpret this evidence and to incorporate it into healthcare decisions [1].
INTRODUCTION
Systematic reviews are considered to be the most authoritative and credible information sources, offering a synthetic overview and an updated state-of-the-art report of the effectiveness of interventions in various clinical problems. Due to the huge amount of information and the large number of clinical trials produced, physicians often possess neither the skills nor the time to search, select and critically evaluate the scientific evidence to be used for patient care.
The Cochrane systematic reviews, published online in the Cochrane Library, are recognized as the gold standard, since their production must follow specific and rigorous methods to avoid bias and random errors [2].
The Cochrane Handbook for Systematic Review of Interventions (latest version 5.1.0, March 2011) is intended to rigorously define the methodology to be used in producing a good systematic review [1].
The proper reporting of the literature search conducted for a systematic review is also required to ensure transparency and repeatability of the scientific study.
For this purpose PRISMA was developed. PRISMA is the acronym for Preferred Reporting Items for Systematic Reviews and Meta-Analyses, published in 2009. PRISMA is the latest and renamed version of the QUOROM statement, which was developed in 1996. PRISMA's mission statement focuses on ways in which authors can ensure the transparent and complete reporting of systematic reviews and meta-analyses [3, 4].
When developing a systematic review, it is necessary first and foremost to clearly formulate the clinical question which is to be considered the main topic. A method in defining the clinical question, before developing search strategies on the bibliographic databases, is PICO (P stands for Patient/Problem/Population, I for Intervention, C for Comparison and O for Outcome).
The second step is to produce a wide-ranging literature search in order to obtain all the relevant clinical studies on the topic [5].
Due to the complex task of literature searching, the Trial Search Coordinator (TSC) in the Cochrane groups is often a librarian or documentalist [6].
A poor information search can affect the results of a systematic review, potentially missing useful studies necessary towards the statement of evidence. For this reason, the sources of information used (such as: bibliographic databases of published articles, grey literature, conference proceedings, etc.) should be clearly and analytically described in the review together with the different types of research adopted (search strategies, hand-searching, references in articles, contacts with authors of the trials, etc).
Reviews require the extensive querying a large number of databases, finding a complex balance between a highly sensitive search (high recall), focused on the retrieval of all clinical studies regarding the topic (without worrying about the possible noise) and the specificity, that is one searches only for relevant literature without noise (high precision), running the risks of missing useful information [7]. A good information search for a systematic review entails the application of complex strategies using both descriptors (unique concepts from a controlled dictionary/thesaurus), and synonyms, including acronyms, abbreviations and variants (e.g. singular, plural), in order to identify articles not yet indexed or to avoid missing documents as a result the different interpretations given by the indexers [8].
Support may arise from standard search strategies, such as those provided by some Cochrane Groups on their Editorial Page.
The two most important international medical databases are PubMed/Medline (Medical Literature Analysis and Retrieval System Online) and Embase (Excerpta Medica Database). However, despite their justifiably eminent reputation, a comprehensive search for information cannot be merely limited to these two sources alone, since this would retrieve only part of the relevant literature.
Although these two archives are very rich in information, and should always be used when developing a systematic review, they may not report all the relevant clinical trials. For this reason, it is often necessary to integrate the research with other databases. Among these is the Cochrane Central Register of Controlled Trials (Central), which is the most complete register of clinical trials, being a bibliographic database that collects the studies found for producing the Cochrane systematic reviews. Each Cochrane Review Group (CRG) creates a Specialized Register containing bibliographic citations of clinical trials retrieved by searching different databases and also by hand-searching the literature on the topic covered by the group. In addition to articles from PubMed and Embase, Central is comprised of these Specialized Registers which also include articles deriving from other published and unpublished sources. The Cochrane Handbook recommends that both Central and the Specialized Register itself be searched separately and that Medline and Embase be searched in addition, in order to obtain all the retrievable studies. The reason is that there is a delay before records appear in Central. This can be several months for Medline records and one to two years for Embase records [1].
Databases that might also be used are: Scopus, Web of Science, Biosis and others offering a large coverage of biomedicine, and in some cases including the social sciences and humanities.
Also disciplinary databases (among others, PsycINFO, specializing in mental health and behavioral sciences, CINAHL the Cumulative Index to Nursing and Allied Health Literature), and national and local databases are key sources of information for some systematic reviews. For example, if one carries out a systematic literature search on acupuncture, Chinese archives (often in Chinese) cannot be neglected [9].
In addition to these sources, the Cochrane Handbook suggests also the use of web search engines such as Google Scholar.
Grey literature, not/not yet published, as abstracts or conference proceedings distributed to the participants or unpublished trials, is very important due to publication bias, i.e. the tendency to publish studies with positive results and not to publish studies with negative findings (e.g. showing side effects of drugs). A critical point concerns the cost of bibliographic databases and access to scientific information. What happens if the authors of a systematic review cannot financially afford to access the data and procure the documents? Most of the databases are not free of charge. For example, Embase is only available via an expensive subscription. Failing to access this source of information one which is recommended by the Cochrane Handbook can negatively impact the quantity and quality of evidence.
The quality of information retrieval for producing a systematic reviews is not always well assessed. The literature search in many cases is not described according to the MECIR standards and to the PRISMA statement.
The aim of this study is to analyze a small sample of Cochrane systematic reviews on the same or a similar topic from the point of view of literature searching, in order to highlight differences that could affect the results of the studies. This brief study could stimulate researchers to pay greater and particular attention to this fundamental aspect of the production of a systematic review. This also holds true for standard scientific articles, as some peer reviewers now ask that the literature search methodology also be reported for scientific articles regardless of whether or not they are systematic reviews. Verifying how different information search strategies impact the results of the systematic reviews is beyond the scope of this article. Our sole aim is to highlight the need to adopt a rigorous methodology so as not to miss potentially vital pieces of the clinical puzzle that a systematic review attempts to solve.
MATERIAL AND METHODS
Three Cochrane systematic reviews published in 2012 were retrieved using the Medical Subject Heading (MeSH) descriptor breast neoplasms combined with the subheading surgery (explosion applied). In March 2014, an update of one review was also considered in the study. The reporting of the information retrieval process was assessed with reference to the dates when searches were performed, the databases and documentation used (including the types of electronic platforms) and the search strategies adopted that should be unique to each different repository.
In particular, the following aspects were considered:
number and types of electronic databases searched;
search date and date range applied to the literature search;
search strategies;
language restrictions;
use of other sources (hand-searching of journals, conference proceedings, unpublished studies, grey literature, reference lists and direct contact with experts);
authoring of the literature search.
The rationale of the study, started in 2012, and the choice of the topic breast cancer surgery arose from the fact that, during a literature search on this subject using Cochrane Library, the authors immediately noticed that different methods had been used for the information retrieval in the selected reviews. Therefore, the aim of the study was to develop a case report analysis. A search was performed on 22 August 2013 to retrieve Cochrane systematic reviews, published in 2012 on breast cancer surgery.
We did not apply any free text terms, so as to retrieve a small, but relevantly-focused sample. We identified three systematic reviews:
1) Staley, et al. 2012, Postoperative tamoxifen for ductal carcinoma in situ [10];
2) Andreae, et al. 2012, Local anaesthetics and regional anaesthesia for preventing chronic pain after surgery [11];
3) Bunn, et al. 2012, Prophylactic antibiotics to prevent surgical site infection after breast cancer surgery [12].
In March 2014, the search was repeated. We only considered any updates of the systematic reviews present in the sample. Jones et al. has been included in the study being an updating of Bunn, et al. [13].
Undoubtedly the sample analyzed is not representative of all published Cochrane reviews. However, this contribution does underline the need to check carefully this important aspect during the production and evaluation of a systematic review.
RESULTS
All the reviews were produced by different Cochrane Review Groups: the first by the Cochrane Breast Cancer Group, the second by the Cochrane Anaesthetic Group, the third by the Cochrane Wounds Group (Table 1).
Electronic sources for the literature search
All the reviews specifically reported the names of the databases used. Between three and seven databases were used in the different reviews.
All reviews used the Cochrane Central Register of Controlled Trials (Central). Three used the Cochrane Group Specialised Registers (those of the Breast Cancer Group and of the Wounds Group) except Andreae 2012; Medline, Embase and Cinahl were used in all the reviews except Staley 2012. Staley 2012 used only Central, Cochrane Breast Cancer Group Specialised Register and the WHO Ictrp (International Clinical Trials Registry Platform). Jones 2014 added DARE (Database of Abstracts of Reviews of Effects) to the six databases searched in the previous version of the review (Bunn 2012).
Researchers also specified for each database the name of the host. Although not explicitly required in the latest versions of the Cochrane Handbook, it undoubtedly facilitates the reproducibility of the search strategy: Andreae 2012 used PubMed, and Bunn 2012 and Jones 2014 used Medline on the Ovid platform.
Reporting the search strategies
Andreae 2012, Bunn 2012 and Jones 2014 reported the date range for each search in every archive.
Staley 2012 specified only the date on which the searches were conducted without reporting the range. There is also some confusion regarding the date: in the abstract, the search date for all the databases is stated as 16 August 2011. In the article section Search methods for identification of studies the dates indicated are the following: Central 12 August 2011; Cochrane Breast Cancer Group Specialised Register date not specified; WHO Ictrp 12 August 2011.
All the four systematic reviews reported on the search strategies for the databases that were consulted, except for DARE in Jones 2014. The search strategies in all reviews have been placed in the appendix, as suggested by the Cochrane Handbook, to avoid interrupting the flow of the text. Only the third and fourth reviews located the Central search strategy in the section Search methods for identification of studies. Bunn 2012 reports the other strategies in the appendices only for the original version (2006) of the review; the search strategies have not been updated as recommended by the EBM methodology. Neither Bunn 2012 nor Jones in updating the systematic review carried out a search of the comments, errata, retractions to verify the validity of studies over time.
Also the number of records retrieved, and useful information to render the search more transparent, is only indicated in Jones 2014 (except for CINAHL). Overall, all of the search strategies were described in sufficient detail.
In Staley 2012, the Central search strategy did not explicitly indicate the use of MeSH Terms and field operators (ti, ab, etc). The other reviews applied descriptors and text words to all the databases provided by thesaurus, including also Central. The search strategies in Jones 2014 are much more complete than the 2012 version, as filters were used for retrieving randomized clinical trials that exclude animal experimentation. The number of records retrieved for each search line were added for Medline, Embase and CINAHL.
Language restriction and other sources consulted
All reviews report on the hand-searching used to complete the literature search, thus also retrieving the unpublished trials as recommended by the Cochrane Handbook. Bunn 2012 and Jones 2014 explicitly state the consultation of grey literature and discussion with experts.
Moreover all reviews explicitly claim that the searches have not applied any language restriction for retrieving trials, as can be seen from the search strategies adopted, although sources such as the database LILACS (Latin-American and Caribbean System on Health Sciences Information) containing medical literature in Spanish and Portuguese were not consulted. Although the inclusion of LILACS is under discussion, consulting it is cited by the Cochrane Handbook as being of importance. Not only is it an underused source of trials, but as there are no access charges, there are also no financial hindrances to its use.
Authoring of the literature search
The name of the person/s who actually conducted the search strategies is not explicitly stated except for Bunn 2012 and Jones 2014 where, under the section Contribution of authors in subsection Contribution of editorial base, it is indicated that Ruth Foxlee: designed the search strategy and edited the search methods section for the update.
DISCUSSION
In recent years a number of studies have highlighted problems regarding the bibliographic searching aspect of systematic reviews, with reference to the sources used (databases, registries, etc.) and to the search strategies adopted. In many cases the literature search is not appropriately applied and reported as recommended by the Cochrane Handbook [14, 15].
Although this study examines a very small sample, some interesting facts emerge: literature searches, even when conducted on a similar topic often use different sources: Staley 2012 did not use Embase, one of the three key databases recommended by the Cochrane Handbook. Possibly this was due to the high subscription cost. However, there do exist many cases in which even those databases that are available for consultation free of charge are not used, such as Lilacs, which could enrich the research of the scientific evidence by providing Latin American literature.
The search strategies were incomplete in Staley 2012: the use of descriptors (thesaurus MeSH) is not clearly indicated, and search filters and field operators are not explained. These shortcomings limit the reproducibility of the search.
The author of the literature search was indicated only in two reviews. Naming the author of the literature search is crucial for assigning responsibility of this part of a systematic review. Otherwise the systematic review can remain without any professional attribution in this important section.
CONCLUSION
Improving the quality of systematic reviews is a great challenge today even for the Cochrane Collaboration the most important producer of quality EBM documentation. In 2013 this organization introduced the Methodological Expectations of Cochrane Intervention Reviews (MECIR) program in order to ensure the application of standards in the conduct of the systematic reviews. These standards, in the section Searching for the study, are compliant with the PRISMA statement and confirm the methodology given by the Cochrane Handbook in the same section.
To achieve these standards the involvement of professional staff specialized in information retrieval is required. Furthermore, information specialists, librarians or documentalists, should be indicated as authors of the information search in the systematic reviews in order to assign, assert and attribute their professional responsibility in this process [16]. No one else possesses the necessary array of skills, experience and expertise to guarantee the quality of the literature search.
Even prestigious scientific journals publish systematic reviews but omit to fully and appropriately report how the literature search was carried out [15]. Peer reviewers of systematic reviews should include the presence of information professionals in order to adequately assess this aspect.
The librarian/trial search coordinator needs to be specialized in the methodology of information retrieval for producing systematic reviews and play a key role in this process [17].
As Beverley, et al. reported: Information professionals have evolved from simply acting as evidence locators and resource providers to being quality literature filterers, critical appraisers, educators, disseminators, and even change managers [18].
The role of the information specialist, working in close collaboration with other researchers of the group, is to analyse the clinical queries in order to identify and discern the optimal sources for retrieving the information needed for the research question under discussion, and to formulate complex search strategies for every individual database; the strategy must be constantly adjusted, rewritten, described and reported in the systematic review and, after a maximum of two years, updated. Updating a systematic review should include the formulation of new search strategies. Clearly this should not be a mere repetition of those strategies already executed in the past, as the databases and their terminological systems (e.g. thesauri) are subject to continuous developments involving the introduction of new and updated terms that will prove useful if the search results are to faithfully represent an effective, accurate and current reflection of the field of inquiry. It is also necessary, when updating, to retrieve errata, retractions and comments. These may subsequently affect the evaluation of the studies upon which the systematic review is based. The recently-created social tool, PubMed Commons, might prove very useful in this regard, as it stimulates and records the comments and critical feedback offered by the scientific community on the articles indexed by PubMed.
To quote the title of an interesting article published by the Journal of the Medical Library Association [6] Systematic reviews need systematic searchers.
Acknowledgements
We are grateful to William Russell-Edu of the European Institute of Oncology Milan, Italy for reviewing the paper.
Conflict of interest statement
There are no potential conflicts of interest or any financial or personal relationships with other people or organizations that could inappropriately bias conduct and findings of this study.
REFERENCES
1. Higgins JPT, Green S (Eds). Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration 2011. Available from: www.cochrane-handbook.org.
2. Cipriani A, Furukawa TA, Barbui C. What is a Cochrane review. Epidemiol Psychiatr Sci 2011;20(3):231-3. DOI: http://dx.doi.org/10.1017/S2045796011000436.
3. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009;151(4):264-9. W64. DOI:10.7326/0003-4819-151-4-200908180-00135.
4. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 2009;6(7):e1000100. DOI: 10.1371/journal.pmed.1000100.
5. Webster AC, Cross NB, Mitchell R, Craig JC. How to get the most from the medical literature: Searching the medical literature effectively. Nephrology 2010;15:12-9. DOI: 10.1111/j.1440-1797.2009.01263.x.
6. McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc 2005;93(1):74-80.
7. Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol 2009;62:1021-30. DOI: 10.1016/j.jclinepi.2008.09.018.
8. Relevo R. Effective search strategies for systematic reviews of medical tests. J Gen Intern Med 2012;27(Suppl. 1):S28-32. DOI: 10.1007/s11606-011-1873-8.
9. Lui S, Smith EJ, Terplan M. Heterogeneity in search strategies among Cochrane acupuncture reviews: is there room for improvement? Acupunct Med 2010;28:149-53. DOI: 10.1136/aim.2010.002444.
10. Staley H, McCallum I, Bruce J. Postoperative tamoxifen for ductal carcinoma in situ. Cochrane Database Syst Rev 2012;10:CD007847. DOI: 10.1002/14651858.CD007847.pub2.
11. Andreae MH, Andreae DA. Local anaesthetics and regional anaesthesia for preventing chronic pain after surgery. Cochrane Database Syst Rev 2012;10:CD007105. DOI: 10.1002/14651858.CD007105.pub2.
12. Bunn F, Jones DJ, Bell-Syer S. Prophylactic antibiotics to prevent surgical site infection after breast cancer surgery. Cochrane Database Syst Rev 2012;1:CD005360. DOI: 10.1002/14651858.CD005360.pub3.
13. Jones DJ, Bunn F, Bell-Syer SV. Prophylactic antibiotics to prevent surgical site infection after breast cancer surgery. Cochrane Database Syst Rev 2014;3:CD005360. DOI: 10.1002/14651858.CD005360.pub4.
14. Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol 2008;61(5):440-8. DOI: 10.1016/j.jclinepi.2007.06.005.
15. Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. J Med Libr Assoc 2009;97(1):21-9. DOI: 10.3163/1536-5050.97.1.004.
16. Tannery NH, Maggio LA. The role of medical librarians in medical education review articles. J Med Libr Assoc 2012;100(2):142-4. DOI: 10.3163/1536-5050.100.2.015.
17. McKibbon KA. Systematic reviews and librarians. Libr Trends 2006;55(1):202-15.
18. Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Info Libr J 2003;20(2):65-74. DOI: 10.1046/j.1471-1842.2003.00411.x.
Address for correspondence:
Antonio Lucon,
Istituto Nazionale Tumori Regina Elena,
Via Chianesi 53, 00144 Rome, Italy.
E-mail: antonio.lucon@gmail.com.
Received on 31 July 2014.
Accepted on 17 February 2015.