Abstract

Objective

To describe the biomedical journal characteristics that are associated with the retraction of papers.

Method

A descriptive cross-sectional study was conducted. All papers retracted and indexed in PubMed from January 1^st 2013 to December 31^st, 2016 were included. We used nine main categories to classify retractions: aspects related with data, authors issues, plagiarism, unethical research, journal issues, review process, conflict of interest, other, and unknown. These categories were further classified as: misconduct, suspicion of misconduct, or no misconduct.

Results

The proportion of retraction was 2.5 per 10,000 publications. Retractions appeared in 611 journals. During the study period, retraction due to misconduct was more frequent among journals with low-impact factor. Within these retracted publications, among low-impact journals the presence of misconduct was higher with a 73% compared to 61% for the high-impact journals (p=0.001). There were differences in the percentage of retractions due to misconduct regarding the journal classification category (p<0.001).

Conclusions

Retraction of publications is present in both high- and low-impact factor biomedical journals, but misconduct is more frequent among the papers retracted from lower impact journals. Measures before and after publication should be taken to limit misconduct.

Keywords:
Fraud; Impact factor; Peer review; Plagiarism; Plagiarism detection systems; Scientific misconduct

Resumen

Objetivo

Describir qué características de las revistas científicas biomédicas se asocian con la retractación de artículos.

Método

Se realizó un estudio transversal en el que se incluyeron todos los artículos retractados e indexados en PubMed del 1 de enero de 2013 al 31 de diciembre de 2016. Las retractaciones también se clasificaron de forma más detallada: problemas con los datos o manejo de datos; aspectos de autoría; plagio; investigación no ética; aspectos relacionados con la revista; problemas con el proceso de revisión; conflicto de intereses o razones desconocidas. Posteriormente se clasificaron como mala conducta, sospecha de mala conducta o sin mala conducta.

Resultados

La proporción de retractaciones fue de 2,5 por cada 10.000 publicaciones. Hubo retractaciones en 611 revistas. Durante el periodo de estudio, la retractación por mala conducta fue más frecuente en revistas con bajo factor de impacto, y entre los artículos retractados, hubo más mala conducta (73%) entre las revistas de bajo factor de impacto, en comparación con las revistas con alto factor de impacto (61%) (p=0,001). También se observan diferencias en el porcentaje de retractaciones debidas a mala conducta según la categoría de clasificación del Journal Citation Report (p <0,001).

Conclusiones

La retractación de publicaciones está presente tanto en revistas de alto como de bajo factor de impacto, pero la retractación por mala conducta es más habitual en revistas biomédicas de bajo impacto. Deben tomarse medidas antes y después de la publicación para limitar la mala conducta científica.

Palabras clave:
Fraude; Factor de impacto; Revisión por pares; Plagio; Sistemas de detección de plagio; Mala conducta científica

Introduction

Retraction of publications is an increasing problem in science and one of the most troublesome issues for biomedical journals.¹1. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci U S A. 2012;109:17028-33.

2. Haug CJ. Peer-review fraud - hacking the scientific publication process. N Engl J Med. 2015;373:2393-5.^-33. Steen RG. Retractions in the scientific literature: is the incidence of research fraud increasing? J Med Ethics. 2011;37:249-53. The impact of retraction is negative whether it is due to misconduct or not, and it is believed that fraud is more frequent than assumed.⁴4. Marcovitch H. Misconduct by researchers and authors. Gac Sanit. 2007;21:492-9.^-55. Fanelli D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS One. 2009;4:e5738. Retracted publications usually cause lack of trust in medical science and may affect the credibility of scientific journals, even in the case that some journals are proactive in retraction policies.

Detection of plagiarism has become a concern for editors, and measures such as the implementation of plagiarism detection software have been taken.⁶6. White C. Plagiarism detection service to be launched in June. BMJ. 2008;336:797.

7. Kleinert S. Checking for plagiarism, duplicate publication, and text recycling. Lancet. 2011;377:281-2.^-88. Chapman JR. Message from the editor-in-chief. Transplantation. 2015;99:1-2. The literature regarding the use of surveillance tools to detect plagiarism and its results is scarce. In fact we have not found any paper analyzing the use of such tools comparing biomedical journals' characteristics such as the impact factor. The use of these tools has increased plagiarism detection, but these detected manuscripts do not become public, as when detected during the editorial process authors are warned and therefore the manuscript is not formally retracted because it is directly rejected. Therefore, the increase in retractions due to plagiarism is a result of the manuscripts that have escaped from the journals detection system, which are now increasingly used by high-impact journals.⁹9. Masic I, Begic E, Dobraca A. Plagiarism detection by online solutions. Stud Health Technol Inform. 2017;238:227-30.

Literature focusing on misconduct has been centered more on authors instead of journal's characteristics¹1. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci U S A. 2012;109:17028-33.^,1010. Casadevall A, Steen RG, Fang FC. Sources of error in the retracted scientific literature. FASEB J. 2014;28:3847-55.

11. Foo JY. A retrospective analysis of the trend of retracted publications in the field of biomedical and life sciences. Sci Eng Ethics. 2011;17:459-68.

12. Foo JY, Tan XJ. Analysis and implications of retraction period and coauthorship of fraudulent publications. Account Res. 2014;21:198-210.

13. Woolley KL, Lew RA, Stretton S, et al. Lack of involvement of medical writers and the pharmaceutical industry in publications retracted for misconduct: a systematic, controlled, retrospective study. Curr Med Res Opin. 2011;27:1175-82.

14. Stretton S, Bramich NJ, Keys JR, et al. Publication misconduct and plagiarism retractions: a systematic, retrospective study. Curr Med Res Opin. 2012;28:1575-83.^-1515. Campos-Varela I, Ruano-Ravina A. Misconduct as the main cause for retraction. A descriptive study of retracted publications and their authors. Gac Sanit. 2018;pii S0213-9111:30072-4, http://dx.doi.org/10.1016/j.gaceta.2018.01.009. [Epub ahead of print].
https://doi.org/10.1016/j.gaceta.2018.01... involved in retraction or misconduct. Misconduct rates by journal type do not necessarily mean a weak peer-review policy by a specific journal but a high ethical standard or a higher surveillance or sensitivity on claims of what a particular journal has published. The surveillance of what is published is expected to be higher in a high impact journal. We aimed to describe the journal characteristics, mainly the impact factor and journal categories, involved in the retraction process to have a better picture of retraction frequency, particularly that related with misconduct. This paper is part of a previously published study,¹⁵15. Campos-Varela I, Ruano-Ravina A. Misconduct as the main cause for retraction. A descriptive study of retracted publications and their authors. Gac Sanit. 2018;pii S0213-9111:30072-4, http://dx.doi.org/10.1016/j.gaceta.2018.01.009. [Epub ahead of print].
https://doi.org/10.1016/j.gaceta.2018.01... and is exploiting additional information from the same database.

Method

Search strategy

A PubMed search was conducted looking for articles indexed by PubMed and labeled as retracted publications from January 1^st, 2013 to December 31^st, 2016. PubMed is the most used database in biomedical journals and practically all biomedical journals with an impact factor (IF) are indexed there. In fact, other research papers similar to the present one have previously chosen only PubMed as the source to evaluate retractions.¹1. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci U S A. 2012;109:17028-33.^,1010. Casadevall A, Steen RG, Fang FC. Sources of error in the retracted scientific literature. FASEB J. 2014;28:3847-55.^-1111. Foo JY. A retrospective analysis of the trend of retracted publications in the field of biomedical and life sciences. Sci Eng Ethics. 2011;17:459-68.^,1414. Stretton S, Bramich NJ, Keys JR, et al. Publication misconduct and plagiarism retractions: a systematic, retrospective study. Curr Med Res Opin. 2012;28:1575-83.^,1616. Fang FC, Casadevall A. Retracted science and the retraction index. Infect Immun. 2011;79:3855-9.

17. Fang FC, Bennett JW, Casadevall A. Males are overrepresented among life science researchers committing scientific misconduct. MBio. 2013;4:e00640-712.^-1818. Steen RG, Casadevall A, Fang FC. Why has the number of scientific retractions increased? PloS One. 2013;8:e68397. As previously described,¹⁵15. Campos-Varela I, Ruano-Ravina A. Misconduct as the main cause for retraction. A descriptive study of retracted publications and their authors. Gac Sanit. 2018;pii S0213-9111:30072-4, http://dx.doi.org/10.1016/j.gaceta.2018.01.009. [Epub ahead of print].
https://doi.org/10.1016/j.gaceta.2018.01... the search strategy was the following: (“retracted publication”[Publication Type] OR “retracted publication”[All Fields]) AND (“2013/01/01”[PDAT]: “2016/12/31”[PDAT]).

Data extraction

For each retracted article identified, title, author/s, journal name, date of publication and date of retraction were obtained. The time to retraction was calculated as time elapsed between both dates.

Web of Science was used to determine the Journal Citation Report (JCR) category, 5-year impact factor (5-IF) and the quartile in the category. Impact factor from year of publication or nearest year available was reflected. Journals without 5-IF were left blank. The subject category and, if documented, the contact information for the corresponding author's institution were recorded. For journals belonging to more than one category, the one corresponding to the highest quartile was selected. In the case of a tie, the first category in order of appearance was considered.

Definition of cause of retraction

Retracted publications were classified based on retraction notice narrative and on the categories proposed by the Committee of Publishing Ethics definitions and previous classifications,¹⁹19. COPE - Committee on Publication Ethics. Retraction guidelines. Available from: https://publicationethics.org/files/retraction guidelines 0.pdf
https://publicationethics.org/files/retr... and were the same as in a previous paper.¹⁵15. Campos-Varela I, Ruano-Ravina A. Misconduct as the main cause for retraction. A descriptive study of retracted publications and their authors. Gac Sanit. 2018;pii S0213-9111:30072-4, http://dx.doi.org/10.1016/j.gaceta.2018.01.009. [Epub ahead of print].
https://doi.org/10.1016/j.gaceta.2018.01... Nine main categories were used: aspects related with data, authors issues, plagiarism, unethical research, journal issues, review process, conflict of interest, other, and unknown.⁴4. Marcovitch H. Misconduct by researchers and authors. Gac Sanit. 2007;21:492-9.^,1919. COPE - Committee on Publication Ethics. Retraction guidelines. Available from: https://publicationethics.org/files/retraction guidelines 0.pdf
https://publicationethics.org/files/retr... These categories were further classified as misconduct (fraud), suspicion of misconduct and no misconduct suspicion. Reason for retraction was manually identified from the retraction note. If more than one reason was present, reason was assigned hierarchically, such that manuscripts were assigned to the first reason in the list. Data collection and classification were performed by ICV, and doubtful assignments were discussed with ARR.

Data preparation and analysis

The percentages of total retracted publications and misconduct per 10,000 PubMed registries over time were calculated by dividing the total number of retracted publications due to misconduct by the total number of registries in PubMed for the study period. Quartile category was categorized as high quartile (quartiles 1 and 2) and low quartile (quartiles 3 and 4).

Differences between categorical variables were assessed by Chi-square test or Fisher's exact test. Continuous variables were compared using Mann-Whitney test. Non-parametric trend test analysis of variance by ranks was used to test for differences over time. STATA/SE® v13 (College Station, Texas) was used for all statistical analyses.

Results

Ratio of retraction, journal quartile and JCR categories of retracted articles

For the study period, the total number of retracted publications was 1,082 out of a total of 4,384,945, representing a proportion of 2.5 per 10,000 publications. Six hundred and eleven different biomedical journals retracted papers. The distribution by quartile is shown in Table 1. Median 5-IF was 2.63 (interquartile range [IQR]: 1.74-4.24). Of the retracted papers, 51.4% were published in high quartile journals, 33.1% in low quartile journals, and 15.5% were published in journals without IF (p <0.001). Retracted papers were mainly of the biochemistry and molecular biology areas (9.9%), followed by oncology (7.1%) (p <0.001) (Table 2).

Annual evolution, median time to retraction and reasons for retraction

Despite no significant trend was detected (p=0.9), the crude number of retractions has increased over time, with 70 retractions (6.4%) in 2013 and 418 (38.6%) in 2016. Plagiarism has steadily increased with only 35 papers retracted for that reason in 2013 (9.9%) and 161 in 2016 (45.5%). Plagiarism was also the main cause of retraction in 2016.

Median time to retraction was 10.0 months (IQR: 5.0-17.0). Time to retraction was similar between articles published in high and low quartile journals (median time to retraction was of 10.0 months [IQR: 5.0-19.0] for the former and 10.0 months [IQR: 5.0-17.5] for the latter). There was no difference in time to retraction regarding the specific reason for retraction (p=0.2).

Presence of misconduct among retracted papers

Definitive misconduct was found in 707 (65.3%) of all retracted papers. Additionally, proportion of misconduct among retracted papers or reason for retraction did not change over time (p=0.2).

Within the retracted publications, among low-impact journals the presence of misconduct was higher with a 73% compared to 61% for the high-impact journals (p=0.001) (Table 3).

There was a significant trend to a longer time to retraction for papers retracted because of misconduct compared to those papers not retracted due to misconduct (11 months [IQR: 5.9-18.0] vs. 5.9 months [IQR: 2.0-14.0]; p <0.001).

There were differences in the percentage of retractions due to misconduct regarding the JCR category. 91% of retracted papers were due to misconduct for pathology journals and 48% for pharmacology and pharmacy journals, (p <0.001) (Table 4). It can be observed that there are important differences for the burden of misconduct among retracted papers between JCR categories.

Among the papers retracted because of misconduct, plagiarism was the main reason for retraction, independently of the journal IF (p <0.001). Data manipulation was the second cause of retraction among those papers retracted in high impact journals.

Of all retractions, only 16 (1.5%, all because of misconduct) clearly stated that the journal contacted authors and/or institutions in order to communicate the retraction and the misconduct.

Discussion

We present here a description of retractions broken down by journals' IF. We have found that, of the retracted manuscripts, 51.4% belong to high-impact journals and that it seems that misconduct among retracted papers has apparently a higher presence in low-quartile journals. The most frequent cause of misconduct is plagiarism, accounting for 38% of all retractions in 2016. Finally, only 1.5% of retractions were marked as having triggered an action by journals, a very low percentage. We do not have data on the percentage of journals contacting authors when there is a retraction and if there are differences on the frequency of this contact regarding the journal's IF. However, and despite no clear data were available, different implications for authors have been previously described following retractions.²⁰20. McCook A. Fallout for co-authors. Science. 2018;362:394-5. Similarly, some journals decide to penalize authors by preventing them from publishing in the journal for a period of time.⁸8. Chapman JR. Message from the editor-in-chief. Transplantation. 2015;99:1-2.^,2121. Kim SY. The detection of plagiarism. Korean J Fam Med. 2017;38:1.

Fang et al.¹1. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci U S A. 2012;109:17028-33. have shown in a study including papers retracted until 2012 that the mean time to retraction was 32.9 months, which is more than three times longer that the time observed in the present study. We might speculate that this finding served editors to speed up fraud investigation when there is a suspicion and, as a consequence, the time to retraction has been shortened. The same group has also shown that time to retraction because of fraud is longer than when retractions were due to other reasons. This result is in agreement with our finding of a longer time to retraction because of misconduct. This might be related to the difficulty to ensure the facts and to confirm the fraudulent conduct. Also, when retraction is due to reasons other than misconduct, authors might be involved in the process of retraction, and that seems to be a factor related with reduced time to retraction.

Journal IF also has an influence in retraction with a higher proportion of misconduct among retracted papers published in low-impact journals. Previously, the contrary has been shown, suggesting that the greater visibility of high-impact journals promotes higher inspection and a quicker retraction. We have also found that 51.4% of the retracted papers were published in high-quartile journals. It is noteworthy that in high-impact journals, a significant proportion of the misconduct is related to data manipulation by authors, suggesting that high IF might be more attractive as these publications are generally linked to improvements in CVs, grant success, and better job opportunities. This might provide some incentive to commit misconduct to publish in a high-ranking journal even though IF and quantitative measurement of publications has been largely questioned.¹⁶16. Fang FC, Casadevall A. Retracted science and the retraction index. Infect Immun. 2011;79:3855-9.^,2222. Angell M. Publish or perish: a proposal. Ann Intern Med. 1986;104:261-2.^-2323. Hansson S. Impact factor as a misleading tool in evaluation of medical journals. Lancet. 1995;346:906.

Plagiarism accounts for the highest number of retractions for the study period. These data are striking and deserve to be commented given that we have reasons to think that the real number of authors plagiarizing is higher. The use of plagiarism detection software has become more widespread. Despite the fact that the use of these systems prevents having plagiarized published manuscripts, retractions due to plagiarism are increasing by year. It is worth to mention that plagiarism was the main reason for retraction independently of the journal impact, indicating that the implementation of plagiarism detection systems are in great need, independently of the journal. However, these systems have probably prevented finding a significant increase. Hence, and despite awareness of editors, reviewers and readers who undoubtedly are contributing to this increase in detection and retraction, more action has to be done.⁸8. Chapman JR. Message from the editor-in-chief. Transplantation. 2015;99:1-2.^,2424. Mavrogenis AF, Panagopoulos GN, Megaloikonomos PD, et al. Scientific misconduct (fraud) in medical writing. Orthopedics. 2018;41:e176-83.^-2525. Higgins JR, Lin FC, Evans JP. Plagiarism in submitted manuscripts: incidence, characteristics and optimization of screening-case study in a major specialty medical journal. Res Integr Peer Rev. 2016;1:13. However, efforts to limit this phenomenon have to start from the very moment when a manuscript arrives to a medical journal and is assigned to an editor. Detection of plagiarism can be done in three ways. First, detection can rely on the capacity of reviewers and editors to identify previously published papers. Second, it can use plagiarism detection software including that for image fraud detection.²⁴24. Mavrogenis AF, Panagopoulos GN, Megaloikonomos PD, et al. Scientific misconduct (fraud) in medical writing. Orthopedics. 2018;41:e176-83.^-2525. Higgins JR, Lin FC, Evans JP. Plagiarism in submitted manuscripts: incidence, characteristics and optimization of screening-case study in a major specialty medical journal. Res Integr Peer Rev. 2016;1:13. And finally, detection can occur when a reader or the author is capable of identifying the plagiarism from an already published paper. This method seems to be the only way to identify plagiarism when the original publication is in a language other than English and is not published in a widely visible journal.²⁶26. Baydik OD, Gasparyan AY. How to act when research misconduct is not detected by software but revealed by the author of the plagiarized article. J Korean Med Sci. 2016;31:1508-10. Actions other than warning authors are needed, such as those in based in education. It has already been tested that warnings against plagiarism are insufficient.²⁷27. Marshall T, Taylor B, Hothersall E, et al. Plagiarism: a case study of quality improvement in a taught postgraduate programme. Med Teach. 2011;33:e375-81.^-2828. Braumoeller B, Gaines, B. Actions do speak louder than words: deterring plagiarism with the use of plagiarism-detection software. Polit Sci Polit. 2001;34:835-9. There is also a lack of standardization and clarity on retraction management, and databases should provide tools and procedures to trace the retractions and authors committing misconduct.²⁹29. Ruano-Ravina A, Campos-Varela I. Some proposals on tracking scientific misconduct. Gac Sanit. 2019;33:92-3.

It is worth to mention, that the entire editorial process is not paid, nor valued, and that in general there is not any specific training for the work. Also, the editorial process is based on good faith and goodwill. Consequently, to ask Editors and reviewers an excessive amount of work and to be fully aware of new ways of committing fraud might not be realistic. Some tasks using plagiarism detection using commercial software should be done by biomedical journals' administrative staff before allowing the manuscript to enter in the editorial process.

Biochemistry and molecular biology and pathology journals are, respectively, the JCR categories with a greater number of retractions, and with the highest proportion of misconduct among retracted papers, though we do not know the total number of published papers within those categories. Despite retractions and misconduct are widespread, this basic discipline accounts for the greater proportion of retractions. Before, non-association was found between a general medicine journal (vs. other) and misconduct.¹³13. Woolley KL, Lew RA, Stretton S, et al. Lack of involvement of medical writers and the pharmaceutical industry in publications retracted for misconduct: a systematic, controlled, retrospective study. Curr Med Res Opin. 2011;27:1175-82. In general terms, we believe that the threat of retraction and misconduct is everywhere, and all journals have to be aware of it and implement measures to limit it.

The main limitation of our study results from the fact that we are only evaluating retractions that have been identified after being published, and we do not address all the plagiarism detected by the journals including any other fraud avoided by early identification during the editorial process affecting rejected papers. Also, as has been previously acknowledged,¹⁵15. Campos-Varela I, Ruano-Ravina A. Misconduct as the main cause for retraction. A descriptive study of retracted publications and their authors. Gac Sanit. 2018;pii S0213-9111:30072-4, http://dx.doi.org/10.1016/j.gaceta.2018.01.009. [Epub ahead of print].
https://doi.org/10.1016/j.gaceta.2018.01... the ability to classify publications as misconduct or not is limited and some data are not accessible, such as quartile of all the publications of the study period. Also, we cannot evaluate the percentage of journals contacting authors and/or institutions when there is a retraction, and if this is related to the journal's IF. The dynamic nature of data obliges us to be cautious. An additional limitation is that PubMed was the only database consulted to gather the reported data, though we need to take into account the fact that, for medical literature, Medline is the main database, and where the vast majority of journals are indexed. PubMed is also the database where a reader would notice if an article has been retracted. Finally, and perhaps most important, we do not know the total number of papers published in the analysed journals and therefore we cannot infer that the incidence of retractions is higher in low-impact biomedical journals compared to high impact journals. We only can conclude that the crude number of retractions (and also retractions due to misconduct) is higher in low impact journals

To conclude, retraction of scientific publications is associated with both high and low-impact journals, with plagiarism as the main reason of misconduct, independently of the journal IF. It would be recommendable for journals to implement the use of plagiarism detection systems and to standardize measures both before and after the publication of the manuscript. We encourage journal editors detecting misconduct to promote actions beyond those limited to the journal.

References

Publication Dates

History