Impacts of Social Media Sentiments on Retractions of Scholarly Literature

Autumn Toney-Wails^* and Sara Abdulla^**

^*autumn.toney@georgetown.edu

Center for Security and Emerging Technology, Georgetown University, United States

^** sa1764@ georgetown.edu

Center for Security and Emerging Technology, Georgetown University, United States

We explore the citation activity and social media engagement of retracted and non-retracted scientific research publications. While prior research has mainly studied retraction trends among specific areas of research, author countries, and publication venues, we focus on Twitter activity differences between retracted and non-retracted publications across all of science. We analyze over 62,000 research publications and 60,000 tweets that contain links to publications in their posts. Our findings highlight that citations and tweet activity are not correlated (i.e., high academic impact does not imply high social media impact), and that temporal trends of publication counts and tweet counts differ. Comparing tweet text from retracted and non-retracted publications, we train a random forest classifier that achieves 0.92 accuracy in predicting if a tweet references a retracted or non-retracted publication.

1. Introduction

From 2009 to 2011 more than 7,300 scientific research publications that were issued in a variety of the Institute of Electrical and Electronics Engineers (IEEE) conferences were retracted (McCook, 2018). The dramatic volume in publication retractions prompted analysis of retracted publications’ trends and characteristics, such as retractions in specific fields and citation behavior between retracted and non-retracted research (King et al., 2018; Peng, Romero, and Horvat, 2022; Rapani et al., 2020; Soltani and Patini, 2020; Chen et al., 2013; Fang, Steen, and Casadevall, 2012; He, 2013; Kuroki and Ukawa, 2018; Q. H. Vuong et al., 2020). However, with the increase in information sharing on social media platforms, there is an aspect to retracted research that has not been fully explored: how is retracted research discussed on social media?

Citation analysis of retracted research is useful in understanding how the scientific research community, specifically, responds to the retracted publications, but social media activity has become another metric for scientific and academic impact for scholarly publications. Social media activity is a particularly useful measure of reach for its immediate availability, as opposed to the inevitable lag that occurs with relying on peer citation metrics. For example, a publication that might be retracted in the future can receive public commentary via online platforms, such as Twitter, whereas it might take years for the article to be retracted. Figure 1 illustrates this example with a Twitter post about a scientific publication that was eventually retracted, with the tweet gaining significantly more attention than the initial publication (using citation count).

Figure 1. Example tweet referencing a retracted publication (the publication was not retracted at the time of the tweet)

Social media is also useful for assessing reach in relevant non-academic audiences, including journalists, non-academic subject matter experts, and general subject stakeholders. It is important to examine the social media landscapes surrounding retracted publications and non-retracted publications, including if social media users discuss retracted publications differently from non-retracted publications or if retracted publications leave larger digital footprints than non-retracted publications.

This work explores Twitter activity from posts that reference retracted research publications and compare it to tweets that reference non-retracted publications. We analyze the trends and characteristics of retracted and non-retracted publications’ citations and compare the citation statistics to tweet statistics. Using the Retraction Watch Database (RWD)(The Center For Scientific Integrity, 2018), containing over 32,000 retracted publications, and a sample of non-retracted publications from Digital Science’s Dimensions (Hook, Porter, and Herzog, 2018), we query the Altmetrics¹API (Altmetric.com 2012) to retrieve Twitter data for 15,822 research publications that are hyperlinked in at least one tweet. We aim to answer three main research questions:

• RQ1: How do qualities of retracted research—including post-retraction citation and social media engagement on Twitter—differ across areas of research?

• RQ2: Is there a relationship between the citations and the twitter post activity for retracted and non-retracted research publications?

• RQ3: Are tweets that reference retracted publications distinguishable from tweets that reference non-retracted publications in the language that they use or their popularity?

2. Methodology

2.1. Datasets

For our analysis, we source three datasets: Retraction Watch Database (RWD), Digital Science, and Twitter. The publicly available RWD contains 32,230² manually labeled retracted scientific publications with relevant metadata (e.g., original publication DOI, retracted DOI, subject, and retraction reason) (The Center for Scientific Integrity 2018). For our analysis, we remove retracted publications that do not have an original article DOI listed, resulting in 24,828 retracted publications. Our general scholarly literature dataset is Digital Science's Dimensions, which we use to generate a stratified sample of 37,137 non-retracted publications, controlling for publication year and country (Hook, Porter, and Herzog 2018).

Using the original publication DOI, we query the Altmetrics API for social media interaction data, which provides the tweet IDs for tweets that post a hyperlink to a publication that is matched by publication the DOI. We generate a dataset of tweet IDs that mention retracted publications (34,192 tweet IDs) and non-retracted publications from our sample (26,292 tweet IDs).

2.2. Citation Analysis

We compute citation percentiles, by research area and publication year, to rank publications by their citation counts, accounting for differences research areas and publication year. There are a total of 19 broad areas of research we use for this grouping: art, biology, business, chemistry, computer science, economics, engineering, environmental science, geology, geography, history, materials science, mathematics, medicine, philosophy, physics, political science, psychology, and sociology. We compare the distribution of citation percentiles for retracted papers versus non-retracted papers using the Kolmogorov–Smirnov test. Additionally, we analyze the citations that retracted publications receive post-retraction.

2.3. Social Media Interaction and Impact

We analyze tweet activity surrounding publications pre- and post-retraction by computing summary statistics, such as how many total tweets were posted before and after a publication was retracted and how many publications have more tweets post retraction. To assess if the tweets that reference retracted publications are distinguishable from tweets that reference non-retracted publications, we implement three classification models: decision tree, logistic regression, and random forest. We train each model using TF-IDF vectors of normalized tweet text from each publication type (variations of the word retracted are removed).

We investigate the difference in sentiment scores between tweets that reference retracted publications versus non-retracted publications. We use the Valence Aware Dictionary for sEntiment Reasoning (VADER), designed for sentiment analysis on social media text (Hutto and Gilbert, 2014). VADER produces a sentiment score ranging from -1 (most negative) and +1 (most positive). We apply categorical labels in our analysis for the following score bins: Negative: [-1, -0.2), Neutral: [-0.2,0.2], Positive: (0.2, 1].

3. Results

We first analyze the characteristics of the retracted publications from the RWD. Figure 2 displays the number of publications over a 20-year period (2001–2021) by main area of research. We compare the counts over time by the year of publication (using the original publication DOI) and the counts over time by the year of retraction (using the retraction DOI). There is a notable spike in publication retractions between 2009 and 2011 that can be largely attributed to a rapid swath of retractions of IEEE conference papers (McCook, 2018). McCook (2018) explains that after increased scrutiny into the retractions an IEEE spokesperson stated that these publications did not meet the publisher’s guidelines, with no other detailed explanations for the retractions, and this is considered to be a unique event.

Figure 2. Publication counts over time (2001-2021) by main area of research for the publication year (top) and the retraction year (bottom).

Figure 3 shows the average number of years that it took publications to be retracted). This metric is relevant when considering citation activity (longer duration of time between publication and retraction may partially explain high citation activity), or patterns in retraction. Between 2009 to 2014, the average number of years to retraction was consistently four, which then dropped to between two and three years by 2018. Papers with more recent publication years will have a downward year-to-retraction bias, as a short amount of time has passed since publication (e.g., there are only 4 years for a paper published in 2018 to be retracted in this analysis).

Figure 3. Average number of years for a publication to be retracted by publication year, 2001-2021.

We also provide the top 10 retraction reasons, publication countries, and subjects in Tables 2–4. Retraction reasons, countries, and subjects are not mutually exclusive, thus counts in Tables 2-4 represent the number of times those values appear, and do not reflect unique publication counts.

Table 2. Publication counts for the top 10 retraction reasons.

Retraction Reason	Count
Limited or No Information	6,035
Investigation by Journal/Publisher	5,359
Withdrawal	2,514
Breach of Policy by Author	2,482
Issues about Data	2,102
Investigation by Company/Institution	2,051
Duplication of Article	1,980
Duplication of Image	1,891
Unreliable Results	1,779
Date of Retraction/Other Unknown	1,709

Table 3. Publication counts for the top 10 publishing countries.

Country	Count
China	11,319
United States	3,850
India	1,319
United Kingdom	1,085
Japan	1,056
Iran	847
Germany	795
South Korea	580
Italy	568
Canada	493

Table 4. Publication counts for the top 10 subjects.

Subject	Count
Biology - Cellular (BLS)	5,214
Biochemistry (BLS)	3,493
Genetics (BLS)	3,216
Biology - Molecular (BLS)	2,604
Biology - Cancer (BLS)	2,563
Technology (B/T)	1,952
Computer Science (B/T)	1,743
Medicine - Oncology (HSC)	1,563
Chemistry (PHY)	1,508
Medicine - Pharmacology (HSC)	1,181

Table 2 shows that most commonly the notice of retraction provided limited or no information about the reason for retraction. The second and third most popular retraction reasons also provide minimal descriptive insight, but mention investigation by publisher and withdrawal. Table 3 shows that publications with authors that are affiliated with Chinese organizations have a significantly outnumber the remaining countries, and that smaller countries with lesser research output make the top 10 country list, such as Iran and South Korea. Table 4 highlights that basic life science areas are the most prevalent subject areas, with business and technology and health sciences appearing in two out of the 10 subjects, respectively.

4.1. Citation Analysis

Before our comparative analyses, we first evaluate post-retraction citation activity for publications that have a retraction DOI in the RWD. Table 5 contains the average number and percentage of post-retraction citation counts for retracted publications by subject area. Across all subjects except for the Basic Life Sciences, more than half of citations occur post-retractions for retracted papers. For papers with data on publication and retraction years and citations available, the average citation counts range from 17.83 (Environmental Sciences papers) to 40.90 (Basic Life Science papers).

Table 5. Average percentage of post-retraction citations by subject (sorted by mean citation count).

Subject	Mean Citations	% Citations	Number of Papers
BLS	40.90	48%	5,346
HSC	36.0	53%	4,154
PHY	26.43	58%	2,367
SOC	26.04	59%	684
HUM	23.54	73%	121
B/T	19.80	60%	1,122
ENV	17.83	67%	712

We investigate the differences in citation behavior between retracted publications and non-retracted publications. With the computed citation percentiles, we compare the distribution of percentiles between retracted and non-retracted publications. Figure 4 shows the empirical cumulative distributions of citation percentiles for the two publication sets. The Kolmogorov-Smirnov test determines that retracted and non-retracted papers have different citation-percentile distributions with a statistic of 0.10 and a p-value of ≈ 0.

Figure 4. The empirical cumulative distributions of the citation percentiles for retracted and non-retracted publications.

4.2. Twitter Analysis

We source tweet IDs linked to scientific publications using Altmetrics, which does not guarantee data or tweets for every DOI queried. For non-retracted papers, we retrieved data for 10,091 DOIs, with 6,528 DOIs having at least one tweet ID linked. For retracted papers we retrieved data for 10,788 DOIs, with 9,294 DOIS having at least one tweet ID. We generate a set of 26,292 tweets that reference non-retracted publications and a set of 34,192 tweets that reference retracted publications.

We count the number of tweets that link to retracted publications by their reference publication’s year and main area of research. Figure 5 displays the counts from 2011 to 2021; Twitter was created in 2006, thus for visual clarity we display the data over a 10-year period. Tweets referencing publications in the health sciences and biology maintain the highest frequencies over time, with health science-related tweets having notable spikes in 2018 and 2020. In 2018 the top three subjects are Immunology, Infectious Disease, and Obstetrics/Gynecology and in 2020 the top three subjects are Infectious Disease, Pharmacology, and Nutrition.

Figure 5. The number of tweets by the subject area and publication year of the retracted publication they reference.

We analyze the tweet activity pre-retraction date and post-retraction date. Analyzing publications that have complete data for retraction date, tweets linked, and tweets’ posted date we find that 28% (1,465 total) of retracted publications have more tweets posted after the publication was retracted. In total, there are 23,658 tweets posted pre-retraction date and 7,648 tweets posted post-retraction date.

The Twitter API provides tweet favorite and retweet counts for each tweet ID, which we summarize in Table 7. For both tweet sets, favorites have a higher maximum, with tweets that link to retracted publications having a higher maximum for both favorites and retweets compared to tweets that link to non-retracted publications.

Table 7. Summary statistics for tweets by publication type.

Publication Type		Max	Mean	Std.
Non-Retracted	retweets	72	6.5	11.3
	favorites	519	1.4	7.4
Retracted	retweets	715	44.9	145
	favorites	2,020	1.2	17.6

Table 8 displays the Spearman’s rank correlations which compares the citation percentiles and number of tweets for retracted and non-retracted publications. For both sets of publications, there is no significant relationship between the number of citations that a publication receives and the number of tweets that reference it.

Table 8. Spearman's rank correlation coefficient (ρ) and p-value

for number of tweets and citation percentiles by publication type.

Publication Type	ρ	p-value
Non-Retracted	0.23	≈0
Retracted	0.19	≈0

Table 9. Non-retracted/Retracted classification model performances. The best results are bold.

Model	Accuracy	Recall	FPR
Decision Tree	0.86	0.87	0.19
Logistic Regression	0.88	0.92	0.16
Random Forest	0.92	0.96	0.13

For tweets referencing publications, we train, test, and validate three classification models: decision tree, logistic regression, and random forest. Table 9 contains the model performance statistics, displaying the accuracy, recall, and false positive rates. The random forest classifier performed the best and we further investigate the model’s outputs to identify the distinguishing words/features. The random forest identified words such as vaccine, vitamin, hpv, covid, review, and cancer as important features.

We implement VADER to compute tweet sentiment scores for each publication type. Using the categorical sentiment bins (Section 3), Figure 7 displays the percentage of tweets that fall into the negative, positive, and neutral categories by reference publication type; the majority of tweets have neutral sentiment. While both publication sets have average sentiment scores that are considerably low, indicating neutral language, non-retracted publications have more positive language. The 75th percentile of sentiment scores is 0.32 and 0.08 for tweets that reference non-retracted publications and retracted publications, respectively.

Figure 7. The percentages of tweet sentiment category by reference publication type.

To further investigate sentiment scores, we manually review the top 100 most positive and most negative tweets by publication type. Tweets that reference non-retracted publications have a general categorical trend; positive tweets are congratulatory and negative tweets contain words with strong negative connotations (e.g., sexual assault and cancer). However, tweets that reference retracted articles are slightly different. Positive tweets contain sarcasm (“You should rush right out and get vaccinated. They’re SO trustworthy and love you so much. [link to publication] ”) and negative tweets discuss retraction reasons (“Looks like the paper was officially retracted, which is great. But it shouldn’t have taken the death of a poor grad student for such bad practices to have been detected and rooted out. [link to publication]”).

5. Discussion

Here we revisit our three main research questions to summarize our findings.

In general, Basic Life Science and Health Science publications lead summary statistics (number of retractions, mean citations post-retraction, and number of tweets), addressing RQ1. More specifically, research relating to Cellular Biology, Biochemistry, Genetics, Molecular Biology, and Cancer research are the leading subject areas for retracted publications. The temporal trends in retracted publication counts and number of tweets by subject area differ. Where publication count trends reflect the number of retractions by year (e.g., the IEEE retraction spike), tweet counts reflect events or social concerns (e.g., the COVID-19 pandemic in 2020).

For RQ2, we do not find a strong relationship between the number of citations that a publication receives and the number of tweets that reference it. Both retracted and non-retracted publications have insignificant Spearman’s rank correlations between citation and tweet counts. This indicates that academic impact and social media impact of scholarly literature are not directly related, but that social media impact can signal controversial research for its topic, quality, or cultural relevance.

We achieve 92% accuracy using a random forest classifier to predict if a tweet references a retracted or non-retracted publication. Additionally, we compare the sentiment of tweets referencing both publication types and find variation in positive and negative language usage. These results answer RQ3 in regards to language. Table 7’s results answer RQ3 in regards to tweet activity—tweets that reference retracted publications have significantly higher engagement via favorites and retweets.

6. Conclusion

Our analysis finds, that across subjects, publications continue to be cited post-retraction and in fact, most citations occur post-retraction. Within the context of prior research, which has shown that the majority of post-retraction citations are favorable or non-negative, our paper buttresses other literature that questions the impact of publication retraction on curbing academic impact. Using statistical learning methods and sentiment analysis, we find that tweet text contains indicators of the retraction likelihood or status of a publication.

As evidenced by high-profile cases of problematic science and research that demonstrate that these studies may impact practice, such as medicine and healthcare, including biomedical trials, the stakes for non-credible research can be high. When cited and propagated, faulty scientific studies could be poison in the proverbial well across all academic disciplines—not only life and health sciences— for years post-retraction, as evidenced by our analysis and others. Future analyses may aim to more comprehensively use altmetrics, citation, full-text, and subject data to develop warning signals for problematic science.

Open science practices

Several data sources used in this research are not open source, however in our final version we will link to our github which will contain the tweet IDs used in our analysis. We are contractually unable to provide data on the non-retracted publications that we used, but the Retraction Watch Database is publicly available and can be linked to other open source publication databases (using DOI). Our github will also provide the code used to produce our results.

Author contributions

Both authors contributed equally to this research.

Competing interests

The authors have no competing interests.

References

Altmetric.com [Accessed: 2022-01-25]. (2012).

Avenell, A., Stewart, F., Grey, A., Gamble, G., & Bolland, M. (2019). An investigation into the impact and implications of published papers from retracted research: Systematic search of affected literature. BMJ Open, 9(10). https://doi.org/10.1136/bmjopen-2019-031909

Bar-Ilan, J., & Halevi, G. (2017). Post retraction citations in context: A case study. Scientometrics, 113(1), 547–565. https://doi.org/10.1007/s11192-017-2242-0

Brainard, J. (2018). Rethinking retractions. Science, 362(6413), 390–393. https://doi.org/10.1126/ science.362.6413.390

Chen, C., Hu, Z., Milbank, J., & Schultz, T. (2013). A visual analytic study of retracted articles in scientific literature. Journal of the American Society for Information Science and Technology, 64(2), 234–253.

Copiello, S. (2020). Other than detecting impact in advance, alternative metrics could act as early warning signs of retractions: Tentative findings of a study into the papers retracted by PLoS ONE. Scientometrics, 125(3), 2449–2469. https://doi.org/10.1007/s11192-020-03698-w

Fabiano, N., Hallgrimson, Z., Kazi, S., Salameh, J.-P., Wong, S., Kazi, A., Unni, R. R., Prager, R., & McInnes, M. D. (2020). An analysis of covid-19 article dissemination by twitter compared to citation rates. https://doi.org/10.1101/2020.06.22.20137505

Fang, F. C., Steen, R. G., & Casadevall, A. (2012). Misconduct accounts for the majority of retracted scientific publications. Proceedings of the National Academy of Sciences, 109(42), 17028–17033.

He, T. (2013). Retraction of global scientific publications from 2001 to 2010. Scientometrics, 96(2), 555–561.

Hook, D. W., Porter, S. J., & Herzog, C. (2018). Dimensions: Building context for search and evaluation. Frontiers in Research Metrics and Analytics, 3, 23.

Hsiao, T.-K., & Schneider, J. (2021). Continued use of retracted papers: Temporal trends in citations and (lack of) awareness of retractions shown in citation contexts in biomedicine. Quantitative Science Studies, 2(4), 1144–1169. https://doi.org/10.1162/qss a 00155

Hutto, C., & Gilbert, E. (2014). Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the international AAAI conference on web and social media, 8(1), 216–225.

King, E. G., Oransky, I., Sachs, T. E., Farber, A., Flynn, D. B., Abritis, A., Kalish, J. A., & Siracuse, J. J. (2018). Analysis of retracted articles in the surgical literature. The American Journal of Surgery, 216(5), 851–855.

Kuroki, T., & Ukawa, A. (2018). Repeating probability of authors with retracted scientific publications. Accountability in research, 25(4), 212–219.

McCook, A. (2018). One publisher, more than 7000 retractions. Science, 362(6413), 393–393. https://doi.org/10.1126/science.362.6413.393

Omer, S. B. (2020). The discredited doctor hailed by the anti-vaccine movement. Nature, 586(7831),

668–669. https://doi.org/10.1038/d41586-020-02989-9

Peng, H., Romero, D. M., & Horvat, E.-´ A. (2022). Dynamics of cross-platform attention to retracted´ papers. Proceedings of the National Academy of Sciences, 119(25), e2119086119.

Rapani, A., Lombardi, T., Berton, F., Del Lupo, V., Di Lenarda, R., & Stacchi, C. (2020). Retracted publications and their citation in dental literature: A systematic review. Clinical and experimental dental research, 6(4), 383–390.

Soltani, P., & Patini, R. (2020). Retracted covid-19 articles: A side-effect of the hot race to publication. Scientometrics, 125(1), 819–822.

Summers, E., Brigadir, I., Hames, S., van Kemenade, H., Binkley, P., tinafigueroa, Ruest, N., Walmir, Chudnov, D., recrm, celeste, Lin, H., Chosak, A., McCain, R. M., Milligan, I., Segerberg, A., Shahrokhian, D., Walsh, M., Lausen, L., ... Shawn. (2022). Docnow/twarc: V2.10.4 (Version v2.10.4) [https://doi.org/10.5281/zenodo.6503180]. Zenodo. https://doi.org/10.5281/ zenodo.6503180

The Center For Scientific Integrity. (2018). The Retraction Watch Database [ISSN: 2692-465X. Accessed: 2022-01-25].

van der Vet, P. E., & Nijveen, H. (2016). Propagation of errors in citation networks: A study involving the entire citation network of a widely cited paper published in, and later retracted from, the journal nature. Research Integrity and Peer Review, 1(1). https://doi.org/10.1186/s41073-0160008-5

Vuong, Q. H., La, V.-P., Ho, M. T., Vuong, T.-T., & Ho, M.-T. (2020). Characteristics of retracted articles based on retraction data from online sources through february 2019. Science Editing, 7(1), 34–44.

---

1. www.altmetrics.com↩︎

2. We accessed the dataset in January 2022, but more articles have been added since.↩︎