How can altmetrics improve the Public Communication of Science and Technology? An analysis on universities and altmetrics

How can altmetrics improve the Public

Communication of Science and Technology?

An analysis on universities and altmetrics

Cibele Maria Garcia de Aguiar*; Sergio Luiz Monteiro Salles-Filho**; Ana Carolina Spatti**; Karen E F Pinto**; Sérgio Parreiras Pereira**

*cibele.aguiar2@gmail.com

ORCID: 0000-0002-2416-9818

Science and Technology Policy Department-University of Campinas

^** sallesfi@unicamp.br; anaspatti@ige.unicamp.br; karenefp@unicamp.br; sergio.parreiras@sp.gov.br

0000-0002-8727-4500; 0000-0001-8839-8276; 0000-0002-7217-2632; 0000-0002-6388-8284

Science and Technology Policy Department-University of Campinas; Instituto Agronômico/IAC – Campinas, SP, Brasil.

In current research evaluation models, monitoring and impact evaluation are extended beyond peer-reviewed articles to include Science and Technology Public Communication activities. Through an online survey, we analyzed the perceptions of relevance and degree of application of the altmetric indicators for the PCST of 51 sampled Brazilian federal universities. Perceptions of relevance and application of altmetrics proved to be an outlier in 26 indicators. 66.7% of respondents said they did not know the relevance of altmetrics for the PCST or considered it not applicable to the field. Regarding the perception of relevance, the indicator “Mentions tracked by altmetrics” received high relevance scores (7 and 9) from 21.5% of respondents. The indicator was also the least applied, with only one university (1.9%) using it. In addition, 45% of respondents reported having no intention of applying it, 41.1% intend to apply it in the long term, and 11.7% in the short term.

1. Introduction

The management of scientific funding involves obtaining objective criteria for evaluating the results and performance of researchers, departments, institutions, and even countries (Garfield, 2003). The current scientific evaluation paradigm remains based on classic metrics, almost always at the article level, reviewed by peers, and in scientific journals with different impact factors, measured mainly by citations and derivated indicators (Ioannidis, Boyack & Bass, 2020). However, in current research evaluation frameworks, objects and activities are being extended beyond the science community to reach a broader and more diverse public and impact, to which citation-based research evaluation shows many limitations and biases (Mingers & Leydesdorf, 2015; Fonseca Jr., 2019; Jonker, Vanlee & Ysebaert, 2022).

Recent demands for measuring the societal impact of research or the broader impact of research have reinforced the importance of Public Communication of Science and Technology (PCST). Higher education institutions, funding research agencies, and other policymakers have, in response, increased efforts to develop and apply methods and indicators in their PCST activities for measuring the broader impact of their research (Mayne, 2015; Buhmann & Likely, 2018; Ziegler, Hedder & Fischer, 2021). In this context, alternative metrics (Altmetrics) are considered a new way for assessing scholarly communication on the web and public engagement with research.

Altmetrics represent an alternative way of tracking the involvement of different actors and scientific products in the online environment (Priem et al., 2010). This field of study is associated with social impact indicators, primarily through mentions on platforms and social networks (Sugimoto et al., 2017; Zahedi & Costas, 2018; Joubert & Costas, 2020). It is due to the consideration that the digital space has an inherent social aspect (Soós & Kiss, 2020). Furthermore, academics face increasing pressure for evidence that guarantees the desired social impact (Haustein, Bowman & Costas, 2016).

This article analyzes the perception of altimetric relevance and application in Brazilian federal universities. We question whether altmetrics is an effective way of measuring and communicating the impact of research. We use primary data from a survey with science communication professionals from Brazilian federal universities.

2. Theoretical background and purpose

In a bibliographical review of informatics as a methodological framework in analyzing science-society communications, Soós and Kiss (2020) emphasize that Internet platforms provide new channels and modes for the research community to interact with the public and have allowed the digital traces of these communications. Through the mapping, they considered that despite the great thematic diversity, the impact of science is highlighted as a relevant aspect and constitutes one of the observed clusters. In this context of communication beyond the academy, they describe as main points the social impact of scientific research, the management of social dissemination, and the use of research production.

Alomoto, Ninerola, and Pié (2021) deal with the Social Impact Assessment (SIA) concept and recognize that it plays a fundamental role in the public sector. The authors cite the Horizon 2020 program (European Union) and the 17 Sustainable Development Goals (SDGs) proposed by the UN (2015) as milestones in the appreciation of SIA, both highlighting the impact assessment as crucial to highlight the achievement of goals, create better methods, adhere to the results of initiatives and support decision-making. The Horizon 2020 initiative required that research projects be accompanied by a communication plan, including activities for disseminating results, incorporating mechanisms for dialogue, and establishing systems for measuring results. This demand from funders contributed to a change in the way of conceiving and planning scientific communication (Gertrudix & Rajas, 2020).

For Haustein, Bowman, and Costas (2016), alternative metrics can appear not only as article-level metrics but can also be applied to a broad spectrum of research objects. The CPCT is directly related since it can be considered a non-academic object linked to an academic object. The lack of a general conceptualization of social media's role as a specific interface between science and society has led the field of altmetrics studies to research the phenomenon more broadly.

In what Costas, Rijcke, and Marres (2021) called heterogeneous couplings, they suggest considering the different acts related to different research objects. It is interesting to highlight the separation they suggest of these acts into engagement categories: access, evaluation, and application, presented within a spiral scheme. That is, the level of engagement increases moving between categories and, therefore, between layers (Haustein, Bowman & Costas, 2016). This involvement refers to the content of an academic object.

As Aguiar & Salles-Filho (2022) PCST monitoring and evaluation model suggested, the number of scholarly outputs mentioned on social media is one of the indicators listed. The complete list has 26 indicators, including the altmetrics. This model considers communication and interaction a broader and systematic process, not a specific product. A highlighted point is that altmetrics is one of the indicators that make up the model, not being, in isolation, sufficient to explain the effect of CPCT on the science-society relationship.

Developing performance evaluation systems is challenging for teaching and research organizations. These organizations must justify the resources invested, which requires expanding their bibliometric indicators beyond strictly scientific communication (Fonseca Jr. et al., 2019).

On the other hand, it may represent an opportunity for S&T institutions. Montesi and Villaseñor (2018) studied the use of altmetrics from an institutional point of view, suggesting that universities could use these metrics to measure and monitor their participation and ability to interact in digital media. However, they recognize that this way of measuring the social implications of institutional activity is more common in academic libraries than in the communication sectors of universities.

Zahedi and Costas (2018) also confirm broad possibilities for using alternative strategies to study relationships and adolescents between social media and academic entities. However, they warn of the need to understand potential data quality challenges in capturing social media events around academic objects. Also, pay attention to the need to know the specifics of each altmetric aggregator, the influence of the time of data collection, types of sources tracked, use of applications, and choice of identifiers to track social media data. When an institution incorporates altmetrics to evaluate the impact of academic products, including a CPCT, it must consider this complexity.

The difficulty posed in this reflection, however, is related to the knowledge of these relationships and the potential application of these evaluation metrics, particularly from the point of view of scientific journalism and S&T communicators in general. After all, what is the perception of the Communication teams at Brazilian federal universities about altmetrics? What are the perceptions of relevance and the degree of its application?

Our proposal in this article is to present some data about the perception and application of altmetrics in the context of Brazilian universities. Besides that, we justify the inclusion of altmetrics in PCST monitoring and evaluation models, highlighting its importance and recognition as a trend in evaluating S&T due to its wide dissemination and the possibility of standardized comparison.

3. Methodology

Through an online survey, we analyzed the perceptions of relevance and degree of application of altmetrics indicators for PCST from 51 sampled universities (73.4% of the universe of 69 Brazilian federal universities). The responding institutions represent our object of study from a universe of 69 existing ones since they include different foundation ages, sizes, geographic positions, areas of excellence, forms of communication, and strategic objectives. Our objective was to understand how Brazilian federal universities have treated the PCST theme through the perception of Communication managers. We have done this from the perspective of monitoring and evaluating, including a specific section on altmetrics.

Data collection took place from July 7th to September 2nd, 2022. The questionnaire was addressed and answered by the professional responsible for the Communication sector, which receives different nomenclatures. We asked subjects to select one of three options: a) Altmetrics already applied in PCST activities; b) Willing to apply altmetrics in PCST activities in the short term (up to two years); c) Willing to apply altmetrics in PCST activities in the medium or long term (more than two years); and d) No intention to apply altmetrics in PCST activities.

Additionally, we used a five-point Likert scale to select the degree of relevance of these indicators in the universities. To indicate the degree of relevance, we used a five-point Likert scale (1,3,5,7,9), with 1 being the lowest relevance and nine the highest.

For this research, the project and questionnaire were approved by the Research Ethics Committee of the State University of Campinas (CEP/Unicamp) approved the project and questionnaire under number CAAE 50650921.4.0000.8142.

4. Results and Discussion

Based on the survey carried out, as a result, 66.7% of respondents said that they did not know the relevance of altmetrics for PCST or considered them not applicable to the field. For 15.7%, altmetrics is a research metric for researchers' use, and 13.7% of the science communication professionals tried to use alternative metrics but had limitations in using the platforms. Only 3.9% (two universities) already used them to identify the impact of articles and topics of public interest.

Regarding the perception of relevance, the indicator "Mentions tracked by altmetrics" received scores of 7 by 13.7% and nine by 7.8% of respondents. This low relevance may be because the respondents needed to know the importance of altmetrics for S&T communication or its applicability in the context of the surveyed universities.

The indicator "Number of scholarly outputs tracked on social media" was the least applied, with only one university (1.9%) using it. Moreover, 45% of respondents reported no intention to apply it, 41.1% intend to apply it in the long term, and 11.7% in the short term. In the context of Brazilian federal universities, the relevance and application of altmetrics proved to be an outlier of the set of 26 analyzed indicators.

As Montesi & Villaseñor (2018) and Zahedi & Costas (2018) suggest, universities are not taking advantage of the possibilities of using altmetrics. As highlighted by Alomoto, Ninerola, and Pié (2021), we observed that institutions fail to take advantage of altmetric mentions or altmetric scores to assess the social impact of research on the perspective of communication with society. One of the reasons for this stems from the very lack of knowledge of this use for monitoring and evaluating PCST.

5. Final considerations

Through the data presented, we can assume that altmetrics still needs to be discovered for S&T journalists and communicators, with low applicability for evaluating non-academic products. There is an effort to develop metrics and their analysis regarding the relationship between science and society. However, CPCT is still a marginal field of study in this context. Altmetrics could expand its limits of recognition and action by including CPCT theories and models in its analyses.

The inclusion of the CPCT perspective has the potential not only to expand the frontiers of the field of altmetrics studies but also to favor the practice of CPCT by incorporating evidence in the choice of topics, research of interest, sources, and relevant actors in the process of communicating S&T beyond academic boundaries. Even if today, the perception of altmetrics is of low relevance for professionals at Brazilian universities, this perception must be as informed as possible, even for them to take advantage of the potential of these metrics and claim a review of the barriers that prevent their use.

Open science practices

Those interested in the questionnaire and aggregated data, without mentioning names, send a request to the email cibele.aguiar2@gmail.com.

Acknowledgments

The authors acknowledge support from the São Paulo Research Foundation - FAPESP

Author contributions

Cibele Maria Garcia de Aguiar: conceptualization, investigation, methodology, writing original draft, writing review

Sergio Luiz Monteiro Salles-Filho: conceptualization, investigation, methodology, writing review and supervision

Ana Carolina Spatti: writing original draft, writing review and editing

Karen E. F. Pinto: writing original draft, writing review and editing

Sérgio Parreiras Pereira: writing review and editing

Competing interests

The authors declare that they have no conflict of interest.

References

Alomoto, W., Niñerola, A., & Pié, L. (2022). Social Impact Assessment: A systematic review of literature. Social Indicators Research, 161(1), 225-250. https://doi.org/10.1007/s11205-021-02809-1.

Aguiar Pereira, C. M. G., & Salles-Filho, S. L. M. (2022). Tipos ideais e Teoria da Mudança: proposição de modelo de avaliação para a comunicação pública de ciência e tecnologia. Journal of Science Communication, América Latina, 5(2), A03. https://doi.org/10.22323/3.05020203

Buhmann, A., & Likely, F. (2018). Evaluation and measurement in strategic communication. The international encyclopedia of strategic communication, 1, 625-640.

Costas, R., de Rijcke, S., & Marres, N. (2021). “Heterogeneous couplings”: Operationalizing network perspectives to study science‐society interactions through social media metrics. Journal of the Association for Information Science and Technology, 72(5), 595-610. https://doi.org/10.1002/asi.24427.

da Fonseca Júnior, W. C., de Camargo Penteado Filho, R., Avila, A. F. D., de Desempenho Institucional, C. D. A., & Cardoso, C. C. (2019). A avaliação bibliométrica de instituições de pesquisa para além da comunicação científica: o caso Embrapa.

Garfield, E. (2003). The meaning of the impact factor. International Journal of clinical and Health Psychology, 3(2), 363-369.

Gertrudix, M., & Rajas, M. (2020). Gestión de la comunicación científica de los proyectos de investigación en H2020. Funciones, modelos y estrategias. Profesional de la información, 29(4). https://doi.org/10.3145/epi.2020.jul.24.

Haustein, S., Bowman, T. D., & Costas, R. (2016). Interpreting “altmetrics”: viewing acts on social media through the lens of citation and social theories. Theories of informetrics and scholarly communication, 372-406.

Ioannidis, J. P., Boyack, K. W., & Baas, J. (2020). Updated science-wide author databases of standardized citation indicators. PLoS biology, 18(10), e3000918.

Jonker, H., Vanlee, F., & Ysebaert, W. (2022). Societal impact of university research in the written press: media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium. Scientometrics, 1-18. https://doi.org/10.1007/s11192-022-04374-x

Kassab, O., Bornmann, L., & Haunschild, R. (2020). Can altmetrics reflect societal impact considerations?: Exploring the potential of altmetrics in the context of a sustainability science research center. Quantitative Science Studies, 1(2), 792-809. https://doi.org/10.1162/qss_a_00032

Mayne, J. (2015). Useful theory of change models. Canadian Journal of Program Evaluation, 30(2).

Mingers, J., & Leydesdorff, L. (2015). A review of theory and practice in scientometrics. European journal of operational research, 246(1), 1-19. https://doi.org/10.1016/j.ejor.2015.04.002

Montesi, M., & Villaseñor Rodríguez, I. (2018). El impacto social de las instituciones de educación superior: un estudio de caso con la Universidad Complutense de Madrid. Información, cultura y sociedad, (39), 37-60.

Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the association for information science and technology, 68(9), 2037-2062.

Zahedi, Z., & Costas, R. (2017). How visible are the research of different countries: an analysis of global vs. local reach of WoS publications on Twitter. In 16th International Conference on Scientometrics & Informetrics (ISSI2017). https://doi.org/10.1002/asi.23833/.

Ziegler, R., Hedder, I. R., & Fischer, L. (2021). Evaluation of science communication: current practices, challenges, and future implications. Frontiers in Communication, 6, 669744. https://doi.org/10.3389/fcomm.2021.669744