Comparing Finnish universities’ publication profiles using multidimensional field-normalized indicators

Otto Auranen^* and Janne Pölönen^**

^* otto.auranen@aka.fi

ORCID: https://orcid.org/0000-0001-9507-2805

Academy of Finland, Hakaniemenranta 6, 00531, Helsinki, Finland

^** janne.polonen@tsv.fi

ORCID: https://orcid.org/0000-0003-1649-0879

Federation of Finnish Learned Societies, Kirkkokatu 6, 00170, Helsinki, Finland

In this study we use eight field-normalized indicators to analyze Finnish universities’ publication profiles across major fields of arts and sciences, and possible changes in publication profiles in 2016-2021. Our data consists of 241,575 publications (publication years 2016–2021) from the national VIRTA publication information service. Results indicate that the Finnish universities differ considerably in their publication profiles in Science communication, Bibliodiversity, Multilingualism, Domestic publishing, Domestic collaboration, International collaboration, Research performance and Open Access. Our indicators show the variety of organizations’ research and publishing profiles as well as the variety of overall organizational landscape. Use of comprehensive data and multidimensional indicators of publication profiles could inform and support strategic planning and monitoring of research performing organisations.

1. Introduction

Knowledge production to support science and higher education policy making, as well as most university rankings and assessments, typically rely on Web of Science (WoS) or Scopus based bibliometric indicators to evaluate and compare the performance of research organizations. Similarly, academic research into research performance of universities or other research organizations tends to rely on publication and citation data from international commercial databases, usually WoS or Scopus. Often these choices of data stem from the need to analyze citation impact, to make international comparisons, or simply from a lack of alternative data sources.

A major drawback is that organizations are evaluated based on a narrow subset of peer-reviewed English language articles in international journals. Focus is on volume of scientific publishing and its scientific impact (research performance) not on diversity of publishing (publication profiles) (Pölönen & Auranen, 2022). There are studies on other aspects of publishing, such as linguistic diversity (Linkov et al., 2021), and language and type of publishing across different fields of research (Kulczycki et al., 2018; Aksnes & Sivertsen, 2019), but systematic analyses of publication profiles at organizational level seem to be largely missing.

Several international initiatives, movements and policies around responsible research and innovation, metrics and assessment as well as open science, call for consideration of the disciplinary diversity and plurality of research contributions, societal interaction and impact (https://sfdora.org; Hicks et al., 2015; Wilsdon et al., 2015). Recently, over 500 organisations have signed the Agreement for reforming research assessment, which aims at ensuring that research assessments “recognize (…) valuable contributions that researchers make to science and for the benefit of society, including diverse outputs beyond journal publications and irrespective of the language in which they are communicated” (CoARA, 2022).

To promote multidimensional assessment of research activity, we have developed several indicators for analyzing research output (publications) of research organizations using comprehensive publication data on Finnish universities (Auranen & Pölönen, 2022). In this study we use these indicators to analyze Finnish universities’ publication profiles across major fields of arts and sciences, and possible changes in publication profiles in 2016-2021.

2. Data and methods

We created a publication dataset from VIRTA Publication Information Service, consisting of 241,575 publications (publication years 2016–2021), of which 176,327 are peer-reviewed publications and 65,248 are non-peer-reviewed publications (Table 1). All publications are validated by the 13 Finnish universities and reported annually to the Ministry of Education and Culture. Number of outputs reported for publication year 2021 is not yet entirely complete.

For each publication, the authors have to indicate at least one of 66 fields of science (Appendix 1). In addition, also the peer-review status, target audience, publication type, language, open access, number of authors, as well as international co-authorship of publications is indicated in VIRTA based on researchers’ self-reports and/or validation by the data-collection personnel at the universities. Publication Forum (JUFO) levels (Publication Forum, 2022) are indicated in VIRTA data for all peer-reviewed publications, and domestic co-authorship can be derived from duplicate records.

We use the following field-normalized indicators for multidimensional analysis of research output (publications):

1. Science communication: share of not-peer-reviewed publications aimed at professional and general audiences.

2. Bibliodiversity: share of peer-reviewed book publications (chapters, monographs and edited volumes) and conference articles.

3. Multilingualism: share of peer-reviewed publications in languages other than English (Finnish, Swedish and other languages).

4. Domestic publishing: share of peer-reviewed publications in journals and books published in Finland.

5. Domestic collaboration: share of peer-reviewed publications with co-authors from more than one Finnish university.

6. International collaboration: share of peer-reviewed publications with co-authors affiliated with foreign institutions.

7. Research performance: share of peer-reviewed outputs in JUFO levels 2 (“leading”) and 3 (“top”) publication channels.

8. Open access: share of peer-reviewed open access publications, including gold, hybrid and green OA.

Because the different dimensions of publication output differ considerably according to the field of science (Figure 1) and the Finnish universities may have very different disciplinary profiles (Figure 2), we have calculated field-normalized indicators for multidimensional analysis of publication output. For each indicator, the university’s share in each of the 66 subfields was divided by the national average, the quotient was multiplied by the number of outputs in the subfield, and their sum was divided by the total number of university’s outputs. For each indicator, the national average (here: average among Finnish universities) is 1. In order to observe possible developments, we calculated the indicators for outputs published in three-year periods of 2016-2018, 2017-2019, 2018-2020 and 2019-2021.

Table 1. Number of publication outputs 2016-2021 by output type and dimension.

Figure 1: Variation of the Finnish universities’ publication output 2016-2021 by different dimensions across main fields of science.

Figure 2: Variation of the Finnish universities’ publication output profiles 2016-2021 by main field of science.

3. Results

In this section we describe the main results of our analyses for each dimension of the Finnish universities’ publication output as shown in Figure 3 and Table 2 (in the Appendix).

1. Science communication

In the period 2019-2021, the indicator for science communication is most above the national average for the University of Turku (UTU), as measured by the share of publications aimed at professional and general audiences compared to the national averages across 66 subfields. While above the national average in 2019-2021, UTU, University of Lapland (ULA) and Åbo Akademi (AAU) show an upward trend and University of Helsinki (UH), University of Jyväskylä (JYU), Hanken School of Economics, UniArts display a downward trend since 2016-2018. Below the national average, TAU is trending down while Aalto University and University of Eastern Finland (UEF) are trending upwards, whereas University of Oulu (OYO), University of Vaasa (UVF) and Lappeenranta University of Technology (LUT) show only little change.

2. Bibliodiversity

Overall, universities differ less in bibliodiversity than science communication. In the period 2019-2021, ULA’s profile for bibliodiversity, as measured by the share of peer-reviewed conference and book publications as opposed to the journal output, is most above the national average. TAU, UniArts, Hanken and LUT show upward development since 2016-2018, while AAU and UEF display a clear downward trend.

3. Multilingualism

The largest differences between universities are observed in multilingualism. In the period 2019-2021, AAU has by far the largest share of peer-reviewed publications in languages other than English compared to the national average across 66 subfields. Possible explanation is that because AAU is the only multidisciplinary Swedish-speaking university in Finland. Also UVF shows a well above average profile for multilingualism, including substantial Swedish language output. On the other hand, Hanken is also a Swedish-speaking university but shows a predominantly English language profile. UH, UEF and UniArts remain relatively high above national average in the national and other languages besides English, while around the national average ULA, JYU, TAU and UTU show a downward trend. In addition to Hanken, also LUT, Aalto and OYO increasingly prioritise English language publications.

4. Domestic publishing

In the period 2019-2021, UniArts, UH, OYO (with upward trend) show high profiles for domestic publishing, as measured by the share of peer-reviewed output published with the Finnish publishers. Also TAU, UTU, UEF and ULA are around the national average but show downward trends. Hanken and LUT are moving away from domestic channels, while Aalto, UVF and AAU remain well below the national average.

5. Domestic collaboration

In the period 2019-2021, UEF , JYU, UTU and TAU show the highest profiles for domestic collaboration, as measured by the share of peer-reviewed output published with co-authors affiliated with other Finnish universities. UEF is the only university with a clear upward trend, while TAU and Hanken show downward trends. LUT is clearly an outlier, with a very small share of domestically co-authored outputs compared to all other universities.

6. International collaboration

In the period 2019-2021, Hanken has clearly the highest share of peer-reviewed output published with co-authors affiliated with foreign universities compared to the national average across 66 subfields. Also UVF shows a very strong upward trend, AAU is also clearly above, while ULA remains somewhat below the national average.

7. Research performance

Universities show relatively little differences in terms of the research performance, as measured by the share of peer-reviewed outputs in JUFO levels 2 (“leading”) and 3 (“top”) publication channels. In the period 2019-2021, Hanken, Aalto and LUT stand out above the national average. The most important change is the strong upward trend of LUT, UVF and ULA, as the result of which all Finnish universities are close to the national average.

8. Open access

All universities in Finland seem to have adopted and implemented open access policies with very similar results. Almost all universities remain near the national average and with very little changes, when measured by the share of peer-reviewed open access publications, including gold, hybrid and green OA. Only Hanken is slightly above, and ULA slightly below the average.

Figure 3: Publication profiles of the Finnish universities 2016-2021 based on field-normalised indicators compared to the national average (=1).

We also looked at similarities and differences between the universities’ multidimensional publication output profiles. Indicator values above national average have been highlighted with blue in Figure 4. We can see that for example Aalto, Hanken, OYO, JYU and LUT show relatively similar profiles, and very different profiles compared to UVF and AAU. However, no two or more universities have similar profiles in regard to all eight indicators. It also appears that different dimensions do not rule out each other: there are three universities that are above the national average in six partly different sets of dimensions: UH, Hanken and AAU.

When we look at the universities’ publication data through multidimensional indicators, it is also clear that many universities show above national average profiles in different dimensions. Only one university (Hanken) leads in three interrelated dimensions: research performance, international collaboration and open access. Five different universities appear on top of five other indicators: UTU in science communication, ULA in bibliodiversity, AAU in multilingualism, UniArts in domestic publishing, and UEF in domestic collaboration. All universities appear among the “top three” in at least one dimension.

Figure 4: Heatmap of publication profile indicators of the Finnish universities in 2019-2021.

Finally we compare the publication profiles of the three universities with the highest increase in research performance (ULA, LUT and UVF) with the publication profiles of other ten universities (Figure 5). The group of three universities has had a clear decrease in share of domestic publishing of peer-reviewed scientific publications. Overall the publication profile in this group of three shows mostly opposite trends from 2016-2018 to 2019-2021 compared with the group of other ten universities.

Figure 5: Comparison of Finnish universities that have increased research performance most (ULA, LUT & UVF) from 2016-2018 to 2019-2021 with the rest of universities.

4. Discussion and conclusions

Multiple indicators of publication activity bring forward the variety of organizations’ research and publishing profiles as well as the variety of overall organizational landscape; in this case the Finnish university system. Our example of “top three” universities above national average based on different indicators shows that all universities in Finland are in “top three” depending on indicators used (Figure 4). However, it’s important to remember that while research organizations’ research strategies and goal setting are subject to incentives from science and higher education policy and to interests and pressure from various stakeholders, organizations usually have autonomy in forming their own goals for research. Hence being above or below national average according to these indicators is not automatically a good or bad thing for an organization. Assessment of research and publishing in light of these indicators should be based on organizations’ own goals.

The performance-based research funding system (PRFS) for allocating 14% of core-funding annually to Finnish universities has taken into account, since 2015, all publications included in our analysis. PRFS has, however, rewarded universities more for peer-reviewed publications in JUFO level 2 and 3 channels. This appears to have had a clearest impact on the research performance of a group of three universities (ULA, LUT and UVF) which have caught up with the other ten universities. It appears that the increase in publication performance has been achieved by transferring publication activity from domestic to foreign publication channels. Changes in all other dimensions are less important, however there is also some increase in Science communication, Bibliodiversity and International collaboration, and some decrease in Multilingualism and Domestic collaboration. No difference is observed in Open Access.

The main limitation of the study is that the period from 2016 to 2021 is still relatively short for showing trends in publishing behaviour. Comprehensive national publication data provides a good information base for analysing and recognizing strengths and differences in the universities’ publication profiles. Use of multiple indicators of research profiles is useful to inform the management and staff of research organizations to support strategic planning and monitoring.

Open science practices

The original base publication metadata was downloaded from https://wiki.eduuni.fi/display/cscvirtajtp/Vuositasoiset+Excel-tiedostot, where also older and more recent datasets are openly available for download. We also make openly available an enriched and curated dataset that allows reproduction of our analyses (Auranen & Pölönen, 2023).

Acknowledgments

We thank the participants of the 27th Nordic Workshop on Bibliometrics and Research Policy in Turku 21-23 September 2022 for their feedback on an earlier version of this study.

Author contributions

Otto Auranen (otto.auranen@aka.fi): Conceptualization, Methodology, Investigation, Writing original draft, Writing review & editing. Janne Pölönen (janne.polonen@tsv.fi): Conceptualization, Data curation, Methodology, Investigation, Writing original draft, Writing review & editing, Visualisation.

Competing interests

Authors have no competing interests.

Funding information

Authors have not received external funding for this research.

References

Aksnes, D. W., & Sivertsen, G. (2019). A criteria-based assessment of the coverage of scopus and web of science. Journal of Data and Information Science, 4, 1–21. https://doi.org/10.2478/jdis-2019-0001

Auranen, O. & Pölönen, J. (2022). Multidimensional evaluation and comparison of Finnish universities across fields of science. 27^th Nordic workshop on bibliometrics and research policy. Turku, Finland, 21-23 September 2022.

Auranen, O. & Pölönen, J. (2023). Comparing Finnish universities’ publication profiles using multidimensional field-normalized indicators - dataset. Zenodo. https://doi.org/10.5281/zenodo.7847546

CoARA (2022). The Agreement full text. https://coara.eu/agreement/the-agreement-full-text/

Hicks, D., Wouters, P. F., Waltman, L., de Rijcke, S., & Rafols, I. (2015). The Leiden Manifesto for research metrics: Use these 10 principles to guide research evaluation. Nature, 520, 429–431. https://doi.org/10.1038/520429a

Kulczycki, E., Engels, T.C.E., Pölönen, J., Bruun, K., Dušková, M., Guns, R., Nowotniak, R., Petr, M., Sivertsen, G., Istenič Starčič, A. & Zuccala, A. (2018). Publication patterns in the social sciences and humanities: evidence from eight European countries. Scientometrics, 116, 463–486. https://doi.org/10.1007/s11192-018-2711-0ce

Linkov, V., O’Doherty, K., Choi, E., & Han, G. (2021). Linguistic Diversity Index: A Scientometric Measure to Enhance the Relevance of Small and Minority Group Languages. SAGE Open, 11(2). https://doi.org/10.1177/21582440211009191

Pölönen, J. & Auranen, O. (2022). Research performance and scholarly communication profile of competitive research funding: the case of Academy of Finland. Scientometrics, 127, 7415–7433. https://doi.org/10.1007/s11192-022-04385-8

Publication Forum (2022). Publication Forum. https://julkaisufoorumi.fi/en/publication-forum

Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J., & Johnson, B. (2015). The metric tide. Report of the independent review of the role of metrics in research assessment and management. HEFCE. Retrieved from https://doi.org/10.13140/RG.2.1.4929.1363