Comparing Finnish universities’ publication profiles using multidimensional field-normalized indicators

Otto Auranen^* and Janne Pölönen^**

^* otto.auranen@aka.fi

ORCID: https://orcid.org/0000-0001-9507-2805

Research Council of Finland, Hakaniemenranta 6, 00531, Helsinki, Finland

^** janne.polonen@tsv.fi

ORCID: https://orcid.org/0000-0003-1649-0879

Federation of Finnish Learned Societies, Kirkkokatu 6, 00170, Helsinki, Finland

In this study we use eight field-normalized indicators to analyze Finnish universities’ publication profiles across major fields of arts and sciences, and possible changes in publication profiles in 2016-2021. Our data consists of 241,575 publications (publication years 2016–2021) from the national VIRTA publication information service. Results indicate that the Finnish universities differ considerably in their publication profiles in Science communication, Bibliodiversity, Multilingualism, Domestic publishing, Domestic collaboration, International collaboration, Research performance and Open Access. Our indicators show the variety of organizations’ research and publishing profiles as well as the variety of overall organizational landscape. Use of comprehensive data and multidimensional indicators of publication profiles could inform and support strategic planning and monitoring of research performing organisations.

1. Introduction

Knowledge production to support science and higher education policy making, as well as most university rankings and assessments, typically rely on Web of Science (WoS) or Scopus based bibliometric indicators to evaluate and compare the performance of research organizations. Similarly, academic research into research performance of universities or other research organizations tends to rely on publication and citation data from international commercial databases, usually WoS or Scopus. Often these choices of data stem from the need to analyze citation impact, to make international comparisons, or simply from a lack of alternative data sources.

A major drawback is that organizations are evaluated based on a narrow subset of peer-reviewed English language articles in international journals. Focus is on volume of scientific publishing and its scientific impact (research performance) not on diversity of publishing (publication profiles) (Pölönen & Auranen, 2022). There are studies on other aspects of publishing, such as linguistic diversity (Linkov et al., 2021), and language and type of publishing across different fields of research (Kulczycki et al., 2018; Aksnes & Sivertsen, 2019), but systematic analyses of publication profiles at organizational level seem to be largely missing.

Several international initiatives, movements and policies around responsible research and innovation, metrics and assessment as well as open science, call for consideration of the disciplinary diversity and plurality of research contributions, societal interaction and impact (https://sfdora.org; Hicks et al., 2015; Wilsdon et al., 2015). Recently, over 500 organisations have signed the Agreement for reforming research assessment, which aims at ensuring that research assessments “recognize (…) valuable contributions that researchers make to science and for the benefit of society, including diverse outputs beyond journal publications and irrespective of the language in which they are communicated” (CoARA, 2022).

To promote multidimensional assessment of research activity, we have developed several indicators for analyzing research output (publications) of research organizations using comprehensive publication data on Finnish universities (Auranen & Pölönen, 2022). In this study we use these indicators to analyze Finnish universities’ publication profiles across major fields of arts and sciences, and possible changes in publication profiles in 2016-2021.

2. Data and methods

We created a publication dataset from VIRTA Publication Information Service, consisting of 241,575 publications (publication years 2016–2021), of which 176,327 are peer-reviewed publications and 65,248 are non-peer-reviewed publications (Table 1). All publications are validated by the 13 Finnish universities and reported annually to the Ministry of Education and Culture. Number of outputs reported for publication year 2021 is not yet entirely complete.

For each publication, the authors have to indicate at least one of 66 fields of science (Appendix 1). In addition, also the peer-review status, target audience, publication type, language, open access, number of authors, as well as international co-authorship of publications is indicated in VIRTA based on researchers’ self-reports and/or validation by the data-collection personnel at the universities. Publication Forum (JUFO) levels (Publication Forum, 2022) are indicated in VIRTA data for all peer-reviewed publications, and domestic co-authorship can be derived from duplicate records.

We use the following field-normalized indicators for multidimensional analysis of research output (publications):

1. Science communication: share of not-peer-reviewed publications aimed at professional and general audiences of the total number of publications.

2. Bibliodiversity: share of peer-reviewed book publications (chapters, monographs and edited volumes) and conference articles of the total number of peer-reviewed publications.

3. Multilingualism: share of peer-reviewed publications in languages other than English (Finnish, Swedish and other languages) of the total number of peer-reviewed publications.

4. Domestic publishing: share of peer-reviewed publications in journals and books published in Finland of the total number of peer-reviewed publications.

5. Domestic collaboration: share of peer-reviewed publications with co-authors from more than one Finnish university of the total number of peer-reviewed publications.

6. International collaboration: share of peer-reviewed publications with co-authors affiliated with foreign institutions of the total number of peer-reviewed publications.

7. Research performance: share of peer-reviewed outputs in JUFO levels 2 (“leading”) and 3 (“top”) publication channels of the total number of peer-reviewed publications.

8. Open access: share of peer-reviewed open access publications, including gold, hybrid and green OA of the total number of peer-reviewed publications.

The first four indicators relate to societal impact. Publications to professional and general audiences are a major pathway for disseminating research results beyond academia. Publishing research results not only in journal articles but also in conference papers, chapters and monographs, and in different languages, supports a broader accessibility to knowledge for diverse audiences. Publishing in national outlets, often in national languages, contributes to availability of locally relevant research in addition to research on global topics.

In general, collaboration may contribute positively to quality, impact and integrity of the published research, both in the international and national dimension (potentially including collaboration between research sectors). Publishing in the leading outlets provides one possible benchmark of scientific quality and impact. Relying on expert-based JUFO levels and VIRTA data enables more comprehensive recognition of the diverse scholarly publication output across all fields than would be possible if analysis relied on citation indicators (such as JIF) calculated based on publication data only from Web of Science or Scopus. Open availability of the published research is the prime indicator of open science and contributes to dissemination of results within and beyond academia.

We also note that these indicators have been selected partly because of the availability of the required information in the VIRTA data. There could be other relevant indicators, such as interdisciplinarity or open data and methods sharing, for which we cannot readily derive robust information from VIRTA.

Because the different dimensions of publication output differ considerably according to the field of science (Figure 1) and the Finnish universities may have very different disciplinary profiles (Figure 2), we have calculated field-normalized indicators for multidimensional analysis of publication output. For each indicator, the university’s share in each of the 66 subfields was divided by the national average, the quotient was multiplied by the number of outputs in the subfield, and their sum was divided by the total number of university’s outputs. For each indicator, the national average (here: average among Finnish universities) is 1. In order to observe possible developments, we calculated the indicators for outputs published in three-year periods of 2016-2018, 2017-2019, 2018-2020 and 2019-2021.

Table 1. Number of publication outputs 2016-2021 by output type and dimension.

Figure 1: Variation of the Finnish universities’ publication output 2016-2021 by different dimensions across main fields of science.

Figure 2: Variation of the Finnish universities’ publication output profiles 2016-2021 by main field of science.

3. Results

In this section we describe the main results of our analyses for each dimension of the Finnish universities’ publication output as shown in Figure 3 and Table 2 (in the Appendix).

1. Science communication

In the period 2019-2021, the indicator for science communication is most above the national average for the University of Turku (UTU), as measured by the share of publications aimed at professional and general audiences compared to the national averages across 66 subfields. While above the national average in 2019-2021, UTU, University of Lapland (ULA) and Åbo Akademi (AAU) show an upward trend and University of Helsinki (UH), University of Jyväskylä (JYU), Hanken School of Economics, UniArts display a downward trend since 2016-2018. Below the national average, TAU is trending down while Aalto University and University of Eastern Finland (UEF) are trending upwards, whereas University of Oulu (OYO), University of Vaasa (UVF) and Lappeenranta University of Technology (LUT) show only little change.

2. Bibliodiversity

Overall, universities differ less in bibliodiversity of their peer-reviewed output than in science communication (share of non-peer-reviewed outputs for professional and general audiences of the total output). In the period 2019-2021, ULA’s profile for bibliodiversity, as measured by the share of peer-reviewed conference and book publications as opposed to the peer-reviewed journal output, is most above the national average. TAU, UniArts, Hanken and LUT show upward development since 2016-2018, while AAU and UEF display a clear downward trend.

3. Multilingualism¹

Universities’ profiles may differ considerably in terms of multilingualism of their peer-reviewed output compared to science communication. In the period 2019-2021, UEF and UH have the largest share of peer-reviewed publications in languages other than English compared to the national average across 66 subfields. They have also taken over TAU and UTU that show a downward trend but remain above the national average, as does UniArts. UVF shows a strong upward trend, while AAU, Hanken and especially LUT show a downward trend. LUT was above the national average in 2016-20218 but now has the lowest index value for multilingualism.

4. Domestic publishing

In the period 2019-2021, UniArts, UH, OYO (with upward trend) show high profiles for domestic publishing, as measured by the share of peer-reviewed output published with the Finnish publishers. Also TAU, UTU, UEF and ULA are around the national average but show downward trends. Hanken and LUT are moving away from domestic channels, while Aalto, UVF and AAU remain well below the national average.

5. Domestic collaboration

In the period 2019-2021, UEF, JYU, UTU and TAU show the highest profiles for domestic collaboration, as measured by the share of peer-reviewed output published with co-authors affiliated with other Finnish universities. UEF is the only university with a clear upward trend, while TAU and Hanken show downward trends. LUT is clearly an outlier, with a very small share of domestically co-authored outputs compared to all other universities.

6. International collaboration

In the period 2019-2021, Hanken has clearly the highest share of peer-reviewed output published with co-authors affiliated with foreign universities compared to the national average across 66 subfields. Also UVF shows a very strong upward trend, AAU is also clearly above, while ULA remains somewhat below the national average.

7. Research performance

Universities show relatively little differences in terms of the research performance, as measured by the share of peer-reviewed outputs in JUFO levels 2 (“leading”) and 3 (“top”) publication channels. In the period 2019-2021, Hanken, Aalto and LUT stand out above the national average. The most important change is the strong upward trend of LUT, UVF and ULA, as the result of which all Finnish universities are close to the national average.

8. Open access

All universities in Finland seem to have adopted and implemented open access policies with very similar results. Almost all universities remain near the national average and with very little changes, when measured by the share of peer-reviewed open access publications, including gold, hybrid and green OA. Only Hanken is slightly above, and ULA slightly below the average.

Figure 3: Publication profiles of the Finnish universities 2016-2021 based on field-normalised indicators compared to the national average (=1).

We also looked at similarities and differences between the universities’ multidimensional publication output profiles. Indicator values above national average have been highlighted with grey in Figure 4. We can see that for example that Aalto, Hanken and LUT show relatively similar profiles, in which somewhat different dimensions are prominent compared to UH, UEF and UniArts.. However, no two or more universities have similar profiles in regard to all eight indicators.

Certain dimensions of the publication profile tend to be positively related. All five universities that have above national average values in multilingualism also have above average value in domestic publishing but only one of them do so in research performance and two in international collaboration. Yet different dimensions do not rule out each other entirely: there are universities that are above the national average in six (UH and Hanken) or five (Aalto, OYO, UTU and AAU) partly different sets of dimensions.

When we look at the universities’ publication data through multidimensional indicators, it is also clear that many universities show above national average profiles in different dimensions. Hanken leads in three interrelated dimensions: research performance, international collaboration and open access; while UEF leads in multilingualism and domestic collaboration. Different universities appear on top of three other indicators: UTU in science communication, ULA in bibliodiversity, and UniArts in domestic publishing. All universities appear among the “top three” in at least one dimension but there are only two universities that are “top three” in more than two dimensions.

Figure 4: Heatmap of publication profile indicators of the Finnish universities in 2019-2021 (grey colour indicates value above national field-normalised average = 1).

Finally we compared the publication profiles of the Finnish universities with regard to the changes in the interrelated dimensions of research performance, international collaboration, multilingualism and domestic publishing between 2016-2021 (Figure 5). A group of three universities has had a clear increase in research performance as measured by share of JUFO level 2 and 3 peer–reviewed outputs: LUT, ULA and UVF. Yet these three universities differ considerably in development of multilingualism and domestic publishing: while LUT has decreased dramatically, UVF has at the same time improved index values and ULA’s values remain largely unchanged. UVF is also the only one of the three that has increased international collaboration.

Figure 5: Comparison of Finnish universities with regard to change in multilingualism, domestic publishing, international collaboration and research performance.

4. Discussion and conclusions

Multiple indicators of publication activity bring forward the variety of organizations’ research and publishing profiles as well as the variety of overall organizational landscape; in this case the Finnish university system. Our example of “top three” universities above national average based on different indicators shows that all universities in Finland are in “top three” depending on indicators used (Figure 4). However, it’s important to remember that while research organizations’ research strategies and goal setting are subject to incentives from science and higher education policy and to interests and pressure from various stakeholders, organizations usually have autonomy in forming their own goals for research. Hence being above or below national average according to these indicators is not automatically a good or bad thing for an organization. Assessment of research and publishing in light of these indicators should be based on organizations’ own goals.

The performance-based research funding system (PRFS) for allocating 14% of core-funding annually to Finnish universities has taken into account, since 2015, all publications included in our analysis. PRFS has, however, rewarded universities more for peer-reviewed publications in JUFO level 2 and 3 channels. This appears to have had a clearest impact on the research performance of a group of three universities (ULA, LUT and UVF) which have caught up with the other ten universities. It appears that the increase in publication performance has been achieved by transferring publication activity from domestic to foreign publication channels only in the case of LUT, whereas UVF shows an upward trend also in multilingualism and domestic publishing.

The main limitation of the study is that the period from 2016 to 2021 is still relatively short for showing trends in publishing behaviour. Comprehensive national publication data provides a good information base for analysing and recognizing strengths and differences in the universities’ publication profiles. Use of multiple indicators of research profiles is useful to inform the management and staff of research organizations to support strategic planning and monitoring. In further work, we aim to look at trends in publication profiles with longer publication data, clusters of universities based on publication profiles, and correlations between the eight indicators.

Open science practices

The original base publication metadata was downloaded from https://wiki.eduuni.fi/display/cscvirtajtp/Vuositasoiset+Excel-tiedostot, where also older and more recent datasets are openly available for download. We also make openly available an enriched and curated dataset that allows reproduction of our analyses (Auranen & Pölönen, 2023).

Acknowledgments

We thank the participants of the 27th Nordic Workshop on Bibliometrics and Research Policy in Turku 21-23 September 2022 for their feedback on an earlier version of this study.

Author contributions

Otto Auranen (otto.auranen@aka.fi): Conceptualization, Methodology, Investigation, Writing original draft, Writing review & editing. Janne Pölönen (janne.polonen@tsv.fi): Conceptualization, Data curation, Methodology, Investigation, Writing original draft, Writing review & editing, Visualisation.

Competing interests

Authors have no competing interests.

Funding information

Authors have not received external funding for this research.

References

Aksnes, D. W., & Sivertsen, G. (2019). A criteria-based assessment of the coverage of scopus and web of science. Journal of Data and Information Science, 4, 1–21. https://doi.org/10.2478/jdis-2019-0001

Auranen, O. & Pölönen, J. (2022). Multidimensional evaluation and comparison of Finnish universities across fields of science. 27^th Nordic workshop on bibliometrics and research policy. Turku, Finland, 21-23 September 2022.

Auranen, O. & Pölönen, J. (2023). Comparing Finnish universities’ publication profiles using multidimensional field-normalized indicators - dataset. Zenodo. https://doi.org/10.5281/zenodo.7847546

CoARA (2022). The Agreement full text. https://coara.eu/agreement/the-agreement-full-text/

Hicks, D., Wouters, P. F., Waltman, L., de Rijcke, S., & Rafols, I. (2015). The Leiden Manifesto for research metrics: Use these 10 principles to guide research evaluation. Nature, 520, 429–431. https://doi.org/10.1038/520429a

Kulczycki, E., Engels, T.C.E., Pölönen, J., Bruun, K., Dušková, M., Guns, R., Nowotniak, R., Petr, M., Sivertsen, G., Istenič Starčič, A. & Zuccala, A. (2018). Publication patterns in the social sciences and humanities: evidence from eight European countries. Scientometrics, 116, 463–486. https://doi.org/10.1007/s11192-018-2711-0ce

Linkov, V., O’Doherty, K., Choi, E., & Han, G. (2021). Linguistic Diversity Index: A Scientometric Measure to Enhance the Relevance of Small and Minority Group Languages. SAGE Open, 11(2). https://doi.org/10.1177/21582440211009191

Pölönen, J. & Auranen, O. (2022). Research performance and scholarly communication profile of competitive research funding: the case of Academy of Finland. Scientometrics, 127, 7415–7433. https://doi.org/10.1007/s11192-022-04385-8

Publication Forum (2022). Publication Forum. https://julkaisufoorumi.fi/en/publication-forum

Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J., & Johnson, B. (2015). The metric tide. Report of the independent review of the role of metrics in research assessment and management. HEFCE. Retrieved from https://doi.org/10.13140/RG.2.1.4929.1363