The reform of evaluation procedures in research assessment: The positioning of the societal impact of research within research assessment - an interim analysis

Ann Campbell*, Briony Fane**, and Juergen Wastl***

Digital Science, 6 Briset Street, London, EC1M 5NR, United Kingdom

* https://orcid.org/0000-0002-8575-5052

** https://orcid.org/0000-0001-6639-7598

*** https://orcid.org/0000-0001-7757-8001.

Short Abstract

Attention is shifting towards evaluating research in a more responsible manner beyond traditional metrics, and societal impact has now entered the arena becoming a core component of research assessment.

This paper looks at the UK’s Research Excellence Framework 2021 Impact case studies through the lens of the societal impact of research within the UN’s Sustainable Development Goals (SDGs).

Outcomes reveal that a more nuanced and fine grained view of societal impact in the context of the REF impact case study database is possible. This study and its methodology can be applied to any text based narrative of impact case study. Using a component of impact as submitted to a research assessment exercise in conjunction with the UNs sustainability agenda would appear to provide a way to position societal impact that is revealing and informative for the assessment of research.

Introduction

The challenges in our world today are immense, ranging from global issues such as climate change, to regional economic and cultural challenges. The impact of academic research addressing the global challenges has gained importance and has become increasingly embedded in policies referring to contributions to the knowledge society, and to solving global and societal challenges.The world has become so dependent on new and reliable knowledge and a highly educated workforce, that governments have intensified their explicit demands for societal impact from universities in general, and from research in particular (Fecher, Kuper, Sokolovska, 2021). Thus, research institutions face increasing pressure to demonstrate the impact of their research.

To address this, academic research is no longer assessed according to its scientific relevance alone but also according to the value it appears to generate for society (van den Akker, Spaapen, & Maies, 2017) - its societal impact. The concept of ‘societal impact’ has gained momentum within the research ecosystem in recent years, becoming rooted in research policies, funding instruments, and research evaluation methods. From a societal perspective impact is taken to mean the effect of research beyond academia, or the effect or change experienced by society from that research. Research contributions that address the social, environmental and economic needs outside of academia are increasingly important for evaluating and assessing the research of an academic institution. However, finding the most appropriate quantitative measure to assess this impact is not straightforward. One of the major challenges for developing a measure of impact is that it has different meanings for research institutions and is assessed differently (Pedersen, Grønvad, & Hvidtfeldt, 2020).

Approaches taken to measure the societal impact of research are improving despite the challenges. For example, in the UK the research impact agenda assumes that publicly funded research should be beneficial to society and to this end, the UK’s Research Excellence Framework (REF) is one of a number of frontrunners in this area and has incorporated impact case studies and impact narratives into its national research assessment exercises to assess the demonstrable impact of the UK’s excellent research beyond academia. This clearly signifies that assessing research quality based on traditional metrics is no longer considered sufficient. Likewise in Australia, the ERA introduced the Engagement and Impact Assessment as part of its Excellence in Research for Australia (2022). This aspect of the assessment evaluates how well universities are translating research into impacts beyond academia. In the Netherlands the SEP includes an assessment criteria looking at the relevance of research to society, “...assessing the quality, scale and relevance of contributions targeting specific economic, social or cultural target groups, of advisory reports for policy, of contributions to public debates, and so on.” (Standard Evaluation Protocol 2015 – 2021).

What academic research offers for the social, environmental and economic aspects of life can be addressed through the lens of the United Nations sustainability agenda and societal impact is particularly relevant where the research relates to one of the UN’s Sustainable Development Goals (SDGs). Although there are multiple pathways to impact, the inclusion of impact as a component of the UK’s Research Excellence Framework (REF) reflects a growing interest in demonstrating the value of academic research from a societal perspective.

In this paper, we take the UK’s Research Excellence Framework to analyse submitted research impact statements and map these on to the SDG classification system incorporated into Digital Science’s Dimensions database. The SDG classification system balances the three overarching pillars of sustainable development: the economic, social and environmental aspects. REF Impact Case Studies categorise impact into eight types by : Cultural, Economic, Environmental, Health, Legal, Political, Societal and Technological. Both the SDGs and the REF categorisations allow for an analysis of societal impact.

Methodology

Our methodology is based on characterising and visualising impact through the lens of Dimensions’ SDG classification system and REF’s classification of impact, to reveal links and relationships amongst the impact narratives and their underpinning research. To execute this we used Dimensions on Google BigQuery to analyse the Impact Case Studies from REF 2021. We analysed the summary narratives submitted to the UK REF2021 using existing research classification systems to categorise the impact narratives - in this instance the SDG research classification system (Wastl et al 2020) in Digital Science’s Dimensions database. To assess the parameters of interest, we attach the SDG classification to the underpinning research, the summary narratives and grant funding abstracts and present these visually in the form of alluvial charts showing the interconnectedness between the societal impact of narrative and underpinning research and grant funding by the REF classification and SDG classification. We present outcomes within and across the parameters of impact.

We focus on three questions to address research assessment in the context of societal impact:

1. How are REF2021 impact narratives classified? Mapping SDG classifications onto Impact narratives and compiling against the manual tag in the REF database: Is it a good match?

2. What does mapping and cross referencing REF impact types with SDG classification reveal?

3. How do the underpinning research publications and funding, after being categorised by

SDGs, relate to impact type for the case studies?

There are several ways the SDGs can be grouped to outline societal impact. In our analysis we take the three layered (SDGs Wedding Cake - see Figure 1) depiction of the SDGs to align the 17 Goals into three broad categories of social, environmental (biosphere) and economic which make up societal impact (we treat SDG17 within the social goals as an all encompassing, embracing aspect of societal impact). Aggregating individual goals in this way allows us to visualise and analyse overall trends, and enables us to compare SDG-related publications underpinning impact case studies to highlight citation advantages (Fane 2021).

Figure 1 The SDGs Wedding Cake.

This paper expands and goes beyond the previous approach undertaken looking at citations of the underpinning research only, by applying the SDG classification to the impact narrative separately. In order to achieve this classification, we downloaded the REF 2021 Impact case studies and extracted the text for impact summaries and classified 6,737 impact case studies according to the SDG classifier - we retrieved up to four SDGs labels per impact narrative and added the results to GBQ to have the underlying data available for analysis via Google Data Studio.

Table 1 Number of Impact Case Studies with number of assigned SDG’s following SDG Classification

Data Analysis

We first address the question of how many impact narratives can be classified by SDGs. Figure 2 below provides an overview of the number of REF impact case studies (sorted by impact summary type as per the REF database) and the count of SDG classified narratives of case studies in each category. 6,361 out of 6,637 impact case study narratives were classified (95.8%). The predominant reason for not achieving 100% retrieval is that those narratives under the 100 word abstract size would not produce a statistically significant result on the classification (applying a 0.5 threshold for inclusion).

Figure 2 Count of SDG classified impact case studies narratives by REF Impact type

Figure 2 provides a quality check to assess the confidence that each of the eight REF impact type categories are covered correspondingly by the SDG classification.

Applying the broad SDG categories according to the SDGs Wedding Cake we retrieved the following results (see Figure 3): For REF impact case studies labelled ‘environmental’ in the REF database our broad category of environmental SDGs (7,13,14, and 15) is the dominant classification as expected. Interestingly, our classification by environmental SDGs also identifies case studies that are in technological, societal and (to a lesser extent cultural) REF impact types.

Figure 3 SDG broad categories (social, environmental and economic) mapped across the REF impact types

Addressing Question 2 - why and how the use of the eight REF impact types and 17 SDGs to highlight what the additional opportunities in analysis and visualising Impact provides. This additional level of granularity that we applied to the SDG labelled impact narratives, on top of the eight summary impact types in the REF database allows for more nuanced views, both from the perspective of the REF definitions as well as the SDG definitions.

We provide an example to show this by probing into an individual SDG to retrieve impact case studies from different impact types. We take the case of SDG 11 (Sustainable Cities and Communities), and analyse the number and the type of impact cases studies (Figure 4). This helps to improve understanding (e.g. stakeholders and policymakers) on what type of impact in SDG 11 is revealed. Figure 4 shows the types of impact contributing to SDG11 and a number of different impact types are shown.

Figure 4 Share of impact types for SDG 11 Sustainable Cities and Communities

We can also revert the starting point and ask which SDGs contribute to any of the eight REF impact types. For example, the technological impact type is covered by all three broad SDG categories and a high number of SDGs contribute to impact in this area. As seen in Figure 3, legal, health and political impact case studies link to SDGs in the social broad category (e.g. SDG 1-6,11 and 16). In the case of ‘political’ impact type, Figure 5 shows the contributing SDGs for case studies under this impact type and Figure 6 details the number of publications per SDG underpinning political type impact case studies.

Figure 5 Alluvial Chart displaying the broad categories and individual SDGs contributing to political-type REF impact case studies.

Figure 6 Bar Chart displaying the individual SDGs relating to the underpinning research publications informing political REF impact types

To address Question 3 we took the underpinning research informing the REF impact case studies and the grant funding enabling the generation of impact case studies. We extracted the metadata and text from the REF impact case study database and, after matching to the Dimensions database, used the GBQ data platform to display the number of underpinning research publications and funding by REF impact types and broad SDG categories. Figures 7 and 8 give an overview of the quantity of underpinning outputs and grant funding, respectively, while highlighting noticeable differences. The number of publications in the context of SDG are highest in societal Impact case studies, while environmental case studies exhibit the highest of grants that are labelled by broad SDG categories. Both the societal and environmental impact types have the highest share of impact case study type categorisations, followed by health and technological type impact.

Health type impact seems exclusively underpinned by publications labelled with SDG in the area of society impact (most noticeable by publications linked to SDG 3).

With the exception of legal impact type, REF impact types are underpinned by publications and funding from a broad range of SDG related research. Interesting follow up analysis can highlight particular domains but is outside the scope of this particular analysis.

Figure 7: Count of underpinning research publications, grouped by broad SDG categories and impact type

Figure 8: Count of underpinning research funding, grouped by broad SDG categories and REF impact type

It is important to note that not the full number of underpinning research publications or grant funding could be matched to the Dimensions database, nor are all matched objects linked to one or more of the SDGs. The match rate to SDGs for those that were identified and matched was 33% (see Table 2) and exceeds the overall general share of SDG classification for publications in the Dimensions database.

Table 2 Breakdown of Impact Case Studies, Impact Case Studies classified by SDG, Classified Impact Case Studies with identified Publications and Classified Impact Case Studies with SDG Publications

Conclusions

We were able to overlay two classifications in the context of societal impact. For the subsequent analysis we brought together the 8-tier label on impact case study narratives in the REF Impact case study database and the 17 distinct SDG.

This overlay allows a refined and nuanced view starting from either the Impact case study (studying the links to the relevant SDGs), or, with an SDG on the onset, we can look at the respective contribution research in the context of the SDG made in the area of types of impact in REF 2021.

This overlay was extended further to include the SDG classification by the underpinning research and grant funding. Based on these individual aspects of the impact case studies we have been able to highlight different facets of societal impact as submitted to REF2021. This goes some way to providing insights on how societal impact can be assessed in an appropriate manner thereby going some way to addressing today's global challenges.

Open science practices

The research presented in this paper is based on openly available data (REF impact case study database at https://impact.ref.ac.uk/casestudies/ ) The application and use of the SDG classification (ML generated classification) is proprietary and part of Dimensions database offering by Digital Science. The results of the SDG classification for publications is openly available in the free version of Dimensions. The results of the classification in this study will be made available in csv format.

Acknowledgments

The authors would like to thank Lezan Hawizy for the discussion and technical support in applying the SDG classification to the REF impact case study narratives.

Author contributions

The methodology was developed by JW. The extraction of the REFimpact case study dataset, handling the dataset in GBQ was carried out by AC. The analysis and preparation of figures contained in this paper was carried out by AC and the draft of the paper was developed by BF. All co-authors collaborated to edit and finalise the manuscript.

Competing interests

The authors are employees of Digital Science, the owner and operator of Dimensions.

References

The SDGs wedding cake: accessed and retrieved and downloaded 20/04/2023 https://www.stockholmresilience.org/research/research-news/2016-06-14-how-food-connects-all-the-sdgs.html

Reed, M.S., Ferré M., Martin-Ortega, J.,. Blanche, R., Lawford-Rolfe R. , Dallimer, M., & Holden, J. (2021). Evaluating impact from research: A methodological framework. Research Policy, 50, Issue 4.

https://doi.org/10.1016/j.respol.2020.104147

van den Akker, W.,Spaapen, J., & Maies, K. (2017) Productive interactions: societal impact of academic research in the knowledge society. Position paper. https://www.leru.org/publications/productive-interactions-societal-impact-of-academic-research-in-the-knowledge-society

Fecher B, Kuper F, Sokolovska N, Fenton A, Hornbostel S and Wagner GG (2021) Understanding the Societal Impact of the Social Sciences and Humanities: Remarks on Roles, Challenges, and Expectations. Front. Res. Metr. Anal. 6:696804.

doi:10.3389/frma.2021.696804

Pedersen DB, Grønvad, J.F., & Hvidtfeldt, R. (2020). Methods for mapping the impact of social sciences and humanities—A literature review. Research Evaluation 29(1):4-21

DOI:10.1093/reseval/rvz033

The Ludwig Boltzmann Gesellschaft. What is Societal Impact of Research? A literature Review. (2021)

Excellence in Research for Australia. (2022) https://www.arc.gov.au/evaluating-research/excellence-research-australia

Standard Evaluation Protocol 2015 – 2021. Protocol for Research Assessments in the Netherlands. (2016) . https://www.universiteitenvannederland.nl/files/documenten/Domeinen/Onderzoek/SEP2015-2021.pdf

Wastl, J., Hook, D W., Fane, B., Draux, H., & Porter, S. (2020) Contextualising Sustainable Development Research. figshare https://doi.org/10.6084/m9.figshare.12200081.v2