Do they know it’s impact time at all? How researchers outline societal impact in research council applications

Magnus Gulbrandsen^*, Silje Tellmann^**, Gemma Derrick^***, and Knut Jørgen Vie^****

^*magnus.gulbrandsen@tik.uio.no

ORCID: 0000-0001-9976-1608

^**s.m.tellmann@tik.uio.no

ORCID: 0000-0002-4659-281X

^****k.j.vie@tik.uio.no

ORCID: 0000-0002-8228-6078

TIK Centre for Technology, Innovation and Culture, University of Oslo, Norway

^***gemma.derrick@bristol.ac.uk

ORCID: 0000-0001-5386-8653

TIK Centre for Technology, Innovation and Culture, University of Oslo, Norway
and Centre for Higher Education Transformations, University of Bristol, UK.

Abstract

Societal impact has turned into an important criterion for evaluating research funding proposals, yet a universal understanding about how to approach ex-ante impact evaluation is yet to materialise. Moreover, the researchers’ own perspectives on how to express the societal impact of research in the context of applying for funding is largely unexplored. In this paper, we seek to contribute to the understanding and practice of ex-ante impact evaluation by analysing in detail the impact section of research proposals (N=379) submitted to the Research Council of Norway (RCN) and their associated evaluation reports. The study is based on a mixed-methods analysis of proposals submitted to three different research fields (health, education and marine) and two different calls (basic versus applied) in each of them. In this way, we are able to isolate factors associated with disciplinary differences and competing signals within the science system between the concepts of ‘societal’ versus ‘scientific’ impact.

1. Introduction

Society has always expected research to be useful and beneficial, which is mostly referred to as impact in contemporary science policy. Impact has become a key concept in research evaluation, and the burgeoning impact literature has studied how this can be assessed after the societal effects are obvious – ex-post. Nuanced and sophisticated empirical investigations have emerged, where the REF evaluation system from the UK has been a major source of data. Studies have found that impact is a very heterogeneous concept that can be tied to many different sectors or recipients, timelines, processes and visibility (e.g. Derrick 2018; Sivertsen & Meijer 2020; Langfeldt et al. 2020). Experts who assess impact cases have widely different perspectives on what it is, with little agreement (Derrick & Samuel 2016), and the disciplinary differences are substantial.

Impact has also turned into an important criterion for evaluating research funding proposals, which has led to an increase in the corresponding ex-ante impact literature the last decade. Predicting or anticipating the likely future impacts as an evaluative criterion at the early stage of the research process is associated with a level of uncertainty and decision-making requiring a parallel assessment of risk. The majority of present studies on this decision-making process is carried out in the assessment of competitive funding applications. The criteria and assessment procedures differ not just between funding organisations and national contexts, but also within the same organisation (Holbrook & Frodeman 2011), adding further complexity to moves to make the process transparent and accountable to applicants. Furthermore, this complexity if also evident in how reviewers tend to prefer short-term and tangible forms of impact within applications, blending considerations of academic and societal effects (Ma et al. 2020), often referred to as ‘bleeding of criteria’ (Derrick & Benneworth, 2020). Using a mix of theoretical perspectives and empirical findings, several authors have made suggestions about how ex-ante impact evaluation may be improved (Ma & Agnew 2021; Benneworth & Olmos-Penuela 2022), but a universal understanding about how to approach ex-ante impact evaluation is yet to materialise.

Although the literature has made important contributions to how impact can be understood and assessed, the researchers’ own perspectives – when they need to express how their activities will generate societal effects in the context of applying for funding – is largely absent. How do they portray the process through which the research will generate an impact? How do they communicate the societal benefits of their anticipated research within applications? What do they see as the main barriers in this process and as the main partners and recipients? These are the key research questions in our paper.

We seek to contribute to the understanding and practice of ex-ante impact evaluation by analysing in detail the impact section of research proposals submitted to the Research Council of Norway (RCN). We have selected three different research fields and two different calls in each of them, based on an assumption that there are likely disciplinary differences and differences between calls for basic research and calls tied to applied themes and challenges.

Our work builds on a study of the impact sections of RCN proposals as narratives that signal a proposal’s credibility and an applicant’s trustworthiness (Benneworth & Derrick 2020). In the current analysis (which is ongoing and will be finished by the end of summer 2023), we include a wider set of proposals and look also at the details about impact such as what, where, when and for whom. We additionally include the texts concerning ethics in the proposals in the study to investigate how ethics and impact are intertwined within proposals both a matter of preventing negative impact (e.g. preventing harm to research subjects, or to society and nature) and working towards making the world a better place in the form of (positive) impact. Including ethics as a part of the analysis builds on the novel and critical concept grimpact (Derrick et al. 2018), which opens for studying impact as something which is not inherently positive, but that can also have problematic, ambiguous or contested aspects. By including ethics in the study, we aim at expanding the notion of impact by studying self-reported potentially problematic aspects of research, along with the measures researchers state that they will implement to mitigate the risk of grimpact.

Our methodological approach is primarily inductive, and we will score various dimensions that can be compared to the official reviewer scores, in order to identify patterns in impact narrative constructions, as well as how these are valued by proposal reviewers and in different disciplinary contexts.

2. Earlier research

Societal outcomes of research have been an implicit and sometimes explicit theme in many different research traditions, from bibliometrics to economics. Impact is a newer concept and is largely tied to the emergence of the large-scale and influential Research Excellence Framework (REF) evaluation in the UK and its emphasis and scoring of ‘impact cases’. The concept of impact, however, has been an implicit evaluative threshold within competitive funding agencies in response to global movements for research to be more accountable to a society that funds it. As a result, the concept and its resulting science policy messaging – societal effects are important and should be encouraged in funding arrangements – are now found globally with researchers being asked to identify, plan and communicate how their research can/might/should generate impact, not only its potential relevance to specific sectors and actors. Experts who assess scientific excellence and implementation (the qualities of the researcher and the research unit) must now also assess the research’s potential to contribute to societal change, even if their ability to do so can be critically questioned (Holbrook & Frodeman 2011), as well as their position as an expert to assess it.

Creating sufficient capacity to evaluate impact, not only requires researchers and expert evaluators to acquire new skills, but to also challenge existing knowledge and understanding of how research practices can stimulate societal changes given highly variable contextual conditions. There is considerable uncertainty for applicants involved in planning to achieve social impact as well as for reviewers at the early research stages. This can occur because of planned processes, but just as often, impact processes can be characterized by coincidences and unplanned, serendipitous events. Some types of impact may be somewhat easier to control and therefore predict (‘straight runs’), others may be ‘long shots’ and depend on circumstances that are beyond the researchers’ control (cf. Reale et al. 2014). As such, there has been a tendency to use ‘interactions’ or a measure of public engagement as proxies for impact that while providing important markers for evaluating ex-ante impact, are not always necessary nor reliable for predicting future research impact outcomes.

In addition, both researchers, expert evaluators and research funders may suffer from biases when describing, assessing, and making procedures for achieving impact. In funding arrangements, they are likely to prioritise shorter-term and tangible impacts, often in the form of something that has commercial or otherwise (seemingly) easily measurable effects (Ma et al. 2020). Ex-post evaluations, on the other hand, tend to favour extraordinary examples that are hard to replicate (Sivertsen & Meijer 2018). Stakeholders are furthermore likely to mix academic and societal impact, even if these are very different phenomena (Derrick & Samuel 2016; Ma et al. 2020). There is a tendency that evaluators rarely move beyond characteristics of the research activities themselves to understand how impact happens, even if this may depend upon many different societal characteristics (Ma & Agnew 2021). To mitigate these biases, it has been suggested that evaluations of proposals should put more weight on activities that tie the research to non-academic users, and the reciprocity and consistency of such activities throughout the project under assessment (Benneworth & Olmos-Peñuela 2022). More generally, the selection of proposals is a delegated decision-making process where a set of reviewers and/or a panel, on behalf of a research funding organisation, gives a score to applicants. The criteria of the research funder need to be interpreted both by the reviewers and the applicants. This relationship has been analysed with the term ‘policy alienation’, because in many cases researchers neither understand what the funder means by ‘impact’ nor how it will be evaluated (de Jong et al. 2016).

Further understanding the level of uncertainty in ex-ante impact evaluation processes has been limited by the extend of data poverty driven by the reluctance of funders to allow research access to applicant information, evaluation scores and evaluator comments on funding applications (both successful and unsuccessful). As such, there are relatively few studies where research funding proposals constitute the main data source, and we have found no investigations of this type that directly address societal impact. In an analysis of proposals to the U.S. National Institutes of Health, it was found that proposal success largely could be explained by the quality of writing (Boyack et al. 2018), and a study by Cotos (2020) explored the linguistic movements used by applicants within the NSF Broader Impacts category. Within Europe, another investigation used proposals to the European Research Council to determine how these texts could be used to better understand ‘frontier science’ (Hörlesberger et al. 2013), but no study as yet has explored the relationship between funding success and aspects of societal impact within funding proposals. The closest is the exploratory studies by Benneworth and Derrick (2020) and Derrick (2021) upon which this paper builds. In both studies, the authors used a narrative analysis to address uncertainty – about whether the impact will happen (tied to the research itself or other factors – and risk tied, for example, to unintended impacts. This study of the proposal components are matched with observations of evaluation panels to explore how these components influence panel deliberations and evaluation. Narratives of impact were seen as credibility markers tied to trust, and with different types of ‘qualifiers’ used to express risk and uncertainty. Findings are in line with other studies of impact, such as a blending of criteria of academic and non-academic impact (Ma et al, 2020). Through the corresponding panel observations, panels/reviewer did not distinguish between constructive and poor impact statements, and the assessment of risks versus benefits was included as a qualifier alongside more traditional assessments of track record. These studies indicate that there is more to be gained through training of panel reviewers and clearer procedures for evaluation for applicants.

In this paper based on a new and larger set of proposals, we expect to find a large variation in impact statements, for example in who the stakeholders are, what types of societal effects that are envisioned, the role of the research in generating the impact, and whether this is a normal or extraordinary outcome. We also expect that the impact statements will be more tangible in proposals tied to calls for specific societal challenges rather than general announcements of funding for research. The analysis will be tied to reviewer scores and information about panel decisions, which constitutes a more general contribution to the literature on impact evaluation and indicators.

3. Data and methods

The study is based on a textual analysis of proposals with associated evaluation reports from 6 different calls. The calls (n=6) were selected in order to include variety regarding likely disciplinary differences and differences between basic research and applied themes and challenges. We have firstly selected calls tied to three research fields associated with different disciplines (health, education and marine). Secondly, within each of these fields, we have included proposals submitted to calls for basic research (‘Researcher project’) and calls for more societally oriented projects (‘Collaborative project’). The call for ‘researcher projects’ is targeted towards research of high scientific quality and is expected to advance the international research front. The purpose of ‘collaborative projects’ is on the other hand to address stated societal challenges and to support collaboration between research and societal stakeholders.

Through the selection of calls, we are able to isolate factors associated with disciplinary differences and competing signals within the science system between the concepts of ‘societal’ versus ‘scientific’ impact. In total, 379 proposals were submitted to the calls, and the distribution of proposals submitted to the different calls is shown in table 1.

Table 1: Distribution of proposals

All proposals are written over a similar three-part template where the first part shall demonstrate the excellence of the proposed project, the second part the impact of the proposed project, and the third part demonstrates how the project will be implemented. Similarly, the reports of the evaluators are divided into four sections to mirror the structure of the proposals. In addition to an overall review and score, they provide a review/score for excellence, impact and implementation respectively. While our analysis will focus on the impact part of the proposals, we will also look to the other parts if necessary.

The analyses of the proposals will be based on a mixed methods approach. We will do a qualitative textual analysis of the impact statements in the proposals. This analysis will, on the one hand, include a narrative analysis of the statements, in order to identify how the researchers construct impact narratives using specific qualifiers, inspirations and promises to convince the evaluators. On the other hand, we will investigate how the researchers envision their research to have an impact by identifying the combination of impact objectives, impact pathways and stakeholders included in the impact statements. Moreover, we will look to how they address ethics. This will be translated into a descriptive score. Secondly, we will link these scores to the scores awarded to the proposal by the panel of evaluators to run a factor analysis of how evaluators value different impact statements. Finally, we will account for whether, and if so how, differences between disciplines and the orientation of the call play out in the analyses.

4. Preliminary findings

Findings are ongoing and will be presented as part of an oral presentation during the conference.

Drawing from the study of Derrick & Benneworth (2020), preliminary findings identified similar components of Impact sections; Qualifiers, Inspirations and Promises. Inspirations were linguistic or else illustrative tools designed to evoke a wider non-academic relevance to the project without necessarily linking directly to the project itself. Promises, on the other hand were claims that the research would have impacts for specific users that did not allow for the possibilities that they would not be delivered; and Qualifiers were additional claims that grounded inspirations into contexts and softened promises as aspirations to be worked towards with the necessary capacity to deliver that.

Within the proposals, the lack of impact plan (<10% of proposals studied) – where impact activities and interactions are used as tools to build credibility in the proposal that is seen to decrease the risk of the proposal to reach promised impacts (promises) – was a tool that was not widely used in the majority of proposals studied. Further research will examine if the inclusion of these components contribute to the overall score received for the impact section, as well as examine how the minority of proposals that included an impact plan.

Open science practices

Proposals for research funding and the associated evaluation reports are protected by strict rules for confidentiality related to both personal data and business secrets. The authors were granted access to the data by the Norwegian Ministry of Education and Research after signing a non-disclosure agreement and are therefore not able to make the data openly available.

Competing interests

The authors declare no competing interests in this research.

Funding information

The research is carried out within the OSIRIS project, funded by the Research Council of Norway’s Grant Number 256240.

5. References

Benneworth, Paul & Derrick, Gemma (2020). Guidelines for the peer review of ex-ante impact evaluation. Report commissioned and delivered to the Research Council of Norway (Forskningsrådet).

Benneworth, Paul & Olmos-Peñuela, Julia (2022). An openness framework for ex ante evaluation of societal impact of research. Research Evaluation, 1-12. http://dx.doi.org/10.1093/reseval/rvac023

Boyack, Kevin, Smith, Caleb & Klavans, Richard. (2018). Toward predicting research proposal success. Scientometrics, 114, 449–461. https://doi.org/10.1007/s11192-017-2609-2

de Jong Stefan P L & Reetta Muhonen (2020) Who benefits from ex ante societal impact evaluation in the European funding arena? A cross-country comparison of societal impact capacity in the social sciences and humanities, Research Evaluation, 29, (22–33), https://doi.org/10.1093/reseval/rvy036

Derrick, Gemma E., et al. (2018) "Towards characterising negative impact: Introducing Grimpact." 23rd International Conference on Science and Technology Indicators. Leiden University, CWTS, 2018.

Derrick, Gemma. (2021) Evaluating the evaluation of ex-ante impact through panel deliberations. Report commissioned and delivered to the Research Council of Norway. (Forskningsrådet)

Derrick, Gemma. (2018) The Evaluators’ Eye: Impact assessment and academic peer review. Palgrave Macmillan, London, UK.

Derrick, Gemma & Samuel, Gabrielle (2016) The evaluation scale: Exploring decisions about societal impact in peer review panels. Minerva, 54, (75-97).

Holbrook, J Britt & Frodeman, Robert (2011) Peer review and the ex ante assessment of societal impacts, Research Evaluation, 20, (239–246), https://doi.org/10.3152/095820211X12941371876788

Hörlesberger, M., Roche, I., Besagni, D. et al. (2013) A concept for inferring ‘frontier research’ in grant proposals. Scientometrics, 97, 129–148. https://doi.org/10.1007/s11192-013-1008-6

Langfeldt L, Nedeva M, Sörlin S, et al. (2020) Co-existing Notions of Research Quality: A Framework to Study Context-specific Understandings of Good Research. Minerva 58, (115-137).

Ma, Lai & Agnew, Rachael (2022) Deconstructing impact: A framework for impact evaluation in grant applications, Science and Public Policy, 49, (289–301), https://doi.org/10.1093/scipol/scab080

Reale, E., Nedeva, M., Thomas, D., & Primeri, E. (2014). Evaluation through impact: A different viewpoint. Fteval Journal, 39, (36-41(.

Sivertsen, Gunnar & Meijer, Ingeborg (2020). Normal versus extraordinary societal impact:

how to understand, evaluate, and improve research activities in their relations to society? Research Evaluation, 29, (66–70) doi: 10.1093/reseval/rvz032