Commissioned public R&D publications in the social sciences. A document analysis of societal impacts of research

This paper investigates the impact of social sciences, a recent field for considering the societal impacts of science. It is based on a purposeful selection of R&D publications, commissioned by Norwegian ministries, governmental agencies, and public bodies, aimed for the application for public sector. The concept societal impact is here defined as active and potential influential, and interactions between public research organisations (PRO) and governmental agencies and other public bodies in the areas of education and social welfare. We argue that the interactions lead to favourable outputs. We examine interactions, i.e., exchanges between researchers and stakeholders from governmental agencies and public bodies, in which scientifically robust and societally relevant knowledge is constructed and valued. In investigating interactions, we focus on qualitative and quantitative indicators of direct social interactions and indirect interactions through texts (see e.g., Spaape & van Drooge, 2011).


Introduction
More than ever public funding institutions are aware of their obligation to justify public investment in research and development (R&D), and public research funding is linked to both, academic impact and to designated societal returns.
Since 2000, aside from the emphasis put on academic impact, like high scientific productivity and excellence, research policy in the European Union has increasingly addressed the value of societal impact (Bornmann, 2013: 218-220). Distinctions between different types of societal impact have been introduced such as: […] "societal products (outputs), societal use (societal references), and societal benefits (changes in society)" (Bornmann, 2013: 217). Focussing on societal use, we address the following research questions: How are interactions between researchers and stakeholders from governmental agencies and public bodies reflected in commissioned public R&D publications in social sciences, and which indicators are used to measure impacts?
The increased weight given to societal use and impact is concurrent with the development of evidence-based policy (e.g., Young et al. 2002) a policy originated in evidence-based medicine (see Guyatt et al., 1992). Independently and in combination the two trends, societal impact and use, and evidence-based policy, have gradually affected a range of research areas comprising economic and environmental studies (e.g., Garnett et al., 2019;Heringa et al., 2016), Science, Technology, and Innovation (STI-programmes) and in education and social welfare (Slavin, 2002;Lingard, 2013;Krejsler, 2013).
Since 2010 there has been an increase in the inclusion of societal impact, also in research evaluations in social sciences and humanities (Forskningsrådet 2017, RCN 2018, RCN 2021. This paper concentrates on social sciences as a recent field for studying societal impact (e.g., Lauronen, 2022).
The study addresses societal use as a dimension of societal impact in a Nordic context with a particular focus on the social sciences in Norway, a country with a longstanding tradition with commissioned research, primarily done by the government, governmental agencies and other public, but also by the non-profit private sector (Brandt et al. 2019). This analysis follows up Lauronen's article (2022). We argue that a Nordic perspective on assumed productive interactions between researchers and stakeholders can provide an insight into intended applications of social impact in a political system different from systems with more rigorous, formal and often linear conceptions of social impact evaluations. These types of evaluations may have direct consequences for institutional/research funding. One example is the Research Excellence Framework (REF) in the UK with consequences for funding.
Our aim is to analyse a purposefully collected sample of R&D publications, all of them results of commissioned projects conducted in Norway, and consequently carried out with designated intentions of societal impact. These projects were funded by the government, governmental agencies, and public bodies. In investigating commissioned research which addresses societal challenges, we examine how R&D publications express societal use focussing on direct and indirect types of "productive interactions" (Sivertsen & & Meijer, 2020).

Method and data
Drawing on carefully selected cases in social sciences we investigate interactions between researchers and stakeholder groups.
Our analysis combines alternative metrics (altmetrics) and quantitative bibliometric indicators. We apply the following indicators: amounts of personal interactions (e.g., meetings, workshops), frequency of downloaded reports and number of citations in scholarly and non-scholarly literature. Non-scholarly literature comprises green papers, white papers, other relevant governmental reports, and policy documents.
For social sciences and humanities (SSH), non-scholarly literature, and literature in other languages than English, coverage is relatively low in highly indexed international databases, like Web of Sciences (WoS) (e.g., Aksnes & Sivertsen, 2019). In this paper, we use Research Gate, a social networking site for academics which comprises altmetrics and some bibliometric information.
Our approach is informed by document analysis (e.g., Prior, 2016). First, we select cases strategically, comprising R&D publications, commissioned by ministries, other governmental and public bodies. We sample documents that represent variations across disciplines and commissioning bodies. Second, we classify selected documents according to type(s) of indicated productive interactions and societal impact, commissioned body, and scope. Third, we conduct an in-depth, thematic analysis of the selected documents, defined as cases for the study.

Preliminary findings
In this paper, we compare and examine the main findings across the cases. In doing so, we draw on findings of the international literature on societal impact with the field of social sciences (see Table 1).
Case 1: Social sciences, technology, and innovation represents one, interdisciplinary case across the fields of social sciences, technology, and innovation, and includes two longitudinal R&D projects (1A and 1B), each with a three-year duration.
The first project (1A) was commissioned by a municipality. It was an evaluation, designed as a formative dialogue design study. The background was a municipality that had implemented one-to-one digital devices for all students in all its twenty-four primary and lower secondary schools. The pupils were provided with Chromebooks. The evaluation done on the implementation covered many different aspects of the implementation process of digital devices: such as teaching and learning with digital devices, competence development, institutional learning, and development of digital competence. The formative evaluation design implied close involvement across researchers and stakeholders. The researchers provided interactions as feedback to the g municipality. Project findings were continuously disseminated to stakeholders that were part of the initiative. These included status-meetings with the municipalities' coordinator of the initiative, meetings with the head of the municipality for education and sharing of findings with teachers, principals, and parents (use of research).
In this paper, we analyse summative elements using altmetrics and bibliometric indicators (number of citations) in addition to reported outcomes (qualitative indicators), such as changed practices.
The second, (ongoing) project (1B), is called "Innovation Project for the Public Sector", commissioned by the Norwegian Research Council. The aim is to pilot and implement a new model to develop leadership and organisation of a pedagogical development program and is closely related to the one-to-one implementation of Chromebooks in compulsory education. In addressing societal impact, the project aims at leading "to innovation in public sector services […] and generate research activities that will promote innovation within the sector and sustainable value creation for its users." Innovations is here defined as: "new or significantly improved […] processes […] that are introduced to enhance value creation and for the benefit of society." The project draws on the assumption that experiences from municipalities are valuable for other municipalities that have introduced one-to-one implementation. The design comprises communication and interaction strategies with stakeholders to ensure dissemination to the school sector in Norway This R&D project is supposed to contribute with new findings on pedagogical development in these schools, especially addressing school owner and principals.
In this paper, we use altmetrics and citation indicators in our analysis, in addition to qualitative indicators of societal use in short-medium term.
Case 2: Social sciences -interdisciplinary encompass three related projects (five reports) that covering the scholarly literature on discrimination, harassment and racism, and its consequences. The five reports applied methods of mapping and systematic reviews. They were commissioned by three governmental agencies: the Research Council of Norway, The Norwegian Directorate for Children, Youth and Family Affairs (Bufdir) and The Directorate of Integration and Diversity.
The aim of report 2A, a mapping review of Nordic research on discrimination, harassment, and equality, commissioned by the Research Council, was to inform the Ministry's work with an intersectoral R&D-strategy on equality and non-discrimination. To enhance interactions during the review process, a reference group across different governmental agencies was established.
Further, two projects applying systematic review methods and full-text articles address more specific research questions on consequences of discrimination and racism. The aim of project 2.B was to summarise the literature on consequences of racism and discrimination because of ethnicity, religion, and philosophy of life and to identify knowledge gaps. The aim of project 2.C was to summarise the literature dealing with consequences of racism and discrimination for work-life integration (2.C.1), educational integration (2.C.2) and integration in civil society (2.C.3).
In this paper, we combine altmetrics and citation indicators in our analysis, in addition to qualitative indicators of societal use in short-medium term.

Discussion and conclusions
The assessment of societal impact of research is a growing topic, which includes the social sciences standing out as the field with the highest research production (Viana-Lora et al., 2021).
Citation indicators are in general used to measure scientific impact and relevance, but they are difficult to use to assess societal impact and productive interactions between researchers, and different stakeholder groups. A reason is that the literature indexed databases like Web of Sciences mostly comprises scientific publications (Aksnes, Langfeldt & Wouters, 2019) and neglects commissioned R&D publications.
Focussing on societal use we examine how interactions between researchers and stakeholders are reflected in commissioned public R&D publications in social sciences. Given distinctions between different types of societal impact, we elaborate societal uses (Bornmann, 2013) of R&D publications in short-and medium-terms by combining qualitative and quantitative indicators. To capture societal uses, we investigate close interactions between different stakeholders and researchers, e.g., captured by communications of findings, and activities facilitating a responsible use of research for policy making. We suggest further research to explore societal benefits, i.e., changes in society (Bornmann, 2013: 217), applying a multi-methods approach and taking into consideration a longitudinal perspective with a longer timeframe than the actual project period.