Commissioned public R&D projects in the social sciences. A document analysis of societal impacts of research

Topics: Societal impact of research, science policy, altmetrics, bibliometrics

Sabine Wollscheid^* and Vera Schwach^**

^*sabine.wollscheid@nifu.no

ORCID https://orcid.org/0000-0001-7376-9820

Nordic Institute for Studies in Research Innovation and Education (NIFU), Norway

^** vera.schwach@nifu.no

ORCID https://orcid.org/0009-0000-4686-7599

Nordic Institute for Studies in Research Innovation and Education (NIFU), Norway

Abstract (149 words)

This paper investigates the societal impact of social sciences, a recent field in the study of science's social impacts. Our study draws on a purposeful selection of R&D publications, commissioned by Norwegian ministries, governmental agencies, and public bodies, all aimed for application in the public sector. The concept societal impact is here defined as active and potential influential interactions between public research organisations (PRO) and governmental agencies as well as other public bodies in education and social welfare. The paper examines interactions between researchers and stakeholder from ministries, governmental agencies, and other public bodies in the fields of education and social welfare. The study’s emphasis is on processes where scientifically robust and societal relevant knowledge is constructed and valued. The study focuses mainly on direct output indicators – citations and number of publications, but it does also discuss the value of indirect indicators and their potential implications for further research.

1. Introduction

Since 2000 research policy in the European Union has increasingly addressed the value of societal impact (Bornmann, 2013). The increased weight put on societal impact is concurrent with the development of evidence-based policy (Young et al. 2002). Independently and in combination, societal impact and evidence-based policy, have gradually affected a range of research areas comprising Science, Technology and Innovation (STI) studies, and education and social welfare (Slavin, 2002; Lingard, 2013; Krejsler, 2013).

An indication of its importance is how the question of impact has been included in national evaluations of research and higher education in the social sciences, and in humanities (cf. Forskningsrådet 2017; RCN 2017, RCN, 2018; RCN 2021). This paper concentrates on social sciences - a relatively recent field for examining societal impact (Spaapen & Van Drooge, 2011; Bornmann, 2013; Lauronen, 2022

The assessment of societal impact of research is a growing topic and includes the social sciences, (Spaapen & Van Drooge, 2011; Fecher et al., 2021; Viana-Lora et al., 2021). Still, for three reasons is not straight forward to investigate societal impacts here.

First, direct output indicators, such as citation counts and number of publications, are generally used to measure scientific production and impact, but they have methodological limitations for the social sciences with different publication patterns than Natural Sciences. Scientific literature indexed databases, like Web of Sciences (WoS), mostly comprise direct output indicators of journal publications in Natural Sciences and Medicine (Aksnes, Langfeldt & Wouters, 2019).

Second, such large, indexed databases also exclude other output indicators like disseminations of preliminary results (presentations) and commissioned R&D reports, which makes it difficult to fully capture societal and research impact of these publications. Third, direct output indicators can partly be inadequate to assess impact and productive interactions in social sciences.

Societal impact is an ambiguous concept (Muhonen & Tellmann, 2023). We follow a distinction between different types, as introduced by Bornmann (2013: 2017): “societal products (outputs), societal use (societal references), and societal benefits (changes in society).”

In exploring societal impact, we emphasis the dimension societal use as one aspect of impact. The study is set in a Nordic context and has a particular attention on Norway. In doing this we follow up Lauronen’s article about the Nordic countries, especially Finland (2022), but with a different country focus. All the Nordic countries, among them Norway, are marked by a long- standing political tradition in which commissioned research, primarily done by the government, governmental agencies, other public organisations, and the non-profit private sector (Brandt et al., 2019). We argue that a Nordic perspective on assumed productive interactions between researchers and stakeholders can provide a broad insight into intended applications of social impact, in a political system different from systems with more rigorous, formal models for conceptions of social impact evaluations, such as the Research Excellence Framework in the UK¹.

We analyse a selected sample of R&D publications, all of them outcome of commissioned projects done in Norway; The projects were funded by the government, governmental agencies, and public bodies. Thus, the research was carried out with intended and designated intentions of societal impact.

We investigate how interactions between researchers and stakeholders from governmental agencies and public bodies, are reflected in public commissioned projects, both during the research process and its results, the R&D publications, all published with open, free public access.

2. Method and data

This is a case study design, an empirical inquiry investigating of a phenomenon in depth by combining different sources and methods. The methodological scope is to examine direct indicators of societal impact and use, but also to consider and value indirect indicators.

Thus, we examine indicators suitable to measure social impacts, primarily in a direct way. Direct indicators measure project outputs quantitatively, such as citations or as oral and written dissemination of the results of the project.

2.1 Case selection

From the wide range of commissioned public R&D projects conducted at NIFU, we selected two cases. The criteria were to include projects with diverse topics and thematic focus within the societal area of education and welfare; the projects inquired should involve various stakeholders; furthermore, we aimed for projects with different methodological approach. The selection should enable an examination of different indicators to measure social use.

Case 1, Social sciences, technology, and innovation represents one, interdisciplinary case across the fields of social sciences, technology, and innovation, and includes two longitudinal R&D projects (1A and 1B), each with a three-year duration. Applying a formative evaluation design, case 1 continuously addresses “inter-actions between researchers and other stakeholders as commissioners and practitioners in one single municipality. Case 1 is longitudinal, and the two projects (1A and 1B) build up on each other. The second project is directly led by the commissioned unit and researchers were “invited” into the project. In contrast, Case 2 more indirectly addresses interactions between researchers and commissioner, a governmental unit. Case 2 contains three projects, 2A, 2B and 2C).

Case 1:

Project 1A, entitled “Digital Learning in Asker School, 2017–2019”, was commissioned by a municipality. It was an evaluation on the implementation process of digital devices. The project was designed as a formative dialogue design study. The background was a municipality that had implemented one-to-one digital devices for all pupils in all its twenty-four primary and lower secondary schools. They were provided with Chromebooks. The evaluation covered many different aspects of the implementation process of digital devices: teaching and learning with digital devices, institutional learning, and the development of digital competence. The formative evaluation design implied close involvement across researchers and stakeholders. The researchers provided interactions as feedback to the municipality. Project findings were continuously disseminated to the stakeholders that had been part of the initiative. The feedback included status-meetings with the municipalities’ coordinator of the initiative, meetings with the head of the municipality for education and sharing of findings with teachers, principals, and parents (use of research).

Project 1B, entitled “Innovation Project for the Public Sector”, was commissioned by the Norwegian Research Council. The overall purpose of the R&D project was to contribute through new findings to pedagogical development in the schools, especially addressing school owner and principals. Its aim was to pilot and implement a new model to develop leadership and organisation of a pedagogical development program; it was closely related to the one-to-one implementation of Chromebooks in compulsory education (cf. project 1A). In addressing societal impact, the project aimed to lead “to innovation in public sector services […] and generate research activities that will promote innovation within the sector and sustainable value creation for its users.” (Norwegian Research Council).² Innovations in this context is defined as: “new or significantly improved […] processes […] that are introduced to enhance value creation and for the benefit of society.” (Norwegian Research Council).³ The project drew on the assumption that experiences done in one municipality are valuable for other municipalities which also had introduced one-to-one implementation of digital devices. The design comprised communication and interaction strategies with stakeholders, this to ensure dissemination to the school sector in Norway.

Case 2: Social sciences – interdisciplinary encompass three related projects (five reports) covering the scholarly literature on discrimination, harassment, and racism, and its consequences. They were commissioned by three different governmental agencies.

Case 2 resulted in five reports, all of them applied methods of mapping and systematic reviews.

The aim of project 2A, a mapping review of Nordic research on discrimination, harassment, and equality, was to inform the Ministry’s work with an intersectoral R&D-strategy on equality and non-discrimination. To enhance interactions during the review process, a reference group across different governmental agencies was established.

Further, two systematic review projects address more specific research questions on consequences of discrimination and racism. Project 2B aimed at summarising scientific and scholarly literature on consequences of racism and discrimination caused by ethnicity, religion, and/or life perspective and to identify knowledge gaps. Project 2C summarised scientific and scholarly literature dealing with consequences of racism and discrimination for work-life integration (2.C.1), educational integration (2.C.2) and integration in civil society (2.C.3).

Table 1: Description of the two cases with its projects

2.2. Operationalization

Drawing on two selected cases in the social sciences, cases with projects that represent variation in methods, theme, and stakeholders involved, we investigate interactions between researchers and defined stakeholder groups with the purpose to capture indications of social impact and use according to research process and output.

Table 2: Direct and indirect indicators to capture social impact and use

Direct output indicators cover summative elements such as bibliometric indicators and altmetrics. Direct process indicators cover measurable interactions between researchers and other stakeholder (commissioners), and possible changes during the research process. Indirect output indicators comprise e.g., new politics and policies in consequence of research. Given limitations in scope, we do not study indirect process indicators in depth, but provide some implications for further research.

2.3. Analysis

Here descriptive bibliometric analysis cover directs measures of publication counts and number of citations and downloads (altmetrics).

For measuring research output directly and related research impact, we use number of publications (reports, journal articles) and number of citations. For capturing social impact and use, we use number of downloads/reads of publications registered in Research Gate and number of social interactions such as meetings, workshops etc. Data were collected via core documents (Table 2).

For social sciences and humanities (SSH), non-scholarly literature, and literature in other languages than English, coverage is relatively low in highly indexed international databases, like WoS covering extensive information on bibliometric indicators (Aksnes & Sivertsen, 2019) and altmetrics. We therefore use Research Gate, a social networking site for academics comprising altmetrics, and Google Scholar and Research gate for some bibliometric information.

To capture process indicators, we apply an analysis of core documents (e.g., Prior, 2016). We conduct thematic analysis of selected documents, covering minutes of status meetings, immediate report etc.

3. Findings

We compare and examine the two cases mapping and comparing direct process and output indicators of social impact and its results.

Table 3 shows a greater variation in direct process and output indicators for Case 1 compared to Case 2. This variety in indicators is also reflected by a greater diversity and complexity in study design and data collection, comprising two succeeding formative evaluation projects over the course of six years. Direct output indicators comprise the number of a range of scholarly publications and master thesis, in addition to reports, number of downloads and citations and conference presentations (of the final report). Similarly, direct process indicators include different types of interactions between researchers and different stakeholder groups, such as regular meetings, dissemination of findings, network activities with the commissioning municipality etc.

Drawing on different types of review research (Kunisch et al., 2013), a kind of secondary research, Case 2 can be characterized as more homogeneous in terms of study design, research questions and stakeholder involvement. Direct output indicators cover citation count of final reports, number of publications (mainly research reports), number of downloads, in addition to process indicators covering meetings with commissioners and reference group, etc. While interactions, measured by process indicators, in Case 1 can be described as rather ongoing and processual (e.g., by dialogue, regular meetings), those in Case 2 can be characterized as rather limited to one or two timely limited events or meetings.

Table 3: Direct output and process indicators

¹ Numbers will be updated.

4. Discussion and conclusions

Focussing on societal impact, we examined how interactions between researchers and different stakeholders were reflected in commissioned public R&D projects in social sciences. Our analyses were limited to societal uses (Bornmann, 2013) of R&D projects in short- and medium-terms by combining direct output and process indicators.

Our aim was to analyse a sample of R&D publications, all of them results of commissioned projects conducted in Norway, carried out with designated intentions of societal impact. These projects made of two cases, were funded by the government, governmental agencies, and public bodies. The main finding relates to differences in number and types of direct output and process indicators across the two cases, and its close association to the scope and design of the projects.

While Case 1 applies two projects with a complex and multi-method and -data longitudinal design, Case 2 covers several projects applying a review research approach, pointing back in time. This type of secondary research might create a new level of “distance” between primary research (included), the reviewer and the user of research.

Given limitations in scope, we limited our analyses to direct output and process indicators. However, we identified one indirect indicator of research impact. Both cases included several projects, that build on each other implying that the original project initiated further research that might have further informed policymaking. To provide an example: The mapping review on Nordic research on discrimination, harassment, and equality with the aim to inform the Ministry’s work with an intersectoral R&D-strategy on non-discrimination, led to succeeding projects with more specifically aims comprising three systematic reviews on consequences of discrimination for integration in work-life, education, and civil society, with relevance for evidence-informed policy making.

For further study of indirect output indicators, a longitudinal approach is required to capture new policies in result of the projects. For studying indirect process indicators, richer data collected during the research processes would be required.

For further study of research and policy impact, we recommend the use of alternative data bases, such as Google Scholar and Semantic scholar, that include citations of broader publication sources and citations in other documents than scientific journals.

References

Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. Sage Open, 9(1), 1-17. doi.org/10.1177/2158244019829575

Aksnes, D. W. & Sivertsen, G. (2019). A Criteria-based Assessment of the Coverage of Scopus and Web of Science. Journal of Data and Information Science, 4(1), 1-21. DOI: doi.org/10.2478/jdis-2019-0001

Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey, Journal of the American Society for Information Science and Technology, 64(2): 217–233. doi.org/10.1002/asi.22803

Brandt, T.; Ingulstad, M.; Larsen, E.; Mangset, M. & Schwach, V. (2019). Avhengig av forskning. De norske forskningsrådenes historie 1946–2016 [Dependent on Research. The History of the Research Councils in Norway 1946–2016] (Fagbokforlaget) Trondheim (In Norwegian].

Fecher B, Kuper F, Sokolovska N, Fenton A, Hornbostel S, Wagner GG. Understanding the Societal Impact of the Social Sciences and Humanities: Remarks on Roles, Challenges, and Expectations. Front Res Metr Anal. 2021 Jul 1;6:696804. doi: 10.3389/frma.2021.696804. PMID: 34278206; PMCID: PMC8281339.

Forskningsrådet (2017). Casestudier over effektene av de samfunnsvitenskapelige instituttenes forskning. Vedlegg til evalueringsrapporten. [Case studies on the effect of research conducted by social research institutes. Attachments to evaluation reports]. Forskningsrådet Oslo.

https://www.forskningsradet.no/siteassets/publikasjoner/1254024993876.pdf,

Krejsler, J. B. (2013). What works in education and social welfare? A mapping of the evidence discourse and reflections upon consequences for professionals. Scandinavian Journal of Educational Research, 57(1), 16-32. https://doi.org/10.1080/00313831.2011.621141

Lauronen, J. P. (2022). Tension in Interpretations of the Social Impact of the Social Sciences: Walking a Tightrope Between Divergent Conceptualizations of Research Utilization. SAGE Open, 12(2), 1-16. doi.org/10.1177/21582440221089967

Lingard, B. (2013). The impact of research on education policy in an era of evidence-based policy. Critical Studies in Education, 54(2), 113-131. https://doi.org/10.1080/17508487.2013.781515

Muhonen, R. & Tellmann, S.M. (2023). Challenges of reporting societal impacts for research evaluation purposes – the case of sociology. In: Tim C.E. Engels & Emanuel Kulczycki (eds.). Handbook on Research Assessment in the Social Sciences, Cheltenham, UK, Northampton, MA, USA. Edward Elgar Publishing, pp. 335-349.

RCN (2017). Evaluation of the Humanities umaniin Norway (HUMEVAL). Impact cases. The Research Council of Norway, Oslo, https://www.forskningsradet.no/siteassets/publikasjoner/125402774200.pdf.

RCN (2018). Evaluation of Social Sciences in Norway (SAMEVAL) – Impact cases. The Research Council of Norway, Oslo https://www.forskningsradet.no/siteassets/publikasjoner/1254035787331.pdf

RCN (2021). Evaluation of Legal Research (JUREVAL). Impact cases. The Research Council of Norway, Oslo https://www.forskningsradet.no/siteassets/publikasjoner/2021/jureval-impact.pdf

Prior, L. (2016). Using documents in social research. In D. Silvermann (ed.). Qualitative Research. Los Angeles, London, New Delhi: Sage, pp. 171-185.

Sivertsen, G., & Meijer, I. (2020). Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society? Research Evaluation, 29(1), 66-70. https://doi.org/10.1093/reseval/rvz032

Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational researcher, 31(7), 15-21. https://doi.org/10.3102/0013189X031007015

Spaapen, J. & van Drooge, L. (2011). Introducing ‘productive interactions’ in social impact assessment, Research Evaluation, (20) 3, September 2011, 211–218, doi.org/10.3152/095820211X12941371876742

Viana-Lora, A., & Nel-lo-Andreu, M. G. (2021). Approaching the social impact of research through a literature review. International Journal of Qualitative Methods, 20, 1-11. DOI: 10.1177/16094069211052189

Yin, R. K. (2009). Case study research: Design and methods (Vol. 5). Sage.

Young, K., Ashby, D., Boaz, A., & Grayson, L. (2002). Social science and the evidence-based policy movement. Social policy and society, 1(3), 215-224. DOI: https://doi.org/10.1017/S1474746402003068

Included reports (case studies)

1 A. Tømte, C., Wollscheid, S., Bugge, M., & Vennerød-Diesen, F. F. (2019). Digital læring i askerskolen. Sluttrapport fra følgeforskning. [Digital learning in schools in the Asker municipality. Final report]. 2019:27. Nordic Institute for Studies in Innovation, Research and Education, Oslo. doi.org/10.13140/RG.2.2.23526.98885

https://nifu.brage.unit.no/nifu-xmlui/handle/11250/2631639

2.A. Aksnes, D. W., Bergene, A. C., Fossum, L. W., & Wollscheid, S. (2021). Nordisk forskning om diskriminering, trakassering og likestilling: En forenklet kunnskapsoversikt. [Nordic research on discrimination, harrassment and equality: A scoping review]. 2021:14. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/2830908

2.B. Wollscheid, S., Lynnebakke, B., & Fossum, L. W. (2021). Konsekvenser av rasisme og diskriminering på grunn av etnisitet, religion og livssyn – en kunnskapsoversikt. [Consequences of racism and discrimination due to ethnicity, religion and life perspective – a literature review. 2021:22. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/2978279

2.C.1. Wollscheid, S., Alne, R., Bergene, A. C., & Karlstrøm, H. (2021). Effekter av rasisme og diskriminering på integrering i arbeidslivet – en kunnskapsoversikt. [Consequences of racism and discrimination on integration into working life – a literature review]. 2022:20. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/3026126

2.C.2. Wollscheid, S., Lynnebakke, B., Kindt, M. T., Karlstrøm, H., & Fossum, L. W. (2022). Konsekvenser av rasisme og diskriminering for integrering innen utdanning: En kunnskapsoversikt. [Consequences of racism and discrimination for integration into education – a literature review.] 2022:22. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/3036342

2.C.3. Wollscheid, S., Lynnebakke, B., Bergene, A. C., Karlstrøm, H., & Fossum, L. W. (2022). Konsekvenser av rasisme og diskriminering for deltakelse i sivilsamfunnet: En kunnskapsoversikt. [Consequences of racism and discrimination for participation in civil society: A literature review]. 2022:27. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/3044154

Open science practices

Data in this paper are commissioned R&D reports. These reports are openly accessible online and can be retrieved for example vis Research Gate.

For publication we consider journals with open access policy (gold or green open access).

Author contributions

Sabine Wollscheid: development and draft of the proposal, methodological development and mapping of the two cases.

Vera Schwach: development and draft of the proposal, substantial contributions to main concepts and terminology in the field of research policy.

Competing interests

Authors declare that they have no competing interests.

Funding information

This project was funded by internal grants by NIFU.