Commissioned public R&D publications in the social sciences. A document analysis of societal impacts of research

Topics: Societal impact of research, science policy, altmetrics, bibliometrics

Sabine Wollscheid^* and Vera Schwach^**

^*sabine.wollscheid@nifu.no

ORCID https://orcid.org/0000-0001-7376-9820

Nordic Institute for Studies in Research Innovation and Education (NIFU), Norway

^** vera.schwach@nifu.no

ORCID https://orcid.org/0009-0000-4686-7599

Nordic Institute for Studies in Research Innovation and Education (NIFU), Norway

Abstract

This paper investigates the impact of social sciences, a recent field for considering the societal impacts of science. It is based on a purposeful selection of R&D publications, commissioned by Norwegian ministries, governmental agencies, and public bodies, aimed for the application for public sector. The concept societal impact is here defined as active and potential influential, and interactions between public research organisations (PRO) and governmental agencies and other public bodies in the areas of education and social welfare. We argue that the interactions lead to favourable outputs. We examine interactions, i.e., exchanges between researchers and stakeholders from governmental agencies and public bodies, in which scientifically robust and societally relevant knowledge is constructed and valued. In investigating interactions, we focus on qualitative and quantitative indicators of direct social interactions and indirect interactions through texts (see e.g., Spaape & van Drooge, 2011).

1. Introduction

More than ever public funding institutions are aware of their obligation to justify public investment in research and development (R&D), and public research funding is linked to both, academic impact and to designated societal returns.

Since 2000, aside from the emphasis put on academic impact, like high scientific productivity and excellence, research policy in the European Union has increasingly addressed the value of societal impact (Bornmann, 2013: 218–220). Distinctions between different types of societal impact have been introduced such as: […] “societal products (outputs), societal use (societal references), and societal benefits (changes in society)” (Bornmann, 2013: 217). Focussing on societal use, we address the following research questions: How are interactions between researchers and stakeholders from governmental agencies and public bodies reflected in commissioned public R&D publications in social sciences, and which indicators are used to measure impacts?

The increased weight given to societal use and impact is concurrent with the development of evidence-based policy (e.g., Young et al. 2002) a policy originated in evidence-based medicine (see Guyatt et al., 1992). Independently and in combination the two trends, societal impact and use, and evidence-based policy, have gradually affected a range of research areas comprising economic and environmental studies (e.g., Garnett et al., 2019; Heringa et al., 2016), Science, Technology, and Innovation (STI-programmes) and in education and social welfare (Slavin, 2002; Lingard, 2013; Krejsler, 2013).

Since 2010 there has been an increase in the inclusion of societal impact, also in research evaluations in social sciences and humanities (Forskningsrådet 2017, RCN 2018, RCN 2021). This paper concentrates on social sciences as a recent field for studying societal impact (e.g., Lauronen, 2022).

The study addresses societal use as a dimension of societal impact in a Nordic context with a particular focus on the social sciences in Norway, a country with a longstanding tradition with commissioned research, primarily done by the government, governmental agencies and other public, but also by the non-profit private sector (Brandt et al. 2019). This analysis follows up Lauronen’s article (2022). We argue that a Nordic perspective on assumed productive interactions between researchers and stakeholders can provide an insight into intended applications of social impact in a political system different from systems with more rigorous, formal and often linear conceptions of social impact evaluations. These types of evaluations may have direct consequences for institutional/research funding. One example is the Research Excellence Framework (REF) in the UK with consequences for funding.

Our aim is to analyse a purposefully collected sample of R&D publications, all of them results of commissioned projects conducted in Norway, and consequently carried out with designated intentions of societal impact. These projects were funded by the government, governmental agencies, and public bodies. In investigating commissioned research which addresses societal challenges, we examine how R&D publications express societal use focussing on direct and indirect types of “productive interactions” (Sivertsen & & Meijer, 2020).

2. Method and data

Drawing on carefully selected cases in social sciences we investigate interactions between researchers and stakeholder groups.

Our analysis combines alternative metrics (altmetrics) and quantitative bibliometric indicators. We apply the following indicators: amounts of personal interactions (e.g., meetings, workshops), frequency of downloaded reports and number of citations in scholarly and non-scholarly literature. Non-scholarly literature comprises green papers, white papers, other relevant governmental reports, and policy documents.

For social sciences and humanities (SSH), non-scholarly literature, and literature in other languages than English, coverage is relatively low in highly indexed international databases, like Web of Sciences (WoS) (e.g., Aksnes & Sivertsen, 2019). In this paper, we use Research Gate, a social networking site for academics which comprises altmetrics and some bibliometric information.

Our approach is informed by document analysis (e.g., Prior, 2016). First, we select cases strategically, comprising R&D publications, commissioned by ministries, other governmental and public bodies. We sample documents that represent variations across disciplines and commissioning bodies. Second, we classify selected documents according to type(s) of indicated productive interactions and societal impact, commissioned body, and scope. Third, we conduct an in-depth, thematic analysis of the selected documents, defined as cases for the study.

3. Preliminary findings

In this paper, we compare and examine the main findings across the cases. In doing so, we draw on findings of the international literature on societal impact with the field of social sciences (see Table 1).

Case 1: Social sciences, technology, and innovation represents one, interdisciplinary case across the fields of social sciences, technology, and innovation, and includes two longitudinal R&D projects (1A and 1B), each with a three-year duration.

The first project (1A) was commissioned by a municipality. It was an evaluation, designed as a formative dialogue design study. The background was a municipality that had implemented one-to-one digital devices for all students in all its twenty-four primary and lower secondary schools. The pupils were provided with Chromebooks. The evaluation done on the implementation covered many different aspects of the implementation process of digital devices: such as teaching and learning with digital devices, competence development, institutional learning, and development of digital competence. The formative evaluation design implied close involvement across researchers and stakeholders. The researchers provided interactions as feedback to the g municipality. Project findings were continuously disseminated to stakeholders that were part of the initiative. These included status-meetings with the municipalities’ coordinator of the initiative, meetings with the head of the municipality for education and sharing of findings with teachers, principals, and parents (use of research).

In this paper, we analyse summative elements using altmetrics and bibliometric indicators (number of citations) in addition to reported outcomes (qualitative indicators), such as changed practices.

The second, (ongoing) project (1B), is called “Innovation Project for the Public Sector”, commissioned by the Norwegian Research Council. The aim is to pilot and implement a new model to develop leadership and organisation of a pedagogical development program and is closely related to the one-to-one implementation of Chromebooks in compulsory education. In addressing societal impact, the project aims at leading “to innovation in public sector services […] and generate research activities that will promote innovation within the sector and sustainable value creation for its users.” Innovations is here defined as: “new or significantly improved […] processes […] that are introduced to enhance value creation and for the benefit of society.” The project draws on the assumption that experiences from municipalities are valuable for other municipalities that have introduced one-to-one implementation. The design comprises communication and interaction strategies with stakeholders to ensure dissemination to the school sector in Norway This R&D project is supposed to contribute with new findings on pedagogical development in these schools, especially addressing school owner and principals.

In this paper, we use altmetrics and citation indicators in our analysis, in addition to

qualitative indicators of societal use in short-medium term.

Case 2: Social sciences – interdisciplinary encompass three related projects (five reports) that covering the scholarly literature on discrimination, harassment and racism, and its consequences. The five reports applied methods of mapping and systematic reviews. They were commissioned by three governmental agencies: the Research Council of Norway, The Norwegian Directorate for Children, Youth and Family Affairs (Bufdir) and The Directorate of Integration and Diversity.

The aim of report 2A, a mapping review of Nordic research on discrimination, harassment, and equality, commissioned by the Research Council, was to inform the Ministry’s work with an intersectoral R&D-strategy on equality and non-discrimination. To enhance interactions during the review process, a reference group across different governmental agencies was established.

Further, two projects applying systematic review methods and full-text articles address more specific research questions on consequences of discrimination and racism. The aim of project 2.B was to summarise the literature on consequences of racism and discrimination because of ethnicity, religion, and philosophy of life and to identify knowledge gaps. The aim of project 2.C was to summarise the literature dealing with consequences of racism and discrimination for work-life integration (2.C.1), educational integration (2.C.2) and integration in civil society (2.C.3).

In this paper, we combine altmetrics and citation indicators in our analysis, in addition to

qualitative indicators of societal use in short-medium term.

Table 1: Mapping of commissioned R&D publications

4. Discussion and conclusions

The assessment of societal impact of research is a growing topic, which includes the social sciences standing out as the field with the highest research production (Viana-Lora et al., 2021).

Citation indicators are in general used to measure scientific impact and relevance, but they are difficult to use to assess societal impact and productive interactions between researchers, and different stakeholder groups. A reason is that the literature indexed databases like Web of Sciences mostly comprises scientific publications (Aksnes, Langfeldt & Wouters, 2019) and neglects commissioned R&D publications.

Focussing on societal use we examine how interactions between researchers and stakeholders are reflected in commissioned public R&D publications in social sciences. Given distinctions between different types of societal impact, we elaborate societal uses (Bornmann, 2013) of R&D publications in short- and medium-terms by combining qualitative and quantitative indicators. To capture societal uses, we investigate close interactions between different stakeholders and researchers, e.g., captured by communications of findings, and activities facilitating a responsible use of research for policy making. We suggest further research to explore societal benefits, i.e., changes in society (Bornmann, 2013: 217), applying a multi-methods approach and taking into consideration a longitudinal perspective with a longer timeframe than the actual project period.

References

Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. Sage Open, 9(1), 1-17. doi.org/10.1177/2158244019829575

Aksnes, D. W. & Sivertsen, G. (2019). A Criteria-based Assessment of the Coverage of Scopus and Web of Science. Journal of Data and Information Science, 4(1), 1-21. DOI: doi.org/10.2478/jdis-2019-0001

Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey, Journal of the American Society for Information Science and Technology, 64(2): 217–233. doi.org/10.1002/asi.22803

Brandt, T.; Ingulstad, M.; Larsen, E.; Mangset, M. & Schwach, V. (2019). Avhengig av forskning. De norske forskningsrådenes historie 1946–2016 [Dependent on Research. The History of the Research Councils in Norway 1946–2016] (Fagbokforlaget) Trondheim (In Norwegian].

Forskningsrådet (2017). Casestudier over effektene av de samfunnsvitenskapelige instituttenes forskning. Vedlegg til evalueringsrapporten. [Case studies on the effect of research conducted by social research institutes. Attachments to evaluation reports]. Forskningsrådet Oslo.

https://www.forskningsradet.no/siteassets/publikasjoner/1254024993876.pdf,

Guyatt, G., Cairns, J., Churchill, D., Cook, D., Haynes, B., Hirsh, J., ... & Tugwell, P. (1992). Evidence-based medicine: a new approach to teaching the practice of medicine. Jama, 268(17), 2420-2425. doi:10.1001/jama.1992.03490170092032

Krejsler, J. B. (2013). What works in education and social welfare? A mapping of the evidence discourse and reflections upon consequences for professionals. Scandinavian Journal of Educational Research, 57(1), 16-32. https://doi.org/10.1080/00313831.2011.621141

Lauronen, J. P. (2022). Tension in Interpretations of the Social Impact of the Social Sciences: Walking a Tightrope Between Divergent Conceptualizations of Research Utilization. SAGE Open, 12(2), 1-16. doi.org/10.1177/21582440221089967

Lingard, B. (2013). The impact of research on education policy in an era of evidence-based policy. Critical Studies in Education, 54(2), 113-131. https://doi.org/10.1080/17508487.2013.781515

RCN (2018). Evaluation of Social Sciences in Norway (SAMEVAL) – Impact cases. The Research Council of Norway, Oslo https://www.forskningsradet.no/siteassets/publikasjoner/1254035787331.pdf

RCN (2021). Evaluation of Legal Research (JUREVAL). Impact cases. The Research Council of Norway, Oslo https://www.forskningsradet.no/siteassets/publikasjoner/2021/jureval-impact.pdf

Prior, L. (2016). Using documents in social research. In D. Silvermann (ed.). Qualitative Research. Los Angeles, London, New Delhi: Sage, pp. 171-185.

Sivertsen, G., & Meijer, I. (2020). Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society? Research Evaluation, 29(1), 66-70. https://doi.org/10.1093/reseval/rvz032

Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational researcher, 31(7), 15-21. https://doi.org/10.3102/0013189X031007015

Spaapen, J. & van Drooge, L. (2011). Introducing ‘productive interactions’ in social impact assessment, Research Evaluation, (20) 3, September 2011, 211–218, doi.org/10.3152/095820211X12941371876742

Viana-Lora, A., & Nel-lo-Andreu, M. G. (2021). Approaching the social impact of research through a literature review. International Journal of Qualitative Methods, 20, 1-11. DOI: 10.1177/16094069211052189

Young, K., Ashby, D., Boaz, A., & Grayson, L. (2002). Social science and the evidence-based policy movement. Social policy and society, 1(3), 215-224. DOI: https://doi.org/10.1017/S1474746402003068

References of included reports (related to case studies)

1 A. Tømte, C., Wollscheid, S., Bugge, M., & Vennerød-Diesen, F. F. (2019). Digital læring i askerskolen. Sluttrapport fra følgeforskning. [Digital learning in schools in the Asker municipality. Final report]. 2019:27. Nordic Institute for Studies in Innovation, Research and Education, Oslo. doi.org/10.13140/RG.2.2.23526.98885

https://nifu.brage.unit.no/nifu-xmlui/handle/11250/2631639

2.A. Aksnes, D. W., Bergene, A. C., Fossum, L. W., & Wollscheid, S. (2021). Nordisk forskning om diskriminering, trakassering og likestilling: En forenklet kunnskapsoversikt. [Nordic research on discrimination, harrassment and equality: A scoping review]. 2021:14. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/2830908

2.B. Wollscheid, S., Lynnebakke, B., & Fossum, L. W. (2021). Konsekvenser av rasisme og diskriminering på grunn av etnisitet, religion og livssyn – en kunnskapsoversikt. [Consequences of racism and discrimination due to ethnicity, religion and life perspective – a literature review. 2021:22. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/2978279

2.C.1. Wollscheid, S., Alne, R., Bergene, A. C., & Karlstrøm, H. (2021). Effekter av rasisme og diskriminering på integrering i arbeidslivet – en kunnskapsoversikt. [Consequences of racism and discrimination on integration into working life – a literature review]. 2022:20. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/3026126

2.C.2. Wollscheid, S., Lynnebakke, B., Kindt, M. T., Karlstrøm, H., & Fossum, L. W. (2022). Konsekvenser av rasisme og diskriminering for integrering innen utdanning: En kunnskapsoversikt. [Consequences of racism and discrimination for integration into education – a literature review.] 2022:22. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/3036342

2.C.3. Wollscheid, S., Lynnebakke, B., Bergene, A. C., Karlstrøm, H., & Fossum, L. W. (2022). Konsekvenser av rasisme og diskriminering for deltakelse i sivilsamfunnet: En kunnskapsoversikt. [Consequences of racism and discrimination for participation in civil society: A literature review]. 2022:27. Nordic Institute for Studies in Innovation, Research and Education, Oslo. https://nifu.brage.unit.no/nifu-xmlui/handle/11250/3044154

Open science practices

Data in this paper are commissioned R&D reports. These reports are openly accessible online and can be retrieved for example vis Research Gate.

For publication we consider journals with open access policy (gold or green open access).

Author contributions

Sabine Wollscheid: Development and draft of the proposal; methodological development and mapping of the two cases.

Vera Schwach: Development and draft of the proposal. Substantial contributions to main concepts and terminology in the field of science policy.

Competing interests

Authors declare that they have no competing interests.

Funding information

This project was funded by internal grants by NIFU.