Exploring ministries’ use of research in policymaking - knowledge stock, circulation, or routines?

Taran Thune^*, Magnus Gulbrandsen^** , Ingvild Reymert^*** and Silje Maria Tellmann^****

^*t.m.thune@tik.uio.no

ORCID: 0000-0002-4951-3734

^**magnus.gulbrandsen@tik.uio.no

ORCID: 0000-0001-9976-1608

^****s.m.tellmann@tik.uio.no

ORCID: 0000-0002-4659-281X

TIK Centre for Technology, Innovation and Culture, University of Oslo, Norway

Department, Institution, Country

^*** ini@oslomet.no

ORCID: 0000-0003-4442-7404

Oslo Business School, Oslo Metropolitan University

A paper must have an abstract of 50 to 150 words.

1. Introduction

Current science policy is very much concerned with impact of research – both its assessment and its funding. Researchers are expected to be able to describe how their research activities and outcomes influence a domain of practice such as healthcare, industrial innovation or policymaking, and impact assessment is integrated in most forms of research evaluation (Donovan, 2011). In the most recent literature, impact is conceptualised as the result of multiple types of pathways in which researchers interact in various ways with non-researchers (e.g. Muhonen et al., 2020).

However, even when highlighting “networks” and “interactions” (Spaapen & van Drooge 2011; Molas-Gallart & Tang 2011), recent conceptual frameworks and related evaluation approaches tend to have a bias towards research outcomes and researchers’ activities and behaviour. From a perspective of interaction and impact pathways, we argue that researchers cannot be held unilaterally responsible for the absorption and use of scientific knowledge in society. Many factors beyond the control, knowledge or imagination of researchers may influence the absorption and use of scientific knowledge (Cozzens & Snoek, 2010). To understand these factors, conceptual and methodological tools to better comprehend the “user side” of research impact are needed.

Moreover, there is a widespread demand that decisions in all domains of policy should be based on scientific evidence (Christensen & Holst, 2017; Greenhalgh et al. 2014), and due to this, systems to systematize, evaluate and diffuse scientific knowledge to policy makers and professionals in various parts of the public sector have been developed in many countries and policy contexts.

Accompanying the interest in the use of scientific evidence in policymaking, the issue of research use has also received considerable attention as a research topic. Studies have tended to focus on knowledge user characteristics and the drivers and barriers to research use, often in contexts of medical practice and/or health policy settings where “evidence-based” policy has been specifically formulated as a goal and implemented in practice (Estabrooks et al., 2003; Squires et al., 2011). Deficiency arguments is often a starting point in these studies (Chagnon et al., 2010), and they seek to uncover the barriers to research utilization, such as lack of access of scientific knowledge, limited competence, and lack of communication between scientists and policy makers. Research in this tradition, sometimes referred to as “the clinical model”, tend to focus on individual users of scientific knowledge and factors that support or limit their access and use of scientific knowledge and evidence in settings of policy or practice (Eastabrooks et al., 2003).

Research use is also a long-standing theme within political science and public administration studies (Belkhodja et al. 2007; Head et al. 2014, Landry et al. 2003; Ouimet et al. 2010). The emphasis is often on understanding the complexity of decision-making process and the varied, slow and often indirect ways (“knowledge creep”) that research influences policymaking (Weiss, 1980). A core assumption is that it requires active involvement of users. Research must be absorbed and transformed into new formats, and it meets already established and sometimes competing sources of knowledge (MacKillop et al. 2020). It is also explicitly recognized that individuals (policymakers or government officials) operate in specific contexts that influence the opportunities and abilities to access, work with and use research in practice. Organizational features that have been found to be conducive for research utilization is competence levels and composition, its size and resources, leadership, organizational structures and organizational routines that support organizational members’ abilities to access and use scientific knowledge in their work (Makkar et al. 2018). Though rarely explicitly recognized, arguments mirror concepts used in the broader knowledge management research field, as specifically the concept absorptive capacity (Cohen & Levinthal, 1990; Zahra and George, 2002) that has been used extensively to explain innovation outcomes in firms.

This paper addresses the use of scientific knowledge in policy work as an organization-level phenomenon and draws on the absorptive capacity concept. Through an empirical study of 14 Norwegian ministries, we seek to understand how organizational characteristics and processes that may play a role for whether and how policy organisations use scientific knowledge in policy making. We are aware that the issues addressed here are highly complex, and that we single out certain dimensions for the sake of comparison. Methodologically, we make a qualitative comparative analysis (Ragin, 1987; Rihoux and Ragin, 2012) of 14 ministries, based on both qualitative and quantitative data.

Through this we seek to contribute to the science policy literature in three ways: First, by bringing closer attention to different user contexts and how they shape research use and impact. Second, we develop an organization level perspective on research use in policymaking. Third, we look into organisations that often remain closed to public scrutiny (ministries and their employees) and make a qualitative comparative analysis to understand different contextual conditions and how they may be interrelated.

2. Analytical framework and assumptions

Capabilities, both at an individual and organizational level, is a key explanation for research utilization among policy makers and professionals, mirroring similar arguments about absorptive capacity as a firm characteristic made several decades ago (Cohen & Levinthal, 1990). The basic idea in the absorptive capacity literature (Zou et al., 2018) is that there is an additive effect where the ability to exploit new knowledge is associated with past “investments” in knowledge.

Both in the research utilization literature and the absorptive capacity literature, capabilities that influence ability to exploit knowledge for different purposes are often described in terms of the stock of existing knowledge (features of the individual or the organization) and not active steps taken to mobilize and act on knowledge in the organization, which is more about organizational processes and routines. However, studies in policy settings have found that research use in policy or practice settings is positively associated with active participation in the generation of knowledge and participation in activities related to circulating and translating knowledge in practice settings. Having a set of characteristics that increase the likelihood of utilizing scientific evidence and having processes and routines to act on scientific knowledge in the organization is of course not unrelated, but the relative importance might influence research use.

The distinction between passive and active (or “realized”) absorptive capability (Zahra and George, 2002) is of relevance here. According to Zahra and George, absorptive capacity is not only about having capability that influence the ability to identify and access knowledge (referred to as “acquisition”), but also about capabilities or organizational routines to relate new knowledge to existing knowledge and having routines for interpreting, circulating, translating, acting, and using new knowledge within the organization.

In this paper, we make use of the distinction between potential and realized absorptive capacity to address research utilization in public policy organization, and we investigate the following issues: First, we explore whether the knowledge stocks of a government organization influence the propensity to acquire research. Second, we are interested in understanding if knowledge circulation activities influence the propensity to use research-based knowledge in government organizations. Finally, we will look at whether having specific organizational routines in place influences both knowledge acquisition and use in government organizations.

Figure 1: Model for analyzing the determinants of knowledge use

3. Data and methods

To address the research question and assumption, we investigate a set of cases – 14 Norwegian government organizations (ministries). Together they represent the highest national authorities for most domains of policy (not including justice and homeland security).

Table 1: Included organizations

We build up a dataset on each organization from a range of data sources for the analysis. First, we draw on data from a survey about the use of scientific knowledge in government organisations (ministries and selected government agencies) collected in 2019. The survey collected data on a range of different questions about how government employees and the organisations in which they work acquire and use scientific knowledge in policy making (Thune, Simensen & Gulbrandsen, 2021).

In this paper, we use aggregated data about how government employees acquire research in their work, how they contribute to circulate research in their organizations and how they use research in their work (see below for details about the variables). The survey also contains data about knowledge handling routines in each organization, but these variables are also underpinned by other sources of data (documents and interviews).

In addition to the survey data, we use organization-level data on R&D investments in ministries from the national budget, employee statistics from the national statistical agency and other relevant data on ministry employees collected from “state administration surveys” (Christensen et al., 2017). We also draw on qualitative data (interviews and documents) to support and enrich the analysis of inter-ministerial similarities and differences.

To analyse the data from the comparative case study of 14 ministries, we will build on the qualitative comparative analysis approach (Ragin, 1987; Rihoux and Ragin, 2012). QCA is a relevant method for systematic analysis of complex cases. The purpose of the method is to compare a set of cases in terms of their configurations, related to an outcome of interest. It is based on a theoretical model, but compared to a variable-oriented approach, one compares patterns of multiple characteristics (configurations) in a set of cases. QCA is often based on a mix between qualitative and quantitative data, but data (also qualitative) are transformed into numerical values (membership scores) to ease the comparison, and the analysis may be performed using statistical software such as R. As the analysis is based on the formulation of a theoretical model of the relationship between a set of characteristics and specific outcomes, it is more deductive than most qualitative approaches. However, initial models are refined in interplay with data, in the spirit of an abductive mode of theorizing (Ragin, 1987).

In our analysis, we are interested in exploring a set of characteristics (configurations of characteristics) associated with the use of research in policymaking in different ministries. We transform our raw data on different conditions into membership scores and use this to determine the configurations of the different cases, and finally analyse whether what configuration is associated with a specific outcome (Oana et al., 2021). Our data can be adapted to a fuzzy set analysis.

Table 2: the raw data that will be included in the analysis.

4. Preliminary results

The descriptive analysis of the survey data (not all shown here) shows that ministry employees regularly consult research as part of their daily work and that research institutes and universities (academic sources) are important sources of knowledge in policymaking. Moreover, it is also common for government employees to contribute to circulating and diffusing research inside the organisations in which they work, but that such activities often are informal, and for instance occur through sharing publications, research news and more. It is also quite common to specifically use research in policy work, particularly by referring to research results in policy documents or by summarizing research with the aim of advising politicians about a certain course of action. Most of the ministries in the survey also have routines and infrastructures in place to support employees use of research in their work, but there is also some variety across the 14 ministries in the study.

The next steps will involve recoding data from the survey and supplementary data sources, and to develop the coding scheme for the fsQCA method. In parallel, we will also analyse documents and interview data that have been collected to guide the coding and the comparative analysis.

5. Bibliographic references

Belkhodja, O., N. Amara, R. Landry & M. Ouimet, 2007. The Extent and Organizational Determinants of Research Utilization in Canadian Health Services Organizations. Science Communication, 28(3): 377-417.

Chagnon, F., Pouliot, L., Malo, C., Gervais, M. J., & Pigeon, M. È. (2010). Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implementation Science, 5, 1-12.

Christensen, J. & C. Holst, 2017. Advisory commissions, academic expertise and democratic legitimacy: the case of Norway. Science and Public Policy, 44(6): 821-833.

Cohen, W. M., & Levinthal, D. A. (1990). Absorptive capacity: A new perspective on learning and innovation. Administrative science quarterly, 128-152.

Cozzens, S., & Snoek, M. (2010, December). Knowledge to policy. In Workshop on the Science of Science Measurement.

Donovan, C., 2011. State of the art in assessing research impact: introduction to a special issue. Research Evaluation, 20(3): 175-179.

Estabrooks, C., J.A. Floyd, S. Scott-Findlay, K.A. O’Leary & M. Gushta, 2003. Individual determinants of research utilization: a systematic review. JAN, 43(5): 506-520.

Greenhalgh, T., J. Howick & N. Maskrey, 2014. Evidence-based medicine: a movement in crisis? BMJ, 2014; 348:3725

Head, B., M. Ferguson, A. Cherney & P. Boreham, 2014. Are policy-makers interested in social research? Exploring the sources and uses of valued information among public servants in Australia. Policy and Society, 33(2): 89-101.

Landry, R., M. Lamari & N. Amara, 2003. The Extent and Determinants of the Utilization of University Research in Government Agencies. Public Administration Review, 63(2): 192-205.

MacKillop, E., Quarmby, S., & Downe, J. (2020). Does knowledge brokering facilitate evidence-based policy? A review of existing knowledge and an agenda for future research. Policy & Politics, 48(2), 335-353.

Makkar, S. R., Haynes, A., Williamson, A., & Redman, S. (2018). Organisational capacity and its relationship to research use in six Australian health policy agencies. PLoS One, 13(3), e0192528.

Molas-Gallart, J. & P. Tang, 2011. Tracing ‘productive interactions’ to identify social impacts: an example from the social sciences. Research Evaluation, 20(3): 219-226.

Morton, S., 2015. Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation, 24(4): 405-419.

Muhonen, R., P. Benneworth & J. Olmos-Peñuela, 2020. From productive interactions to impact pathways: Understanding the key dimensions in developing SSH research societal impact. Research Evaluation, 29(1): 34-47.

Oana, I., Schneider, C., & Thomann, E. (2021). Qualitative Comparative Analysis Using R: A Beginner's Guide. Cambridge: Cambridge University Press. doi:10.1017/9781009006781.005

Ragin, C. C. (1987/2014). The comparative method. University of California Press.

Ouimet, M., Bédard, P. O., Turgeon, J., Lavis, J. N., Gélineau, F., Gagnon, F., & Dallaire, C. (2010). Correlates of consulting research evidence among policy analysts in government ministries: a cross-sectional survey. Evidence & Policy, 6(4), 433-460.

Ragin, C. C. (2014). The comparative method: Moving beyond qualitative and quantitative strategies. Univ of California Press.

Rihoux, B., & Ragin, C. C. (2008). Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques. Sage Publications.

Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011). Individual determinants of research utilization by nurses: a systematic review update. Implementation science, 6, 1-20.

Thune, T., Simensen, E., & Gulbrandsen, M. (2021). Forskning i politikk og forvaltning. Resultater av en spørreundersøkelse blant ansatte i staten. Oslo Institute for Research on Impact of Science (OSIRIS), Senter for innovasjon, kultur og teknolog, Universitetet i Oslo.

Weiss, C. H. (1979). The many meanings of research utilization. Public administration review, 39(5), 426-431.

Zahra, S. A., & George, G. (2002). Absorptive capacity: A review, reconceptualization, and extension. Academy of management review, 27(2), 185-203.

Zou, T., Ertug, G., & George, G. (2018). The capacity to innovate: A meta-analysis of absorptive capacity. Innovation, 20(2), 87-121.

Competing interests

The authors declare no competing interests in this research.

Funding information

The research is carried out within the OSIRIS project, funded by the Research Council of Norway’s Grant Number 256240.