How relevant is public policy and administration research for the policy sector? An empirical analysis based on Overton data

Robin Haunschild^*, Kate Williams^**, and Lutz Bornmann^*,***

^* R.Haunschild@fkf.mpg.de; L.Bornmann@fkf.mpg.de

0000-0001-7025-7256; 0000-0003-0810-7091

Information Retrieval Service (IVS-CPT), Max Planck Institute for Solid State Research, Germany

^** kate.williams@unimelb.edu.au

0000-0002-2882-1068

School of Social and Political Sciences, University of Melbourne, Australia

^*** Lutz.Bornmann@gv.mpg.de

0000-0003-0810-7091

Science Policy and Strategy Department, Administrative Headquarters of the Max Planck Society, Germany

A key goal of public policy and public administration research is to inform policy decisions. It is not clear, however, to what extent this is the case. In this study, therefore, citations from policy documents to public policy and administration research were analyzed to identify which research contributed most to policy reports and decisions. Additionally, we identified which policy institutions used research literature more than others to justify their policy decisions. Our findings show that think tanks use public policy and administration research literature more often than governmental organizations when justifying policy reports and decisions.

Introduction

Expertise has become an integral part of policymaking processes around the world. Accordingly, there is now a substantial literature on expert knowledge and policy processes. Much of this work has focused on the dual logics of science and politics and the various ways these complement or contradict each other (Jasanoff 1990). Attempts to conceptualise the influence of expert knowledge on policymaking have been fragmented (Christensen, 2021). The knowledge utilisation literature, from the field of public policy, has centred the use of information or research in policymaking. This includes evidence-based policymaking (Head, 2016), which aims to understand how evidence is taken up in policy design, as well as the strategic and symbolic use of evidence to gain political legitimacy (Boswell 2009). The field of political science, by contrast, tends to focus on the overarching ideas that arise from academic research and how they shape public policies (Hall, 1989). Other fields including Science and Technology Studies (STS) and sociology have focused on how individuals or groups of experts provide meaningful knowledge to decision-makers, for example, considering the role of professions (Abbott, 1988), networks (Haas, 1992) or knowledge brokers (Sverrisson, 2001). Yet, in part due to the methodological difficulties of capturing influence on audiences outside the academy, there remains limited empirical work on the impact of expert knowledge on policymaking.

One source of expert knowledge that has particular bearing on policymaking is the scholarship in the area of public policy and administration (PPA) itself. The field of PPA encompasses a range of orientations and topics including government, public policy, public management, public administration, and political science. The field can broadly be conceived as oriented towards improving understandings of policymaking and public administration and supplying decision makers with reliable policy- and administration-relevant knowledge about economic and social problems (Fischer, Miller, & Sidney, 2006). While the goals of individual experts are undoubtedly varied, insight into the overarching aims of the field is provided by professional associations and journals. One leading association, the International Research Society for Public Management, aims to facilitate “the creation and dissemination of new knowledge and understanding across [the international research] community and into policy and practice” (IRSPM 2023). Along the same lines, Public Administration Review, the official publication of the American Society for Public Administration, seeks to “identify and analyse current trends, provide a factual basis for decision making, stimulate discussion, and make the leading literature in the field available in an easily accessible format” in order to serve “academics, practitioners, and students” (Public Administration Review, 2023). Similarly, Canadian Public Policy is “directed at a wide readership including decision makers and advisers in business organizations and governments, and policy researchers” (Canadian Public Policy, 2023). Thus, experts in the field seek to go beyond academic impact to have influence in the policy and public management sphere. Yet, the influence of PPA scholars has received little systematic attention.

In this study, we deal with the question of how PPA research and policy are connected. Although reliable data on the connection between research and policy have historically been hard to find, the introduction of the Overton database in 2019 (Szomszor & Adie, 2022) has allowed analysis of research papers that are cited in policy documents. Policy documents have been defined as “‘carriers’ of policies . . . [that] provide a channel through which policy science researchers can study the main contents of policies, policymaking processes, and policy instruments” (Yang, Huang, & Su, 2020). In this empirical study, we use the Overton database to find policy documents that cited PPA research.

Dataset and Methodology

2.1 Dataset

We used a list of 49 PPA journals indexed in the Web of Science (WoS, Clarivate) as basis, downloaded from https://mjl.clarivate.com/ on 01/10/2022. We used the bibliometric in-house database of the Max Planck Society (MPG). We restricted the dataset to the publication years 1980-2019. Most publications belong to one of the following document types: article, book review, editorial or review. Thus, we restricted our analysis to these four document types. Furthermore, we restricted the dataset to the journals that published at least 50% of the publications with a DOI among these four document types within the time period 1980-2019. This procedure removed six journals from our dataset (Amme Idaresi Dergisi, Civil Szemle, Reforma y Democracia, Gestión y Política Pública, Transylvanian Review of Administrative Sciences, and Journal of Homeland Security and Emergency Management). We also removed two other journals (Climate Policy and Environment and Planning C – Politics and Space) that have a key focus on environmental studies (and are classified in the WoS Subject Category Environmental Studies). We expected that these two journals would distort our dataset from PPA too much. We kept the journal Science and Public Policy although it is also assigned to Environmental Studies because it is also assigned to the WoS Subject Category Management. Management is a key area of overlap for public management research. This left us with a list of 41 journals that are shown in Table 1. Some DOIs (n=459) occurred multiple times. The corresponding papers (n=1,352) were removed from our dataset.

Table 1. PPA journals that were included in our study ordered descending by the number of papers with DOI

Table 2 shows the distribution of the WoS PPA publications across the document types. Overall, the relative frequency of document types is similar to the total WoS. The only exception are book reviews with a significantly higher proportion among PPA publications than among the total WoS.

Table 2. Document types of WoS PPA publications that were included in our study ordered descending by the number of papers with DOI

We used a snapshot of the Overton database (Szomszor & Adie, 2022) from 19 January 2023 to obtain information about which policy sources cited which selected PPA journals. This snapshot has been imported into a local PostgreSQL database at the Max Planck Institute for Solid State Research (Stuttgart, Germany). Both databases (WoS and Overton) were linked via the DOIs of the scientific publications.

2.2 Methodology

We used PostgreSQL and R (R Core Team, 2019) commands including the R package ‘tidyverse’ (Wickham, 2017) for data analysis.

3. Results

3.1 Number of documents

Figure 1 shows the distribution of the number of PPA publications and the number of policy documents that cited these publications across the years. Overall, the proportion of PPA publications cited at least once by a policy document was rather constant in the 1980ies. In the 1990ies, this proportion increased and reached a maximum of about 0.45 in 2005. It has been decreasing since then. One explanation could be that the interest in scientific papers has decreased since 2005 in the policy sphere. Another explanation could be that it takes rather long until new scientific findings make it into public policy. The number of policy documents that cited PPA publications was very low until 1997 but has increased seemingly exponential since then. Already in 2006, more policy documents that cited PPA publications were published than PPA publications that were cited at least once by a policy document. In 2015, more policy documents that cited PPA publications were published than PPA publications.

Figure 1: Number of PPA publications and number of policy documents that cited these publications across the years

Table 3 shows the percentages of PPA papers (cited either by papers or policy documents) in comparison with their uncited counterparts. The PPA reviews are less frequently cited than reviews in general whereas PPA book reviews and editorials are more frequently cited than book reviews and editorials in general. Overall, PPA publications are less frequently cited by policy documents than by WoS papers except for book reviews.

Table 3. Percentages of WoS PPA papers cited either by WoS papers or policy documents broken down by the WoS document types ordered by the percentages of WoS PPA papers cited by policy documents

3.2 Sectors and sources of the policy documents

Figure 2 shows the percentage of policy documents per policy sector for all policy documents in the Overton database in the left panel and for the policy documents that cited PPA publications in the right panel. The proportion of policy documents that cited a PPA publication belonging to the governmental sector is much lower than for the full Overton database, although PPA publications should be of especial interest for the governmental sector. Intergovernmental organizations (IGOs) and think tanks are much more frequently the sources of policy papers that cited a PPA publication in comparison to the full Overton database.

Figure 2: Percentage of policy documents per policy sector

Figure 3 shows a scatter plot with the percentage of policy documents that cited PPA publications against the number of all policy documents in the Overton database. The individual points are coloured by their policy sector. Amongst the organizations with high proportions of policy documents citing PPA papers, there are a variety of research areas, political orientations and proximities to policymaking. Some take on advocacy roles, such as the Thomas B. Fordham Institute, which is a conservative American non-profit think tank with a specific mission around education reform. Others such as the Council of Canadian Academies, the Institute for Research on Public Policy, and MySociety bring together expert research to inform public understanding and decision making on a wide range of issues. Some, such as the Centre for Analysis of Social Exclusion and the UCL Institute for Innovation and Public Purpose are university-based multi-disciplinary research centres. Other organizations have closer links to policy, such as PRUComm, which is funded by the UK’s National Institute of Health Research to provide evidence for the Department of Health and Social Care, and the Intergovernmental Panel on Climate Change (IPCC), which is the United Nations body for assessing the science related to climate change. Two of these, SAPEA and the Swedish Institute for European Policy Studies (SIEPS), are independent government agencies. SAPEA provides independent scientific advice to the European Commission to support decision-making, and SIEPS is an independent government agency for research and analysis of European policy affairs.

Figure 3: Percentage of policy documents that cited PPA publications versus number of all policy documents in the Overton database (an interactive version can be accessed at: https://s.gwdg.de/r930Yl)

Table 4 shows the most productive policy sources grouped by policy sector. National governments produce a large volume of policy documents, but relatively few policy documents that rely on PPA publications. However, some key exceptions include large supranational organizations, such as the Publications Office of the European Union, and IGOS including OECD and World Bank, which publish many policy documents that cite PPA research. Think tanks generally produce fewer policy documents, but a higher proportion of these cite PPA research. For example, 8.4% of policy documents published by Germany’s IZA Institute of Labor Economics engage with PPA discourse.

Table 4: Types of policy sources most productive in publishing policy documents that cited WoS PPA publications in comparison with all policy documents

4. Discussion

The interpretation of the policy impact of research can only be undertaken based on an important caveat that policy sources capture two distinct types of documents: those that are authored by scientists to inform decision making, and those that are authored by policymakers to inscribe policy decisions or legislation. The distinction between these types of documents explains some of the results here. For example, some organizations that seek to inform policy decisions, such as IPCC and PRUComm, will include many references to provide a robust evidence base. On the other hand, policy briefs or ministerial notes by governments or intergovernmental organizations may intentionally limit literature sources to improve accessibility or to conform to policy conventions. The policy impact analysis in this study, therefore to some extent highlights different policy document types that may be favoured by certain organization types. Interestingly, PPA book reviews are more frequently cited by policy than by WoS papers in general, perhaps suggesting that highly accessible formats that present concise or synthesised evidence are favoured over more comprehensive formats in this field.

In this study, we examined several aspects of the connection between PPA research and policy. Focusing on the connection over time reveals that while the number of PPA papers and the number of policy documents that cite them have increased in recent years, the proportion of PPA research in policy seems to have peaked in the mid-2000s. This potentially reflects a declining interest in the PPA literature within the policy sphere or a lag between new evidence and its translation into policy.

Our results show that governments and intergovernmental organizations produce a high number of policy documents, but cite rather few papers from PPA research. This suggests that, in general, PPA research is less influential on decision makers and advisers in international organizations and governments, than on policy researchers within think tanks. Think tanks seem to produce fewer policy documents overall, but that are more closely related to PPA research. This is potentially due to the brokerage function of think tanks, whereby they seek to understand how to actively move expert knowledge into the policy sphere (Abelson, 2009). However, our results show some exceptions to this. Some IGOs, including the Publication Offices of the European Union, OECD, and World Bank, are very active in the PPA discourse. National governments in particular are much less likely to be the source of policy papers that cited a WoS PPA publication in comparison to the full Overton database. This is reflected in the most productive government sector source of documents that cited PPA research being the inter-governmental Publications Office of the European Union by a large margin. This may point to a potential disconnect between national government decision making and PPA research, which does not reflect the goals of many PPA journals and associations. Alternatively, it may reflect complex methodological challenges around policy impact.

5. Conclusions

This study highlights the dual challenges of creating and measuring policy impact. One challenge of much of the detailed synthesis work in PPA research is not directly used in policymaking. This means that attempts to provide a solid evidence base may not be as well received by governmental decision makers as more accessible formats. It also produces particular challenges around measurement, whereby documents that inscribe actual policy decisions typically do not contain many citations, making policy impact difficult to capture. The findings here provide support for the idea that the PPA literature is further built upon and brokered by think tanks (and to a lesser extent, IGOs), who then help translate this knowledge into policy briefs, policy reports, briefing notes or ministerial expertise. Final policy decisions made by governments typically do not contain any citations. Thus, PPA research may have different impact for different organizational types, which is complicated because it becomes increasingly challenging to measure impact the closer we get to meaningful decisions.

Open science practices

We are not aware of an open database of policy documents. Thus, we are restricted to using a closed dataset.

Acknowledgments

The bibliometric data used in this study are from a bibliometric in-house database of the Max Planck Society (MPG), developed and maintained in cooperation with the Max Planck Digital Library (MPDL, Munich), derived from the Science Citation Index Expanded (SCI-E), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI) prepared by Clarivate (Philadelphia, Pennsylvania, USA) via the “Kompetenznetzwerk Bibliometrie” (see https://bibliometrie.info/en/about-kb/) funded by BMBF (grant 16WIK2101A). The policy document data used in this study are from an Overton snapshot provided by Euan Adie on 19 January 2023.

Author contributions

Conceptualization: LB & KW

Data curation: RH & KW

Investigation: RH & KW

Methodology: RH, KW, & LB

Visualization: RH

Writing – original draft: RH & KW

Writing – review & editing: RH, KW, & LB

Competing interests

The authors declare that they have no competing interests.

Funding information

Access to bibliometric data was enabled via grant 16WIK2101A.

References

Abbott, A. (1988). The system of professions: An essay on the division of expert labor. London: University of Chicago Press.

Abelson, D. E. (2009). Do think tanks matter? Assessing the impact of public policy institutes introduction. Kingston: Queen's School of Policy Studies.

Boswell , C. (2009). The political uses of expert knowledge: Cambridge University Press.

Canadian Public Policy. (2023). Canadian Public Policy. Retrieved March 10, 2023, from https://www.utpjournals.press/loi/cpp

Christensen, J. (2021). Expert knowledge and policymaking: A multi-disciplinary research agenda. Policy & Politics, 49(3), 455-471. doi: https://dx.doi.org/10.1332/030557320x15898190680037.

Fischer, F., Miller, G., & Sidney, M. (2006). Handbook of public policy analysis: Theory, politics, and methods: CRC Press.

Haas, P. M. (1992). Introduction: Epistemic communities and international policy coordination. International Organization, 46(1), 1-35. doi: https://dx.doi.org/10.1017/S0020818300001442.

Hall, P. A. (1989). Introduction. In P. A. Hall (Ed.), The political power of economic ideas (pp. 1-26). Princeton: Princeton University Press.

Head, B. W. (2016). Toward more “evidence-informed” policy making? Public Administration Review, 76(3), 472-484. doi: https://doi.org/10.1111/puar.12475.

IRSPM (2023). International Research Society for Public Management. Retrieved March 10, 2023, from https://www.irspm.org/

Jasanoff , S. (1990). The fifth branch. Science advisers as policymakers. Cambridge, MA: Harvard University Press.

Public Administration Review. (2023). Public Administration Review - Wiley Online Library. Retrieved March 10, 2023, from https://onlinelibrary.wiley.com/journal/15406210

R Core Team. (2019). R: A language and environment for statistical computing (Version 3.6.0). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.r-project.org/

Sverrisson, Á. (2001). Translation networks, knowledge brokers and novelty construction: Pragmatic environmentalism in Sweden. Acta Sociologica, 44(4), 313-327.

Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624-650. doi: 10.1162/qss_a_00204.

Wickham, H. (2017). Tidyverse: Easily install and load the 'Tidyverse'. R package version 1.2.1. Retrieved 22 June 2020, from https://CRAN.R-project.org/package=tidyverse

Yang, C., Huang, C., & Su, J. (2020). A bibliometrics-based research framework for exploring policy evolution: A case study of China's information technology policies. Technological Forecasting and Social Change, 157, 120116. doi: https://doi.org/10.1016/j.techfore.2020.120116.