How relevant is public policy and administration research for the policy sector? An empirical analysis based on Overton data

A key goal of public policy and public administration research is to inform policy decisions. It is not clear, however, to what extent this is the case. In this study, therefore, citations from policy documents to public policy and administration research were analyzed to identify which research contributed most to policy reports and decisions. Additionally, we identified which policy institutions used research literature more than others to justify their policy decisions. Our findings show that think tanks use public policy and administration research literature more often than governmental organizations when justifying policy reports and decisions.


Introduction
Expertise has become an integral part of policymaking processes around the world.Accordingly, there is now a substantial literature on expert knowledge and policy processes.Much of this work has focused on the dual logics of science and politics and the various ways these complement or contradict each other (Jasanoff 1990).Attempts to conceptualise the influence of expert knowledge on policymaking have been fragmented (Christensen, 2021).The knowledge utilisation literature, from the field of public policy, has centred the use of information or research in policymaking.This includes evidence-based policymaking (Head, 2016), which aims to understand how evidence is taken up in policy design, as well as the strategic and symbolic use of evidence to gain political legitimacy (Boswell 2009).The field of political science, by contrast, tends to focus on the overarching ideas that arise from academic research and how they shape public policies (Hall, 1989).Other fields including Science and Technology Studies (STS) and sociology have focused on how individuals or groups of experts provide meaningful knowledge to decision-makers, for example, considering the role of professions (Abbott, 1988), networks (Haas, 1992) or knowledge brokers (Sverrisson, 2001).Yet, in part due to the methodological difficulties of capturing influence on audiences outside the academy, there remains limited empirical work on the impact of expert knowledge on policymaking.
One source of expert knowledge that has particular bearing on policymaking is the scholarship in the area of public policy and administration (PPA) itself.The field of PPA encompasses a range of orientations and topics including government, public policy, public management, public administration, and political science.The field can broadly be conceived as oriented towards improving understandings of policymaking and public administration and supplying decision makers with reliable policy-and administration-relevant knowledge about economic and social problems (Fischer, Miller, & Sidney, 2006).While the goals of individual experts are undoubtedly varied, insight into the overarching aims of the field is provided by professional associations and journals.One leading association, the International Research Society for Public Management, aims to facilitate "the creation and dissemination of new knowledge and understanding across [the international research] community and into policy and practice" (IRSPM 2023).Along the same lines, Public Administration Review, the official publication of the American Society for Public Administration, seeks to "identify and analyse current trends, provide a factual basis for decision making, stimulate discussion, and make the leading literature in the field available in an easily accessible format" in order to serve "academics, practitioners, and students" (Public Administration Review, 2023).Similarly, Canadian Public Policy is "directed at a wide readership including decision makers and advisers in business organizations and governments, and policy researchers" (Canadian Public Policy, 2023).Thus, experts in the field seek to go beyond academic impact to have influence in the policy and public management sphere.Yet, the influence of PPA scholars has received little systematic attention.
In this study, we deal with the question of how PPA research and policy are connected.Although reliable data on the connection between research and policy have historically been hard to find, the introduction of the Overton database in 2019 (Szomszor & Adie, 2022) has allowed analysis of research papers that are cited in policy documents.Policy documents have been defined as "'carriers' of policies . . .[that] provide a channel through which policy science researchers can study the main contents of policies, policymaking processes, and policy instruments" (Yang, Huang, & Su, 2020).In this empirical study, we use the Overton database to find policy documents that cited PPA research.

Dataset
We used a list of 49 PPA journals indexed in the Web of Science (WoS, Clarivate) as basis, downloaded from https://mjl.clarivate.com/ on 01/10/2022.We used the bibliometric in-house database of the Max Planck Society (MPG).We restricted the dataset to the publication years 1980-2019.Most publications belong to one of the following document types: article, book review, editorial or review.Thus, we restricted our analysis to these four document types.Furthermore, we restricted the dataset to the journals that published at least 50% of the publications with a DOI among these four document types within the time period 1980-2019.This procedure removed six journals from our dataset (Amme Idaresi Dergisi, Civil Szemle, Reforma y Democracia, Gestión y Política Pública, Transylvanian Review of Administrative Sciences, and Journal of Homeland Security and Emergency Management).We also removed two other journals (Climate Policy and Environment and Planning C -Politics and Space) that have a key focus on environmental studies (and are classified in the WoS Subject Category Environmental Studies).We expected that these two journals would distort our dataset from PPA too much.We kept the journal Science and Public Policy although it is also assigned to Environmental Studies because it is also assigned to the WoS Subject Category Management.Management is a key area of overlap for public management research.This left us with a list of 41 journals that are shown in Table 1.Some DOIs (n=459) occurred multiple times.The corresponding papers (n=1,352) were removed from our dataset.

Methodology
We used PostgreSQL and R (R Core Team, 2019) commands including the R package 'tidyverse' (Wickham, 2017) for data analysis.

Number of documents
Figure 1 shows the distribution of the number of PPA publications and the number of policy documents that cited these publications across the years.Overall, the proportion of PPA publications cited at least once by a policy document was rather constant in the 1980ies.In the 1990ies, this proportion increased and reached a maximum of about 0.45 in 2005.It has been decreasing since then.One explanation could be that the interest in scientific papers has decreased since 2005 in the policy sphere.Another explanation could be that it takes rather long until new scientific findings make it into public policy.The number of policy documents that cited PPA publications was very low until 1997 but has increased seemingly exponential since then.Already in 2006, more policy documents that cited PPA publications were published than PPA publications that were cited at least once by a policy document.In 2015, more policy documents that cited PPA publications were published than PPA publications.2 shows the percentage of policy documents per policy sector for all policy documents in the Overton database in the left panel and for the policy documents that cited PPA publications in the right panel.The proportion of policy documents that cited a PPA publication belonging to the governmental sector is much lower than for the full Overton database, although PPA publications should be of especial interest for the governmental sector.Intergovernmental organizations (IGOs) and think tanks are much more frequently the sources of policy papers that cited a PPA publication in comparison to the full Overton database.Table 4 shows the most productive policy sources grouped by policy sector.National governments produce a large volume of policy documents, but relatively few policy documents that rely on PPA publications.However, some key exceptions include large supranational organizations, such as the Publications Office of the European Union, and IGOS including OECD and World Bank, which publish many policy documents that cite PPA research.Think tanks generally produce fewer policy documents, but a higher proportion of these cite PPA research.For example, 8.4% of policy documents published by Germany's IZA Institute of Labor Economics engage with PPA discourse.

Discussion
The interpretation of the policy impact of research can only be undertaken based on an important caveat that policy sources capture two distinct types of documents: those that are authored by scientists to inform decision making, and those that are authored by policymakers to inscribe policy decisions or legislation.The distinction between these types of documents explains some of the results here.For example, some organizations that seek to inform policy decisions, such as IPCC and PRUComm, will include many references to provide a robust evidence base.On the other hand, policy briefs or ministerial notes by governments or intergovernmental organizations may intentionally limit literature sources to improve accessibility or to conform to policy conventions.The policy impact analysis in this study, therefore to some extent highlights different policy document types that may be favoured by certain organization types.Interestingly, PPA book reviews are more frequently cited by policy than by WoS papers in general, perhaps suggesting that highly accessible formats that present concise or synthesised evidence are favoured over more comprehensive formats in this field.
In this study, we examined several aspects of the connection between PPA research and policy.Focusing on the connection over time reveals that while the number of PPA papers and the number of policy documents that cite them have increased in recent years, the proportion of PPA research in policy seems to have peaked in the mid-2000s.This potentially reflects a declining interest in the PPA literature within the policy sphere or a lag between new evidence and its translation into policy.
Our results show that governments and intergovernmental organizations produce a high number of policy documents, but cite rather few papers from PPA research.This suggests that, in general, PPA research is less influential on decision makers and advisers in international organizations and governments, than on policy researchers within think tanks.Think tanks seem to produce fewer policy documents overall, but that are more closely related to PPA research.This is potentially due to the brokerage function of think tanks, whereby they seek to understand how to actively move expert knowledge into the policy sphere (Abelson, 2009).However, our results show some exceptions to this.Some IGOs, including the Publication Offices of the European Union, OECD, and World Bank, are very active in the PPA discourse.National governments in particular are much less likely to be the source of policy papers that cited a WoS PPA publication in comparison to the full Overton database.This is reflected in the most productive government sector source of documents that cited PPA research being the intergovernmental Publications Office of the European Union by a large margin.This may point to a potential disconnect between national government decision making and PPA research, which does not reflect the goals of many PPA journals and associations.Alternatively, it may reflect complex methodological challenges around policy impact.

Conclusions
This study highlights the dual challenges of creating and measuring policy impact.One challenge of much of the detailed synthesis work in PPA research is not directly used in policymaking.This means that attempts to provide a solid evidence base may not be as well received by governmental decision makers as more accessible formats.It also produces particular challenges around measurement, whereby documents that inscribe actual policy decisions typically do not contain many citations, making policy impact difficult to capture.The findings here provide support for the idea that the PPA literature is further built upon and brokered by think tanks (and to a lesser extent, IGOs), who then help translate this knowledge into policy briefs, policy reports, briefing notes or ministerial expertise.Final policy decisions made by governments typically do not contain any citations.Thus, PPA research may have different impact for different organizational types, which is complicated because it becomes increasingly challenging to measure impact the closer we get to meaningful decisions.

Open science practices
We are not aware of an open database of policy documents.Thus, we are restricted to using a closed dataset.

Figure 1 :
Figure 1: Number of PPA publications and number of policy documents that cited these publications across the years

Figure 2 :
Figure 2: Percentage of policy documents per policy sector

Figure 3 :
Figure 3: Percentage of policy documents that cited PPA publications versus number of all policy documents in the Overton database (an interactive version can be accessed at: https://s.gwdg.de/r930Yl)

Table 1 .
PPA journals that were included in our study ordered descending by the number of papers with DOI

Table 2
(Szomszor & Adie, 2022)of the WoS PPA publications across the document types.Overall, the relative frequency of document types is similar to the total WoS.The only exception are book reviews with a significantly higher proportion among PPA publications than among the total WoS.We used a snapshot of the Overton database(Szomszor & Adie, 2022)from 19 January 2023 to obtain information about which policy sources cited which selected PPA journals.This snapshot has been imported into a local PostgreSQL database at the Max Planck Institute for Solid State Research (Stuttgart, Germany).Both databases (WoS and Overton) were linked via the DOIs of the scientific publications.

Table 3 .
Percentages of WoS PPA papers cited either by WoS papers or policy documents broken down by the WoS document types ordered by the percentages of WoS PPA papers cited by policy documents

Table 4 :
Types of policy sources most productive in publishing policy documents that cited WoS PPA publications in comparison with all policy documents