Platform logo
Explore Communities
27th International Conference on Science, Technology and Innovation Indicators (STI 2023) logo
27th International Conference on Science, Technology and Innovation Indicators (STI 2023)Community hosting publication
There is an updated version of this publication, open Version 2.
conference paper

How relevant is public policy and administration research for the policy sector? An empirical analysis based on Overton data

20/04/2023| By
Robin Robin Haunschild,
+ 1
Lutz Lutz Bornmann
1476 Views
0 Comments
Disciplines
Keywords
Abstract

A key goal of public policy and public administration research is to inform policy decisions. It is not clear, however, to what extent this is the case. In this study, therefore, citations from policy documents to public policy and administration research were analyzed to identify which research contributed most to policy reports and decisions. Additionally, we identified which policy institutions used research literature more than others to justify their policy decisions. Our findings show that think tanks use public policy and administration research literature more often than governmental organizations when justifying policy reports and decisions.

Preview automatically generated form the publication file.

How relevant is public policy and administration research for the policy sector? An empirical analysis based on Overton data

Robin Haunschild*, Kate Williams**, and Lutz Bornmann*,***

* R.Haunschild@fkf.mpg.de; L.Bornmann@fkf.mpg.de

0000-0001-7025-7256; 0000-0003-0810-7091

Information Retrieval Service (IVS-CPT), Max Planck Institute for Solid State Research, Germany

** kate.williams@unimelb.edu.au

0000-0002-2882-1068

School of Social and Political Sciences, University of Melbourne, Australia

*** Lutz.Bornmann@gv.mpg.de

0000-0003-0810-7091

Science Policy and Strategy Department, Administrative Headquarters of the Max Planck Society, Germany

A key goal of public policy and public administration research is to inform policy decisions. It is not clear, however, to what extent this is the case. In this study, therefore, citations from policy documents to public policy and administration research were analyzed to identify which research contributed most to policy reports and decisions. Additionally, we identified which policy institutions used research literature more than others to justify their policy decisions. Our findings show that think tanks use public policy and administration research literature more often than governmental organizations when justifying policy reports and decisions.

Introduction

Expertise has become an integral part of policymaking processes around the world. Accordingly, there is now a substantial literature on expert knowledge and policy processes. Much of this work has focused on the dual logics of science and politics and the various ways these complement or contradict each other (Jasanoff 1990). Attempts to conceptualise the influence of expert knowledge on policymaking have been fragmented (Christensen, 2021). The knowledge utilisation literature, from the field of public policy, has centred the use of information or research in policymaking. This includes evidence-based policymaking (Head, 2016), which aims to understand how evidence is taken up in policy design, as well as the strategic and symbolic use of evidence to gain political legitimacy (Boswell 2009). The field of political science, by contrast, tends to focus on the overarching ideas that arise from academic research and how they shape public policies (Hall, 1989). Other fields including Science and Technology Studies (STS) and sociology have focused on how individuals or groups of experts provide meaningful knowledge to decision-makers, for example, considering the role of professions (Abbott, 1988), networks (Haas, 1992) or knowledge brokers (Sverrisson, 2001). Yet, in part due to the methodological difficulties of capturing influence on audiences outside the academy, there remains limited empirical work on the impact of expert knowledge on policymaking.

One source of expert knowledge that has particular bearing on policymaking is the scholarship in the area of public policy and administration (PPA) itself. The field of PPA encompasses a range of orientations and topics including government, public policy, public management, public administration, and political science. The field can broadly be conceived as oriented towards improving understandings of policymaking and public administration and supplying decision makers with reliable policy- and administration-relevant knowledge about economic and social problems (Fischer, Miller, & Sidney, 2006). While the goals of individual experts are undoubtedly varied, insight into the overarching aims of the field is provided by professional associations and journals. One leading association, the International Research Society for Public Management, aims to facilitate “the creation and dissemination of new knowledge and understanding across [the international research] community and into policy and practice” (IRSPM 2023). Along the same lines, Public Administration Review, the official publication of the American Society for Public Administration, seeks to “identify and analyse current trends, provide a factual basis for decision making, stimulate discussion, and make the leading literature in the field available in an easily accessible format” in order to serve “academics, practitioners, and students” (Public Administration Review, 2023). Similarly, Canadian Public Policy is “directed at a wide readership including decision makers and advisers in business organizations and governments, and policy researchers” (Canadian Public Policy, 2023). Thus, experts in the field seek to go beyond academic impact to have influence in the policy and public management sphere. Yet, the influence of PPA scholars has received little systematic attention.

In this study, we deal with the question of how PPA research and policy are connected. Although reliable data on the connection between research and policy have historically been hard to find, the introduction of the Overton database in 2019 (Szomszor & Adie, 2022) has allowed analysis of research papers that are cited in policy documents. Policy documents have been defined as “‘carriers’ of policies . . . [that] provide a channel through which policy science researchers can study the main contents of policies, policymaking processes, and policy instruments” (Yang, Huang, & Su, 2020). In this empirical study, we use the Overton database to find policy documents that cited PPA research.

Dataset and Methodology

2.1 Dataset

We used a list of 49 PPA journals indexed in the Web of Science (WoS, Clarivate) as basis, downloaded from https://mjl.clarivate.com/ on 01/10/2022. We used the bibliometric in-house database of the Max Planck Society (MPG). We restricted the dataset to the publication years 1980-2019. Most publications belong to one of the following document types: article, book review, editorial or review. Thus, we restricted our analysis to these four document types. Furthermore, we restricted the dataset to the journals that published at least 50% of the publications with a DOI among these four document types within the time period 1980-2019. This procedure removed six journals from our dataset (Amme Idaresi Dergisi, Civil Szemle, Reforma y Democracia, Gestión y Política Pública, Transylvanian Review of Administrative Sciences, and Journal of Homeland Security and Emergency Management). We also removed two other journals (Climate Policy and Environment and Planning C – Politics and Space) that have a key focus on environmental studies (and are classified in the WoS Subject Category Environmental Studies). We expected that these two journals would distort our dataset from PPA too much. We kept the journal Science and Public Policy although it is also assigned to Environmental Studies because it is also assigned to the WoS Subject Category Management. Management is a key area of overlap for public management research. This left us with a list of 41 journals that are shown in Table 1. Some DOIs (n=459) occurred multiple times. The corresponding papers (n=1,352) were removed from our dataset.

Table 1. PPA journals that were included in our study ordered descending by the number of papers with DOI

Journal Number of papers with DOI
Public Administration Review 3,926
Journal of Social Policy 2,872
Canadian Public Policy 2,480
Public Administration 1,974
Journal of Policy Analysis and Management 1.962
Australian Journal of Public Administration 1.813
Policy Studies Journal 1.670
Social Policy and Administration 1.659
Local Government Studies 1.648
Public Money and Management 1.643
Public Administration and Development 1.632
Journal of European Public Policy 1.564
Administration and Society 1.343
Public Personnel Management 1.332
Canadian Public Administration 1.220
Policy and Politics 1.168
International Review of Administrative Sciences 1.074
Contemporary Economic Policy 1.040
Governance 958
The American Review of Public Administration 920
Science and Public Policy 901
Public Management Review 875
Policy Sciences 832
Journal of Public Administration Research and Theory 817
Journal of European Social Policy 808
Review of Policy Research 592
Journal of Accounting and Public Policy 547
Policy and Society 453
Lex Localis 420
Journal of Comparative Policy Analysis – Research and Practice 414
Policy Studies 410
Regulation and Governance 356
Public Performance and Management Review 349
Nonprofit Management and Leadership 339
International Public Management Journal 335
Review of Public Personnel Administration 285
Human Service Organizations Management, Leadership and Governance 237
Journal of Public Policy 201
Journal of Chinese Governance 120
Public Policy and Administration 114
Critical Policy Studies 110

Table 2 shows the distribution of the WoS PPA publications across the document types. Overall, the relative frequency of document types is similar to the total WoS. The only exception are book reviews with a significantly higher proportion among PPA publications than among the total WoS.

Table 2. Document types of WoS PPA publications that were included in our study ordered descending by the number of papers with DOI

Document type Number of papers with DOI Proportion of papers Proportion of papers in WoS
Article 32,624 75.15 82.18
Book review 6,899 15.89 8.03
Editorial 3,193 7.35 5.81
Review 697 1.61 3.99

We used a snapshot of the Overton database (Szomszor & Adie, 2022) from 19 January 2023 to obtain information about which policy sources cited which selected PPA journals. This snapshot has been imported into a local PostgreSQL database at the Max Planck Institute for Solid State Research (Stuttgart, Germany). Both databases (WoS and Overton) were linked via the DOIs of the scientific publications.

2.2 Methodology

We used PostgreSQL and R (R Core Team, 2019) commands including the R package ‘tidyverse’ (Wickham, 2017) for data analysis.

3. Results

3.1 Number of documents

Figure 1 shows the distribution of the number of PPA publications and the number of policy documents that cited these publications across the years. Overall, the proportion of PPA publications cited at least once by a policy document was rather constant in the 1980ies. In the 1990ies, this proportion increased and reached a maximum of about 0.45 in 2005. It has been decreasing since then. One explanation could be that the interest in scientific papers has decreased since 2005 in the policy sphere. Another explanation could be that it takes rather long until new scientific findings make it into public policy. The number of policy documents that cited PPA publications was very low until 1997 but has increased seemingly exponential since then. Already in 2006, more policy documents that cited PPA publications were published than PPA publications that were cited at least once by a policy document. In 2015, more policy documents that cited PPA publications were published than PPA publications.

Figure 1: Number of PPA publications and number of policy documents that cited these publications across the years

Table 3 shows the percentages of PPA papers (cited either by papers or policy documents) in comparison with their uncited counterparts. The PPA reviews are less frequently cited than reviews in general whereas PPA book reviews and editorials are more frequently cited than book reviews and editorials in general. Overall, PPA publications are less frequently cited by policy documents than by WoS papers except for book reviews.

Table 3. Percentages of WoS PPA papers cited either by WoS papers or policy documents broken down by the WoS document types ordered by the percentages of WoS PPA papers cited by policy documents

Document type Percentage of WoS PPA papers cited by WoS papers Percentage of WoS PPA papers cited by policy documents Percentage of WoS papers cited by WoS papers
Article 87.93 36.26 88.43
Review 79.91 35.15 94.12
Editorial 52.08 13.40 40.82
Book review 6.52 4.73 3.01

3.2 Sectors and sources of the policy documents

Figure 2 shows the percentage of policy documents per policy sector for all policy documents in the Overton database in the left panel and for the policy documents that cited PPA publications in the right panel. The proportion of policy documents that cited a PPA publication belonging to the governmental sector is much lower than for the full Overton database, although PPA publications should be of especial interest for the governmental sector. Intergovernmental organizations (IGOs) and think tanks are much more frequently the sources of policy papers that cited a PPA publication in comparison to the full Overton database.

Figure 2: Percentage of policy documents per policy sector

Figure 3 shows a scatter plot with the percentage of policy documents that cited PPA publications against the number of all policy documents in the Overton database. The individual points are coloured by their policy sector. Amongst the organizations with high proportions of policy documents citing PPA papers, there are a variety of research areas, political orientations and proximities to policymaking. Some take on advocacy roles, such as the Thomas B. Fordham Institute, which is a conservative American non-profit think tank with a specific mission around education reform. Others such as the Council of Canadian Academies, the Institute for Research on Public Policy, and MySociety bring together expert research to inform public understanding and decision making on a wide range of issues. Some, such as the Centre for Analysis of Social Exclusion and the UCL Institute for Innovation and Public Purpose are university-based multi-disciplinary research centres. Other organizations have closer links to policy, such as PRUComm, which is funded by the UK’s National Institute of Health Research to provide evidence for the Department of Health and Social Care, and the Intergovernmental Panel on Climate Change (IPCC), which is the United Nations body for assessing the science related to climate change. Two of these, SAPEA and the Swedish Institute for European Policy Studies (SIEPS), are independent government agencies. SAPEA provides independent scientific advice to the European Commission to support decision-making, and SIEPS is an independent government agency for research and analysis of European policy affairs.

Figure 3: Percentage of policy documents that cited PPA publications versus number of all policy documents in the Overton database (an interactive version can be accessed at: https://s.gwdg.de/r930Yl)

Table 4 shows the most productive policy sources grouped by policy sector. National governments produce a large volume of policy documents, but relatively few policy documents that rely on PPA publications. However, some key exceptions include large supranational organizations, such as the Publications Office of the European Union, and IGOS including OECD and World Bank, which publish many policy documents that cite PPA research. Think tanks generally produce fewer policy documents, but a higher proportion of these cite PPA research. For example, 8.4% of policy documents published by Germany’s IZA Institute of Labor Economics engage with PPA discourse.

Table 4: Types of policy sources most productive in publishing policy documents that cited WoS PPA publications in comparison with all policy documents

All policy documents Policy documents that cited PPA papers
Policy source Number of policy documents Policy source Number of policy documents
Government
Government of Japan 253,158 Publications Office of the European Union 1,395
State of Texas 126,302 The UK Government 359
State of Hawaii 116,530 Government of Canada 350
State of Maryland 102,252 European Parliamentary Research Service 316
EUR-Lex 101,597 US Office of Planning, Research, and Evaluation 220
French Government Ministries 99,898 European Commission Joint Research Centre 192
State of Colorado 97,783 US Government Publishing Office 168
State of California 93,269 Government of Finland 149
State of Washington 92,403 Banca D'Italia 143
City of New York 86,365 UK Parliament Select Committee Publications 126
IGO
World Health Organization 212,326 OECD 1,674
UNESCO 48,754 World Bank 1,165
Organization for Security and Co-operation in Europe 32,143 Inter-American Development Bank 505
World Bank 31,991 World Health Organization 421
Food and Agriculture Organization of the United Nations 27,069 UNESCO 206
OECD 24,198 Asian Development Bank 177
United Nations 23,361 International Monetary Fund 165
International Monetary Fund 14,318 United Nations CEPAL 142
Inter-American Development Bank 13,398 United Nations Environment Programme 94
United Nations CEPAL 8,900 Food and Agriculture Organization of the United Nations 92
Think tank
blogs.lse.ac.uk (UK) 25,794 IZA Institute of Labor Economics (Germany) 1,358
National Bureau of Economic Research (US) 21,033 National Bureau of Economic Research (US) 999
Brookings Institute (US) 18,283 RAND Corporation 537
International Development Research Centre (Canada) 16,700 Institute of Development Studies (UK) 463
IZA Institute of Labor Economics (Germany) 16,145 Ifo Institute for Economic Research (Germany) 365
Center for Strategic and International Studies (US) 15,810 Brookings Institute (US) 300
Acton Institute (US) 15,013 International Development Research Centre (Canada) 280
Heritage Foundation (US) 14,689 blogs.lse.ac.uk (UK) 271
American Civil Liberties Union (US) 14,262 Overseas Development Institute (UK) 237
Foundation for Economic Education (US) 14,120 DIW German Institute for Economic Research (Germany) 225
Others
Analysis & Policy Observatory 16,733 Analysis & Policy Observatory 880
Guidelines in PubMed Central 14,496 Guidelines in PubMed Central 30
AI Regulation Special Collection 391 AI Regulation Special Collection 14

4. Discussion

The interpretation of the policy impact of research can only be undertaken based on an important caveat that policy sources capture two distinct types of documents: those that are authored by scientists to inform decision making, and those that are authored by policymakers to inscribe policy decisions or legislation. The distinction between these types of documents explains some of the results here. For example, some organizations that seek to inform policy decisions, such as IPCC and PRUComm, will include many references to provide a robust evidence base. On the other hand, policy briefs or ministerial notes by governments or intergovernmental organizations may intentionally limit literature sources to improve accessibility or to conform to policy conventions. The policy impact analysis in this study, therefore to some extent highlights different policy document types that may be favoured by certain organization types. Interestingly, PPA book reviews are more frequently cited by policy than by WoS papers in general, perhaps suggesting that highly accessible formats that present concise or synthesised evidence are favoured over more comprehensive formats in this field.

In this study, we examined several aspects of the connection between PPA research and policy. Focusing on the connection over time reveals that while the number of PPA papers and the number of policy documents that cite them have increased in recent years, the proportion of PPA research in policy seems to have peaked in the mid-2000s. This potentially reflects a declining interest in the PPA literature within the policy sphere or a lag between new evidence and its translation into policy.

Our results show that governments and intergovernmental organizations produce a high number of policy documents, but cite rather few papers from PPA research. This suggests that, in general, PPA research is less influential on decision makers and advisers in international organizations and governments, than on policy researchers within think tanks. Think tanks seem to produce fewer policy documents overall, but that are more closely related to PPA research. This is potentially due to the brokerage function of think tanks, whereby they seek to understand how to actively move expert knowledge into the policy sphere (Abelson, 2009). However, our results show some exceptions to this. Some IGOs, including the Publication Offices of the European Union, OECD, and World Bank, are very active in the PPA discourse. National governments in particular are much less likely to be the source of policy papers that cited a WoS PPA publication in comparison to the full Overton database. This is reflected in the most productive government sector source of documents that cited PPA research being the inter-governmental Publications Office of the European Union by a large margin. This may point to a potential disconnect between national government decision making and PPA research, which does not reflect the goals of many PPA journals and associations. Alternatively, it may reflect complex methodological challenges around policy impact.

5. Conclusions

This study highlights the dual challenges of creating and measuring policy impact. One challenge of much of the detailed synthesis work in PPA research is not directly used in policymaking. This means that attempts to provide a solid evidence base may not be as well received by governmental decision makers as more accessible formats. It also produces particular challenges around measurement, whereby documents that inscribe actual policy decisions typically do not contain many citations, making policy impact difficult to capture. The findings here provide support for the idea that the PPA literature is further built upon and brokered by think tanks (and to a lesser extent, IGOs), who then help translate this knowledge into policy briefs, policy reports, briefing notes or ministerial expertise. Final policy decisions made by governments typically do not contain any citations. Thus, PPA research may have different impact for different organizational types, which is complicated because it becomes increasingly challenging to measure impact the closer we get to meaningful decisions.

Open science practices

We are not aware of an open database of policy documents. Thus, we are restricted to using a closed dataset.

Acknowledgments

The bibliometric data used in this study are from a bibliometric in-house database of the Max Planck Society (MPG), developed and maintained in cooperation with the Max Planck Digital Library (MPDL, Munich), derived from the Science Citation Index Expanded (SCI-E), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI) prepared by Clarivate (Philadelphia, Pennsylvania, USA) via the “Kompetenznetzwerk Bibliometrie” (see https://bibliometrie.info/en/about-kb/) funded by BMBF (grant 16WIK2101A). The policy document data used in this study are from an Overton snapshot provided by Euan Adie on 19 January 2023.

Author contributions

Conceptualization: LB & KW

Data curation: RH & KW

Investigation: RH & KW

Methodology: RH, KW, & LB

Visualization: RH

Writing – original draft: RH & KW

Writing – review & editing: RH, KW, & LB

Competing interests

The authors declare that they have no competing interests.

Funding information

Access to bibliometric data was enabled via grant 16WIK2101A.

References

Abbott, A. (1988). The system of professions: An essay on the division of expert labor. London: University of Chicago Press.

Abelson, D. E. (2009). Do think tanks matter? Assessing the impact of public policy institutes introduction. Kingston: Queen's School of Policy Studies.

Boswell , C. (2009). The political uses of expert knowledge: Cambridge University Press.

Canadian Public Policy. (2023). Canadian Public Policy. Retrieved March 10, 2023, from https://www.utpjournals.press/loi/cpp

Christensen, J. (2021). Expert knowledge and policymaking: A multi-disciplinary research agenda. Policy & Politics, 49(3), 455-471. doi: https://dx.doi.org/10.1332/030557320x15898190680037.

Fischer, F., Miller, G., & Sidney, M. (2006). Handbook of public policy analysis: Theory, politics, and methods: CRC Press.

Haas, P. M. (1992). Introduction: Epistemic communities and international policy coordination. International Organization, 46(1), 1-35. doi: https://dx.doi.org/10.1017/S0020818300001442.

Hall, P. A. (1989). Introduction. In P. A. Hall (Ed.), The political power of economic ideas (pp. 1-26). Princeton: Princeton University Press.

Head, B. W. (2016). Toward more “evidence-informed” policy making? Public Administration Review, 76(3), 472-484. doi: https://doi.org/10.1111/puar.12475.

IRSPM (2023). International Research Society for Public Management. Retrieved March 10, 2023, from https://www.irspm.org/

Jasanoff , S. (1990). The fifth branch. Science advisers as policymakers. Cambridge, MA: Harvard University Press.

Public Administration Review. (2023). Public Administration Review - Wiley Online Library. Retrieved March 10, 2023, from https://onlinelibrary.wiley.com/journal/15406210

R Core Team. (2019). R: A language and environment for statistical computing (Version 3.6.0). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.r-project.org/

Sverrisson, Á. (2001). Translation networks, knowledge brokers and novelty construction: Pragmatic environmentalism in Sweden. Acta Sociologica, 44(4), 313-327.

Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624-650. doi: 10.1162/qss_a_00204.

Wickham, H. (2017). Tidyverse: Easily install and load the 'Tidyverse'. R package version 1.2.1. Retrieved 22 June 2020, from https://CRAN.R-project.org/package=tidyverse

Yang, C., Huang, C., & Su, J. (2020). A bibliometrics-based research framework for exploring policy evolution: A case study of China's information technology policies. Technological Forecasting and Social Change, 157, 120116. doi: https://doi.org/10.1016/j.techfore.2020.120116.

Figures (3)

Publication ImagePublication ImagePublication Image

No comments published yet.

Submitted by20 Apr 2023
Download Publication

No reviews to show. Please remember to LOG IN as some reviews may be only visible to specific users.

ReviewerDecisionType
User Avatar
Hidden Identity
Accepted
Peer Review
User Avatar
Hidden Identity
Accepted
Peer Review
User Avatar
Hidden Identity
Minor Revision
Peer Review