Through the Secret Gate: A Study of Member-Contributed Submissions in PNAS

Jens Peter Andersen^*, Serge P.J.M. Horbach^** and Tony Ross-Hellauer^***

^*jpa@ps.au.dk

0000-0003-2444-6210
Danish Centre for Studies in Research and Research Policy, Aarhus University, Denmark.

^** s.horbach@ps.au.dk

0000-0003-0406-6261

Danish Centre for Studies in Research and Research Policy, Aarhus University, Denmark

^*** ross-hellauer@tugraz.at

0000-0003-4470-7027

Institute of Interactive Systems and Data Science, Graz University of Technology, Austria

Abstract

This work studies “Contributed” articles in the Proceedings of the National Academy of Sciences of the United States of America (PNAS), a streamlined submission track for members of the US National Academy of Sciences (NAS). We assess the nature and impact of those articles and the background and status of their authors and contributors. Analysing over 46,000 articles published 2007-2020, we find that: (1) PNAS-Contributed articles tend to spend less time in the review process than Direct submissions; (2) Direct submissions tend to be slightly higher cited than Contributed articles, which are particularly overrepresented amongst least-cited PNAS papers; (3) PNAS-Contributed articles generally appear in lower per-author citation deciles than Direct submissions, but are more likely to appear in the overall top citation deciles of authors; (4) authors with lower mean normalised citation scores are profiting most from articles published as Contributed papers, in terms of citation impact.

1. Introduction

Despite the ongoing evolution of scholarly communication, journal publications, especially those in ‘high impact’ or prestigious journals, still form the core of the scientific communication and reward system. While novel publication and assessment procedures are being developed, journal articles containing prestigious journals’ seals of approval continue to be the main mechanism of sharing knowledge and comprise the prime currency of academic reward. Even though the relationship between study quality, research impact and publication venue has long been questioned, researchers remain eager to get their work published in prestigious journals.

However, the reliance on journal publications, especially those in high-impact or prestigious journals, has led to a number of problems. These include that it can lead to a focus on sensational or "sexy" research rather than high-quality research addressing societally relevant issues. Additionally, the peer-review process for these journals is often highly competitive and can lead to slow publication processes, with somewhat arbitrary decisions on what gets to be published. Furthermore, the emphasis on these publications can also lead to a lack of diversity in the scientific community, as researchers from underrepresented groups may have more difficulty getting their work published in these journals.

Therefore, the gatekeeping mechanisms deciding who or what gets published on prestigious journals’ pages have been the subject of extensive debate and study. While historically continuously being the centre of critique for being biased, slow, and fallible in many ways, peer review is still considered the best selection mechanism available and undoubtedly the most commonly used one. However, some journals have long had alternative publication models in which a selected group of authors can partly or fully bypass peer review processes. While these mechanisms contribute considerable outputs in some of science’s most prestigious outlets, hence prominently impacting both the scholarly literature and academic reward processes, they have only been minimally subjected to empirical analyses.

This study addresses this knowledge gap by studying “Contributed” articles in the Proceedings of the National Academy of Sciences of the United States of America (PNAS). Through this track, members of the US National Academy of Sciences (NAS) can “contribute” up to two articles every year via a streamlined process, allowing them more control over the review process. Our study assesses the nature and impact of those articles and the background and status of their authors and contributors. This paper reports ongoing work to investigate these issues, focusing on the duration of review processes, relative numbers of citations across submission tracks and disciplines, and comparison of citation performance of author’s PNAS Contributed or Direct submission papers with papers published in other prestigious outlets. Additional work is continuing, which will be completed for presentation to the STI conference, to investigate differences in topics or themes of Contributed/Direct submission, differences in contributor roles, and co-authorship networks across these tracks.

2. Background

Peer review, evolving out of editorial oversight at community or society-led journals, became a “gold standard” in most journals in the latter half of the 20th Century (Baldwin, 2017). At least as an ideal, a scholar’s status came to be judged irrelevant to the assessment of individual pieces of research. Formalised as the Mertonian norm of universalism (Merton, 1973), judgements of a manuscript’s merits were supposed to be blind to its authors' status, demographics or background. Nevertheless, despite a host of innovations in peer review procedures and formats, particularly including various forms of anonymisation (Horbach & Halffman, 2018), the system has continuously been critiqued for failing to support Merton’s universalistic ideal (e.g. Jukola, 2017). Multiple journals have therefore struggled to find an appropriate balance between systems based on trust and efficient publishing on the one hand while aiming to prevent favouritism on the other. Relatedly, the extent to which editors should be allowed to publish their work in the journals they work for has been extensively debated. A recent study shows that editors tend to publish substantially in their journals, especially men (Liu et al., 2023).

PNAS, as the official journal of the National Academy of Sciences, provides a favoured position to members of its society. Founded in 1914, PNAS aimed to represent a “comprehensive survey of the more important results of the scientific research of this country”, as stated in the first issue by Edwin Bidwell Wilson, inaugural PNAS managing editor (Garfield, 1987). NAS members were able to submit directly and usually forgo peer review. Non-members could submit via an NAS member, who would then choose the reviewers and oversee editorial processes. In 1995, a “Direct submission” track was created whereby sponsorship from NAS members was no longer necessary. By 2007, Direct submissions accounted for 50% of articles published and 84% of those submitted (Schekman, 2009).

In 2010, PNAS removed the “communicated” track, whereby non-NAS members could submit via an NAS member, officially, mainly due to a dwindling number of papers submitted via this track (Schekman, 2009). However, many NAS members and outsiders saw the change motivated at least in part by an attempt to mitigate accusations of “cronyism” (Kean, 2009). NAS members retained their privilege to “contribute” up to two articles of their own per year. These Contributed articles go through a somewhat different editorial process. The contributing member submits the manuscript along with the names of peers who have agreed to review the work. After a brief assessment by the editorial board, the suggested referees are assigned. After acceptance, the names of the contributing NAS member and the reviewers are mentioned in the article’s byline. Justifying the decision to maintain this member-contributed track, then-Editor-in-Chief Randy Schekman explained that Contributed articles were amongst the journal’s most cited and that this track also incentivised active contribution to running the journal from NAS members (Kean, 2009). The contributor track has, however, attracted criticisms of cronyism, being “anachronistic” and giving the journal “the appearance of an old boys' club” (Aldhous, 2014).

Two previous studies have touched upon the issues we investigate. In their scientometric analyses of PNAS papers published across all tracks, Rand and Pfeiffer (2009) found that overall Contributed papers tended to garner fewer citations than Direct submission, but that the top 10% of Contributed submissions garnered higher levels of citations than the top decile of Direct submissions They hence concluded that PNAS-Contributed papers, “balance an overall lower impact with an increased probability of publishing exceptional papers” (Rand and Pfeiffer, 2009).

A preprint from Davis (2016) further compared citation performance across PNAS tracks to that point and found general underperformance (9% fewer citations) of Contributed papers when compared to Direct submissions. Papers from Social Sciences had the largest gap (12% fewer citations). However, Davis found that the effect had lessened over the years, from 13.6% (2005) to just 2.2% fewer citations (2014). Davis suggested that this closing of the citation gap was attributable to persistent moves to tighten editorial scrutiny of Contributed papers in the preceding years: “Successive editorial policies placing limits, restrictions, and other qualifications on the publication privileges of NAS members may be responsible for the submission of better performing Contributed papers.” (Davis, 2016).

3. Methods and Data

Publication meta-data on articles published in PNAS from 2007 to 2020 was made available in XML format by the PNAS editorial team. Some information, such as publication dates, author names and DOI, were available directly from XML, whereas information on submission type, submission date and acceptance date was extracted from structured text fields. A total of 49,089 articles were published in this period; however, 2,222 were other types (e.g. communications and inaugural articles). Of the remaining 46,867 articles, 35,878 (76.6%) were Direct submissions and 10,989 (23.4%) were Contributed submissions. Article metadata were matched through DOI to Web of Science publication records (WoS) to allow citation analysis and comparisons to the authors' oeuvres. A total of 46,634 (99.5%) records were matched, hereof 35,693 (76.5%) Direct submissions and 10,941 (23.5%) Contributed submissions.

PNAS publishes articles in sections according to the broad topic of articles. We identified 33 distinct sections containing the above articles, with the majority being special features containing, on average, eight articles each. The three main sections accounted for 46,391 (99.5%) articles, divided between biological sciences (n = 36,050, 77.7%), physical sciences (n = 7,976, 17.2%) and social sciences (n = 2,365, 5.1%). We only use these three sections in the analyses.

Matching authors from PNAS to WoS records and subsequently to their non-PNAS publications is not trivial. Although one should expect names to be identical, some information is lost during the conversion from journal metadata to registrations in WoS. Using the Damerau-Levenshtein distance combined with the position in the author sequence, we can match 93.8% of names where the sequence position is equal, and the Damerau-Levenshtein distance is less than four. A manual inspection at this level found no false positive matches in 100 couplings. The at-most three edits (distance ≤ 3) mostly covered changes from local special characters to standard English letters.

We use the results of D'Angelo and van Eck's (2020) automatic author disambiguation algorithm to create profiles of publications likely to match those of the actual, real authors. This algorithm generally produces very high levels of recall (96.0%) and precision (96.1%), which can be expected to be potentially higher for NAS members due to their assumed greater-than-average contributions to science. However, there is still a risk - which is higher for Eastern Asian names - of merging publication sets of authors with similar or near-similar names (D’Angelo and van Eck, 2020).

Our author analyses focus on the set of authors who published at least one Contributed submission in the three main sections in the period 2007 to 2020. One thousand seven hundred authors qualify for this; however, a small number of these have very few publications. For practical reasons (we want to be able to create deciles of articles), we restrict the author analyses to those with at least ten publications in total (including those prior to 2007), leaving 1,640 authors. We manually checked the twenty authors with the most publications to identify potential cases of merged publication profiles. We found three, here-among the most prolific (1,613 publications) and removed these from the set. The remaining set had a median number of 108 publications (mean publications per author = 144.6) and a maximum of 1,027 publications for one scientist.

4. Results

Figure 1 presents the number of days between submitting and accepting published articles in PNAS, distinguishing between Contributed and Direct submissions. It shows a consistent pattern across all disciplines, though most pronounced in the biological and physical sciences. Direct submissions, on average, spend 121 days in peer review in the biological sciences, 126 in the physical sciences and 145 in the social sciences, whereas Contributed submissions spend 74.8 days in peer review in the biological sciences (Δ = 46.2 days), 80.4 in the physical sciences (Δ = 45.6) and 119 in the social sciences (Δ = 26).

Figure 1. Time in days between submission and acceptance of published articles. Vertical lines indicate the mean duration of the review process.

Figures 2 and 3 address the impact of Contributed articles and Direct submissions, presenting the findings of our citation analyses. Figure 2 shows the median number and IQR of normalised citation scores (NCS), controlling for broad research field and publication year. The figure indicates only marginal differences between Contributed articles and Direct submissions for social and biological sciences, although the gap is somewhat more pronounced in physical sciences. In all cases, Direct submissions receive slightly more citations.

Figure 2. Normalised citation scores for Contributed articles and Direct submissions. Thicker horizontal lines indicate median scores, and colour boxes represent interquartile ranges.

In contrast, figure 3 presents the distribution of Contributed articles over citation deciles. The figure emerged from rank-ordering all PNAS articles in terms of normalised citation scores, dividing them into ten equal-sized segments, and showing the distribution of Contributed articles over these segments, relative to the share of Contributed articles in the entire sample. The first decile contains the least-cited articles. This figure shows that Contributed articles are relatively frequent among the group of least-cited articles for all disciplines. Especially for the physical sciences, Contributed articles are relatively uncommon among the most-cited articles. In contrast, the social sciences show a slight overrepresentation of Contributed articles among the highest-cited works. However, one should note the relatively small sample size for the social sciences, making it more vulnerable to minor variations, and also the quite small effect size (measured in percentage points) in this bin.

Figure 3. Distribution of Contributed articles compared to overall PNAS papers over normalised citation score deciles. Deciles are ranked from lowest to highest NCS.

Figure 4 presents trends over time in terms of the proportion of PNAS articles appearing as Contributed articles and the relative citation impact of these articles compared to Direct submissions. The figure shows that Contributed articles are least common in the social sciences, with a share of about 15%. At the same time, they are more common in the biological and physical sciences, taking up about 25% of articles each year. In the last two fields, there was a rise in Contributed articles between 2012 and 2017, after which the share of Contributed articles fell back to the level of 2010.

In terms of citation impact, Contributed articles are cited slightly less than Direct submissions, but the difference between both article types is gradually diminishing, reaching near-equal citation impact in 2020. Note that the citation impact of social science articles is strongly fluctuating, potentially due to the relatively small sample size in this field. This makes the annual distribution sensitive to outliers.

Figure 4. The proportion of Contributed articles and citations to Contributed articles relative to Direct submissions over time.

Figure 5 shows the distribution of PNAS-Contributed articles, Direct submissions, and articles in Nature and Science journals among the citation deciles of authors of varying mean normalised citation scores (MNCS). In particular, the figure distributes all authors, based on their MNCS, over ten deciles on the x-axis, with authors with the lowest MNCS in the first decile. Subsequently, the papers of those authors are distributed by their normalised citation score (NCS) deciles on the y-axis, with papers receiving the fewest citations in the first decile. The figure panels then present heatmaps indicating the relative frequency of occurrence of specific article types among these citation deciles: panel a) for PNAS-Contributed articles, b) for PNAS Direct submissions, c) for Nature articles, and d) for articles in Science.

The figure indicates that PNAS articles are relatively common among the higher-cited articles for all authors and underrepresented among their lowest-cited works. This pattern is substantially stronger for Contributed articles than for Direct submissions and stronger for authors in the lower range of the MNCS spectrum.

A much stronger pattern emerges for Nature and Science articles, however. Articles in these journals almost exclusively occur among the top 10% highest-cited works for all authors, particularly so for authors in the middle range of the MNCS spectrum.

Figure 5. Heatmap of citation impact of a) PNAS-Contributed papers, b) PNAS Direct submissions, c) Nature articles, d) Science articles. Authors with a Contributed paper in PNAS are distributed among deciles based on their mean normalised citation score (MNCS) on the x-axis. All papers in Web of Science by these authors are distributed over deciles based on their normalised citation score (NCS) on the y-axis. Colour levels show the rate (O/E) of observed (O) submissions per decile of the four article types over the expected (E) submissions of that type for the author.

5. Discussion

This section will briefly reflect on the data presented in the previous section. First, we note that PNAS-Contributed articles tend to spend less time in the review process than Direct submissions. The submission procedure for Contributed articles can likely explain part of this. The contributing author provides the names of suitable reviewers who agreed to review the paper. This means that the potentially time-consuming phase of finding appropriate referees willing to review the manuscript can be skipped for these articles. Other mechanisms could be at play too, but our data does not allow for further analysis of the underlying mechanisms and potential systemic factors at play.

In terms of citation impact, we note that Direct submissions tend to be slightly higher cited than Contributed articles. In particular, Contributed articles are overrepresented among the group of least-cited PNAS papers. This partially contrasts with Rand and Pfeiffer (2009), who found that Contributed articles were overrepresented both among the group of least and most-cited PNAS papers. The closing of the citation gap over time between Contributed and Direct submissions, visible in our data, is in line with previous work by Davis (2016). Davis suggested that this closing was attributable to persistent moves to tighten editorial scrutiny of Contributed papers in the preceding years.

Our main contribution here is to combine data on citations to PNAS papers with the citation data of contributing authors. We note that while PNAS-Contributed articles tend to generally appear in lower per-author citation deciles than Direct PNAS submissions, they are still more likely to appear in the top citation deciles of authors. This indicates that authors ‘use’ the Contributed-articles-track in diverse ways, with especially authors with lower mean normalised citation scores profiting most from articles published as Contributed papers, in terms of citation impact. This finding may be related to the seniority and productivity of authors, which will be further analysed in the future.

In particular, we will build on these findings by extending our analyses both from an author and a journal perspective. At the conference, we aim to present work on the topics and themes addressed in Contributed and Direct PNAS submissions, the role that contributing authors play in the research and publication process, and the co-authorship network of contributing authors and their suggested reviewers. This should help us further understand the privileged route to publication for NAS members and the epistemic and social aspects of this publication process.

Open science practices

As a supplementary file to this manuscript, we attach a dataset that includes aggregate data on article’s: review time, disciplinary section, normalized citation score, and submission type. This dataset allows reproduction of our analyses. The dataset can be found here: https://doi.org/10.5281/zenodo.7845375.
In principle, all analyses are based on openly available information on PNAS articles, available from the PNAS website. However, the structured dataset used for our analyses is proprietary and was provided by PNAS under condition of non-disclosure, hence preventing open sharing.

Acknowledgments

The authors thank Emil Alnor for help cleaning parts of the data and extend their gratitude to the editorial team of PNAS for facilitating access to publication metadata records.

Author contributions

All authors participated in the conceptualisation, investigation, methodology and writing of the manuscript. JPA curated and validated the data and performed the formal analysis.

Competing interests

The authors declare no competing interests.

Funding information

JPA was funded by the Independent Research Fund Denmark, grant number 0133-00165B.

References

Aldhous, P. (2014). Scientific publishing: The inside track. Nature, 510(7505), 330–332. https://doi.org/10.1038/510330a

Baldwin, M. (2017). In referees we trust? Physics Today, 70(2), 44–49. https://doi.org/10.1063/PT.3.3463

D’Angelo, C. A., & van Eck, N. J. (2020). Collecting large-scale publication data at the level of individual researchers: A practical proposal for author name disambiguation. Scientometrics, 123(2), 883–907. https://doi.org/10.1007/s11192-020-03410-y

Davis, P. M. (2016). Comparing the Citation Performance of PNAS Papers by Submission Track [Preprint]. Scientific Communication. https://doi.org/10.1101/036616

Garfield, E. (1987). Classic Papers from the Proceedings of the National Academy of Sciences. Part 1. Each Year’s Most-Cited Paper, 1955-1984—And Then Some. Essays of an Information Scientist, 10(36), 247–255.

Horbach, S. P. J. M., & Halffman, W. (2018). The changing forms and expectations of peer review. Research Integrity and Peer Review, 3(1), 8. https://doi.org/10.1186/s41073-018-0051-5

Jukola, S. (2017). A Social Epistemological Inquiry into Biases in Journal Peer Review. Perspectives on Science, 25(1), 124–148. https://doi.org/10.1162/POSC_a_00237

Kean, S. (2009). PNAS Nixes Special Privileges for (Most) Papers. Science, 325(5947), 1486–1487. https://doi.org/10.1126/science.325_1486b

Liu, F., Holme, P., Chiesa, M., AlShebli, B., & Rahwan, T. (2023). Gender inequality and self-publication are common among academic editors. Nature Human Behaviour, 7(3), 353–364. https://doi.org/10.1038/s41562-022-01498-1

Merton, R. K. (Ed.). (1973). The Sociology of Science: Theoretical and Empirical Investigations. University of Chicago Press.

Rand, D. G., & Pfeiffer, T. (2009). Systematic Differences in Impact across Publication Tracks at PNAS. PLoS ONE, 4(12), e8092. https://doi.org/10.1371/journal.pone.0008092

Schekman, R. (2009). PNAS will eliminate Communicated submissions in July 2010. Proceedings of the National Academy of Sciences, 106(37), 15518–15518. https://doi.org/10.1073/pnas.0909515106