Who writes what? The academic age patterns of review genres in biomedicine

Alexander Schniedermann^*

^*schniedermann@dzhw.eu

0000-0003-2132-7419

German Centre for Higher Education Research and Science Studies (DZHW), Germany

Centre for Science and Technology Studies (CWTS), Germany

1. Introduction

In 2020, the movement towards more responsible research assessment in the biomedical sciences led to the already prominent Hong Kong Principles for Assessing Researchers. The statement lies in a tradition that criticizes traditional research assessment and bibliometrics as being partly responsible for the many issues of contemporary publishing (Rushforth and Hammerfelt 2022). In strong alliance with the ethos of Open Science, the second of the Hong Kong principles aims to increase the transparency of research reports, or in other words, more insightful and useful scientific texts. It suggests the dissemination and usage of so-called reporting guidelines to achieve this goal (Moher et al. 2020).

Reporting guidelines are checklists developed by publishing experts and metascientists in order to intervene on scientific writing. They provide guidance to authors by defining the information that has to be included in a publication in order to consider it being transparent. For example, the "Preferred Reporting Items for Systematic Reviews and Meta-Analyses" (PRISMA) suggest that authors should describe whether a protocol was registered and where, or how many researchers performed a risk of bias assessment (Moher et al. 2009). Today, there are over 600 reporting guidelines for various study types and scientific specialties, mostly close to biomedicine (see www.equator.org).

Although the compliance with the PRISMA guideline is regularly evaluated by various metascientific studies to refine and enhance it (e.g. Pager and Moher 2017), no measures have been taken to understand this phenomenon on a wider scale by making use of bibliometric data. While a bibliometric analysis cannot be used to assess the quality of guideline compliance, it can shed light on the who, the where and the when of its usage, especially because PRISMA is usually cited by those systematic reviews that apply the guideline. Studying it bibliometrically offers two benefits. First, measures and interventions to further disseminate the guideline can be refined and improved by a better overview of its user and non-user base. Second, such an analysis adds to the general understanding of standardization in science and the regulation of research practices as a social phenomenon. Although there is a rich tradition in the science studies that analyzies standardization qualitatively, only few studies draw on quantitative and generalizable findings.

This study takes a step into a quantitative understanding of PRISMA's user base by investigating the relation between research experience and level of standardization and aims at a better understanding of both.

The level of standardization of scientific literature is approximated by making use of the review literature at which PRISMA tries to intervene. This type of scientific literature fueled early visions of the differentiation of scientific practices in which scientific reviewing becomes an own profession (Garfield 1977), a vision that further manifested with the rise of more standardized review types and the emergence of the Cochrane Collaboration and its descendants - influential and highly visible organizations that develop the most rigorous methods for research syntheses (Chalmers et al. 2005; Grant and Booth 2009). The research experience or academic age of authors can be understood and assessed in different ways but is usually done with respect to an author's publishing history (Sugimoto et al. 2016). Such analyses focus on individual careers in terms of ranges and productivity but are only occasionally used to make characterizations at meso or macro levels, for instance journals, institutes, or disciplines.

This study aims to answer the research question "what are the differences between less experienced and more senior researchers in relation to their authorship of different review genres and levels of standardization, respectively?" And "what role does academic age play for the dynamics of standardization attempts?" These questions are motivated by the idea that new researchers make the most out of new standards and new ways to professionalize because this makes research projects doable and career paths most promiseful (Fujimura 1987). Based on this assumption, this study focuses on the following hypotheses: More standardized genres have more first-ever authors (A). Relatedly, more standardized review genres have lower academic ages in general (B) and PRISMA-citing systematic reviews have a lower academic age compared to systematic reviews that do not cite the guideline (C). Lastly, the average academic age of authorships increases with the age of the cited PRISMA version (D).

2. Data and Methods

In a first step, data about document types and other characteristics were downloaded from the PubMed Baseline provided in December 2022 (Sayers et al. 2022). A Python script using Spyder and psycopg2 package retrieved data from the server, filtered it and stored it in a PostgreSQL database. Because publications are multi-classified in PubMed, a classification hierarchy was developed that makes use of the base as well as the types “Review” and “Systematic Review”. In addition, systematic reviews that were published in the Cochrane Library were reclassified as “Cochrane Systematic Reviews”. After matching these data with the in-house version of SCOPUS provided by the German Competence Network Bibliometrics (www.bibliometrie.info), all systematic reviews that cited one of the many PRISMA documents have been identified and relabeled into an own category. After restricting, filtering, and cleaning, the resulting dataset consists of 1,203,687 reviews, 149,334 systematic reviews that do not cite the PRSMA guideline, 83,788 systematic reviews that cite any of the PRISMA versions, and 9891 Cochrane Reviews.¹

Academic age was calculated as the distance between an authors first publication year and the publication year of the focal paper (Milojevic 2012). It has an inclusive counting so that “1” is the lowest value which represents an author's first year of publishing experience. For the disambiguation of authors and the clustering of publication histories, Scopus AuthorID was used which is based on an algorithm that takes criteria such as affiliation, subject area, source titles and co-authors into account. In addition, it features a manual feedback system for validating assignments (Moed et al. 2013). Kruskal-Wallis and Dunn tests for group comparisons as well as visualizations have been done in RStudio and the ggstatsplot package (Patil 2021).

3. Results

The dataset features 2,375,281 unique authors disambiguated by Scopus AuthorID’s. These have led to 4,724,590 different authorships of traditional or narrative review articles and 1,583,502 authorships of systematic reviews or Cochrane reviews. For all document types, most authorships are first year authorships, having an academic age of 1 as a highest value. Authorships generally decrease with growing academic ages with a seemingly lesser strength in the case of Cochrane reviews. Notably, there is a bump in authorship counts for the academic age of 14 due to the combination of almost no available data in Scopus prior to 1996 and this study’s first publication year of 2010.

Figure 1. Number of authorships per academic age and different document.

The different types of review articles offer very different authorship profiles (Table 1). While the sizes of authoring teams are similar for systematic reviews in general (6.16 and 6.81), teams are smaller in the case of Cochrane reviews (4.92) and smallest for more traditional review articles (3.92).

Authorships of systematic reviews, whether citing PRISMA or not, have the highest rate of first ever authorships with an academic age of 1 (15%, RR = 1.24 compared to reviews), which partly confirms hypothesis (A). While traditional reviews have a slightly slower rate (12%), only very few authors of Cochrane reviews which is considered as the most standardized review genre, are in their first year of publishing (7%, RR = 0.58 compared to reviews).

Although differences in median academig age between the different levels of standardization are overall significant for all authorship types, there is no decrease with increasing standardization. With traditional reviews and Cochrane reviews both having a median academic age of 12, hypothesis (B) has to be droppted. However, the progression holds true for PRISMA-citing systematic reviews with a median academic age of 9, to 10 for those without PRISMA citation, up to 12 for traditional reviews for all authors combined, thereby confirming hypothesis (C).

Table 1. Main configurations of academic age values for the different document types or levels of standardization. Separated by type of authorship.

In addition to the general differences, research syntheses also vary in academic age patterns between first authorships and last authorships. For all genres and levels of standardization, the median academic age is lower for first authors as compared to their respective last authors or all authors (see Q2 column). In the case of systematic reviews, the median values triple (from both 5 to 15 and 16) offering a larger range compared to traditional reviews (from 9 to 16) or Cochrane reviews (from 10 to 16). The portions of first-ever authors are higher for first authors compared to last authors for all genres. In the former case, these are mostly double digit (from 14% for traditional reviews to 21-22% for systematic reviews) except for Cochrane reviews (7%). In contrast, last authors are much less frequent first ever authors and values range from 4% (Cochrane reviews) to 8% (Systematic reviews without PRISMA citation).

Lastly, authorship profiles also differ among the different versions of the PRISMA guideline. Although differences between versions are not significant if all authors are considered which strictly falsifies hypothesis (D), differences between PRISMA versions are highly significant if first and last authors are separated.² In case of first authorships, the median academic age reversely follows the age of the guideline version with 6 of those who cite the oldest version (QOROM 1999) down to 4 of those who cite the newest version (PRISMA 2020). Likewise, the rate of authors in their first year of publishing (first ever authors) is highest for the most recent guideline (27%) and lowest for QUOROM (15%). In contrast, last authorships do not second these patterns. For example, their median academic age ranges from 14 to 16 and increases with more recent versions of PRISMA. Although the rates of first ever authors increase like those of the first authors, the differences are much smaller (from 4% to 5% to 7%).

4. Discussion and Conclusion

Over the last decades, the genre of research synthesis has become substantially differentiated. The high academic age and the lower number of authors per review article suggest that this genre still serves its traditional characterization. As such, review articles were written by experienced and proliferated authors who provided an overview over their research field, drew the lines of consensus or controversy, and set the research agenda for the coming years. Commonly, reviews accompany the emergence of new research fields and authors are invited by journals to write them (Blümel and Schniedermann 2021).

In contrast, systematic reviews and meta-analyses feature more authors with a younger academic age because of their different epistemic profile which is more independent from the authors' experience and time in the field. Instead of providing an overview over a field or setting agendas, systematic reviews aim to answer a very particular research question or identify gaps in the literature. They do this in a more procedural manner that draws from an interplay of different but often highly codified methodologies, expertise, and standards. Thus, the conduct and writing of a systematic review requires formal education and handbook knowledge (Moreira 2007). Having the highest portion of first-ever authors confirms their role as starting points of dissertation projects in biomedicine (Puljak and Sapunar 2017). Demonstratively, the triumph of the highly procedural systematic review led to the retronym "narrative review" which is nowadays commonly used for the traditional review.

In general, Cochrane reviews are systematic reviews with a comparable epistemic and social profile regarding scientific practices, collaboration, and authority. However, their conduct and reporting are even more standardized and extensive. They aim at exhausting all available studies regarding a particular research question and feature stronger claims with regards to comprehension and treatment advice at the cost of novelty. As such, they can feature hundreds of pages of technical material, for instance search strategies and terms, assessment tables, or bibliographies. In addition, the Cochrane Collaboration provides a 700-page manual, software tools, and extensive editing and dissemination services (Levin 2001). This makes Cochrane reviews much more laborious, turning them into an unreasonable workpackage for a dissertation, except if there is nothing else and Cochrane reviewer is the aspired professionalization (see Sambunjak and Puljak 2010).

Shedding light on the different versions and updates of the PRISMA guideline, the analysis has shown how younger academic ages can be considered as the early adopters of new standards. In general, standards and guidelines represent and aggregate a field's methodological advancement. Even more, it has been argued how standards play important roles in establishing professions and communities of practice by making research problems not only "doable" (Fujimura 1987), but also codified and communicable (Bowker and Star 1999). This means that a standard or guideline helps in setting tasks and to-do's even when there is only little background knowledge available. As such, it has been characterized as a cookbook of which "new 'cooks' tend to rely on it more than older ones" (Fujimura 1987, 278). In contrast, it has been shown how more senior authors use less references in general and may just cease to cite the PRISMA guideline after the years (Milojević 2012). In turn, improving scientific practices and evaluative cultures by standardization may be more fruitful if more emphasis is put on educational interventions compared to other measures, especially since there is no central regulating body and a rather slow rate of endorsement by academic journals or science funders.

Open science practices

The data used for this study is not openly available due to licensing restrictions. In addition, no new data has been created to conduct this study, rather available data from different databases has been combined in a novel way. Replication of the analysis here would be possible solely with using publicly available data. SQL and R scripts well be provided upon request.

Acknowledgments

The author wants to thank Dr. Wolfgang Kaltenbrunner for commenting on an earlier version of this manuscript.

Competing interests

The authors declared that they have no conflict of interest.

Funding information

Funded by the German Federal Ministry for Education and Research (01PU17017) and the Competence Network Bibliometrics 01PQ17001.

References

Blümel, C., & Schniedermann, A. (2020). Studying review articles in scientometrics and beyond: A research agenda. Scientometrics, 124(1), 711–728. https://doi.org/10.1007/s11192-020-03431-7

Bowker, G. C., & Star, S. L. (1999). Sorting things out: Classification and its consequences. MIT Press.

Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A Brief History of Research Synthesis. Evaluation & the Health Professions, 25(1), 12–37. https://doi.org/10.1177/0163278702025001003

Fujimura, J. H. (1987). Constructing `Do-able’ Problems in Cancer Research: Articulating Alignment. Social Studies of Science, 17(2), 257–293. https://doi.org/10.1177/030631287017002003

Garfield, E. (1977). Proposal For a New Profession: Scientific Reviewer. Essays of an Information Scientist, 3, 84–87.

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies: A typology of reviews, Maria J. Grant & Andrew Booth. Health Information & Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Levin, A. (2001). The Cochrane Collaboration. Annals of Internal Medicine, 135(4), 309. https://doi.org/10.7326/0003-4819-135-4-200108210-00035

Milojević, S. (2012). How are academic age, productivity and collaboration related to citing behavior of researchers? PloS One, 7(11), e49176. https://doi.org/10.1371/journal.pone.0049176

Moed, H. F., Aisati, M., & Plume, A. (2013). Studying scientific migration in Scopus. Scientometrics, 94(3), 929–942. https://doi.org/10.1007/s11192-012-0783-9

Moher, D., Bouter, L., Kleinert, S., Glasziou, P., Sham, M. H., Barbour, V., Coriat, A.-M., Foeger, N., & Dirnagl, U. (2020). The Hong Kong Principles for assessing researchers: Fostering research integrity. PLOS Biology, 18(7), e3000737. https://doi.org/10.1371/journal.pbio.3000737

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ, 339(jul21 1), b2535–b2535. https://doi.org/10.1136/bmj.b2535

Moreira, T. (2007). Entangled evidence: Knowledge making in systematic reviews in healthcare. Sociology of Health & Illness, 29(2), 180–197. https://doi.org/10.1111/j.1467-9566.2007.00531.x

Page, M. J., & Moher, D. (2017). Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: A scoping review. Systematic Reviews, 6(1), 263. https://doi.org/10.1186/s13643-017-0663-8

Patil, I. (2021). Visualizations with statistical details: The “ggstatsplot” approach. Journal of Open Source Software, 6(61), 3167. https://doi.org/10.21105/joss.03167

Puljak, L., & Sapunar, D. (2017). Acceptance of a systematic review as a thesis: Survey of biomedical doctoral programs in Europe. Systematic Reviews, 6(1), 253. https://doi.org/10.1186/s13643-017-0653-x

Rushforth, A., & Hammarfelt, B. (2022). The rise of ‘responsible metrics’ as a professional reform movement: A collective action frames perspective [Preprint]. SocArXiv. https://doi.org/10.31235/osf.io/cdmqz

Sambunjak, D., & Puljak, L. (2010). Cochrane systematic review as a PhD thesis: An alternative with numerous advantages. Biochemia Medica, 319–326. https://doi.org/10.11613/BM.2010.041

Sayers, E. W., Bolton, E. E., Brister, J. R., Canese, K., Chan, J., Comeau, D. C., Connor, R., Funk, K., Kelly, C., Kim, S., Madej, T., Marchler-Bauer, A., Lanczycki, C., Lathrop, S., Lu, Z., Thibaud-Nissen, F., Murphy, T., Phan, L., Skripchenko, Y., … Sherry, S. T. (2022). Database resources of the national center for biotechnology information. Nucleic Acids Research, 50(D1), D20–D26. https://doi.org/10.1093/nar/gkab1112

Sugimoto, C. R., Sugimoto, T. J., Tsou, A., Milojević, S., & Larivière, V. (2016). Age stratification and cohort effects in scholarly communication: A study of social sciences. Scientometrics, 109(2), 997–1016. https://doi.org/10.1007/s11192-016-2087-y