Platform logo
Explore Communities
27th International Conference on Science, Technology and Innovation Indicators (STI 2023) logo
27th International Conference on Science, Technology and Innovation Indicators (STI 2023)Community hosting publication
You are watching the latest version of this publication, Version 1.
conference paper

Open access and research evaluation: first evidence from the Italian National Evaluation of Research Quality (VQR)

21/04/2023| By
Tindaro Tindaro Cicero,
+ 1
Scipione Scipione Sarlo
629 Views
0 Comments
Disciplines
Keywords
Abstract

The latest Italian National Evaluation of Research Quality exercise carried out by ANVUR, referred to the period 2015-2019 (VQR 2015-2019), has tackled for the first time the concept of open-access publications: according to VQR rules, scientific publications submitted for evaluation and financed at least by 50% with public fundings should have been made available to the public by July 30th, 2022; their list was published on the ANVUR website. In this paper, two major issues are investigated: first, we analyse the magnitude and incidence of Open Access publications among those submitted for evaluation across scientific fields and geographic areas. Secondly, we identify the articles published in DOAJ indexed journals and compare their evaluation outcomes with those of the articles appeared in journals that are not in the list.

Preview automatically generated form the publication file.

Open access and research evaluation: first evidence from the Italian National Evaluation of Research Quality (VQR)

Tindaro Cicero*, Marco Malgarini**, and Scipione Sarlo***

*tindaro.cicero@anvur.it

0000-0003-1926-8317

National Agency for the Evaluations of Universities and Research Institutes (ANVUR), 00153, Rome, Italy

**marco.malgarini@anvur.it

0000-0001-9553-5435

National Agency for the Evaluations of Universities and Research Institutes (ANVUR), 00153, Rome, Italy

***scipione.sarlo@anvur.it

0000-0002-2448-5220

National Agency for the Evaluations of Universities and Research Institutes (ANVUR), 00153, Rome, Italy

Abstract

The latest Italian National Evaluation of Research Quality exercise carried out by ANVUR, referred to the period 2015-2019 (VQR 2015-2019), has tackled for the first time the concept of open-access publications: according to VQR rules, scientific publications submitted for evaluation and financed at least by 50% with public fundings should have been made available to the public by July 30th, 2022; their list was published on the ANVUR website. In this paper, two major issues are investigated: first, we analyse the magnitude and incidence of Open Access publications among those submitted for evaluation across scientific fields and geographic areas. Secondly, we identify the articles published in DOAJ indexed journals and compare their evaluation outcomes with those of the articles appeared in journals that are not in the list.

Introduction

Since the 1980s, performance-based research funding systems (PBRFS) have started to be introduced worldwide, with the aim of increasing accountability of research institutions (Hicks, 2012). In Italy, according to Presidential Decree No. 76/2010, the Italian National Agency for the evaluation of Universities and research institutes (ANVUR) has been tasked with the responsibility for the conduction of a national Research Assessment Exercise (VQR in the Italian acronym), to be performed on a five-years basis. A first assessment started in 2012, with reference to the period 2004-2010, followed by a second one referring to the period 2011-14, and a third, concerning the period 2015-19. Results are used for allocating a performance-based share (“quota premiale”) of the national block funding (FFO, Ordinary Financing Fund): this share amounted in 2022 to EUR 2.336 billion, almost 30% of the whole fund.

VQR 2015-19 was started by Ministerial Decree 1110/2019, which defined the general assessment criteria of the exercise, which were further specified by ANVUR with the call issued on January 3, 2020, subsequently updated on September 25, 2020 to consider delays linked to the pandemic and amendment of Ministerial Decree 1110/2019 by Ministerial Decree 444/2020. The scientific products submitted by the Institutions (journal articles, contributions to conference proceedings, contributions to books, monographs, patents, and other forms of products) were assigned scores corresponding to previously defined categories. The evaluation was based on the so-called informed peer review method: when appropriate, according to the scientific field, bibliometric indicators were used to inform the judgement of peer experts (ANVUR, 2022). The evaluation involved overall more than 182,000 research products, authored by more than 65,000 scholars, and 676 case studies for Third Mission/Impact initiatives.

Among the main innovations introduced in the new exercise, research institutions were required to identify whether the publications submitted for evaluation were available Open Access (hereafter OA), and in this case to provide the URL to the document submitted for evaluation. At the conclusion of the assessment, the Agency published the list of all the evaluated research products with their metadata on its website, including a link to the OA version when available1. Indeed, in the last twenty years, the development of OA practices has significantly modified the way in which scientific results are disseminated to the academic community (Zhu, 2017), removing the major obstacles to accessing, sharing, and re-using the outputs of scholarly research. This rapid growth has stimulated scholars and practitioners to identify different models of Open Access, even if not all of them are universally recognized. Piwowar et al. (2018) reviewed three major topics from the OA literature, focusing on definition and subtypes of OA contributions; according to the 2002 Budapest Access Initiative2 (BOAI) a publication is defined as OA when it makes its content both free to read and free to use (Creative Commons, 2018, shared this definition), while authors retain all rights to their manuscript and data. Under Creative Commons (CC) licenses, authors grant specific and non-exclusive rights to publishers and other users. Other definitions restrict OA to digital and free of charge contents. It is also possible to distinguish among different sub-types of OA publications, such as Gold, Green and Hybrid. Gold journals directly publish their articles on their website and may be indexed in the Directory of Open Access Journal (hereafter DOAJ) database (Archambault et al., 2014; Gargouri et al., 2012). On the other hand, Green articles are self-archived in OA archives (i.e. ArXiv, Hal, etc.) or institutional repositories (Harnad et al. 2008). Many journals adopt a hybrid model, where articles are published in a classic subscription journal and are accessible upon the payment of a fee (Articles Processing Charge, APC) or after an embargo period. The publication, distribution and preservation of scientific articles published in journals that do not charge fees either to the authors or to the readers is also identified as Diamond or Platinum OA.

The advantages and disadvantages of Open Access have been extensively discussed (Tennant et al. 2016, McKiernan et al. 2016): first, Open Access increases dissemination and re-use of scientific results and provides benefits to research. The most striking example is the latest Coronavirus pandemic, where virologists have had an advantage in vaccine research after Wuhan China has given free access to the virus genomic sequence (Lu et al., 2020). OA articles are also considered to have advantages in terms of number of citations (so called OACA) and speed of reception but the estimates of the magnitude of this effect vary considerably (Langham-Putrow et al., 2021; Piwowar et al., 2018). Analysing 983 universities worldwide by combining data from Web of Science, Unpaywall and Leiden Ranking disambiguation of institutions, Robinson-Garcia et al. (2022) showed that the median share of OA publications of universities worldwide is equal to 43%, with a peak for European universities. According to available data, Italian universities show a level of OA share of around 47%; British universities have the largest share of OA publications (74%), followed by Sweden and Austria, United States and Brazil, with the most OA publications accessible via the green model. In the case of gold OA, Brazilian universities stand out with a median of 30% publications, followed by Sweden, Taiwan, and Italy. However, researchers face economic barriers to publishing in OA, linked to the payment of Article Processing Charges (APC). Inequalities among nations, characterized by an inability of national, organisational, and technological infrastructure to support OA (Fox and Hanlon, 2015), can slow down a stable development of these models. The effect of charging APC was investigated by Klebel and T Ross-Hellauer (2022), with mixed results. Indeed, higher institutional resourcing is found not to be linearly associated with researchers publishing in journals with higher APCs, with differences across scientific fields. On the other hand, countries with a high Gross Domestic Product are characterised by high average APCs but very heterogeneous levels of APCs were identified for less wealthy countries.

Considering these results and according to the definition of OA in Article 8 of the VQR Call3, in this paper we analyse the magnitude and incidence of open-access research publications submitted for evaluation. The main goal is to provide a first general overview of OA practices in Italy, offering an analytical snapshot of existing differences among scientific fields and geographic areas. With the support of the DOAJ dataset, we also calculate the share of Gold OA among journals hosting the submitted publications and provide information about how many journals charging an APC have been considered in the latest evaluation exercise. Some first considerations on the possible effects of publishing OA on evaluation results conclude the paper.

The dataset

The analysis carried out in this paper is based on two different sources of data: the primary source is the full dataset of publications submitted for the VQR 2015–2019. The VQR dataset contains the evaluations of 181,2804 research products submitted by the 134 assessed Institutions (98 Universities; 14 Public Research Organizations, supervised by the Ministry of University and Research - MUR; 22 Other Institutions performing research activities and participating in the VQR on a voluntary basis), involving 65,119 researchers. Publications were evaluated with reference to 17 disciplinary research areas; for each of them, ANVUR appointed a Group of Experts responsible for the evaluation (GEV)5. Upon evaluation, each publication was assigned a score ranging from A – Excellent and extremely relevant (with a score equal to 1) to E – Low relevance or not eligible (with a score equal to 0).

Figure 1 shows the distribution of the 181,820 research products according to the type of publication. As it can be seen, the Call allowed for the submission of a wide range of research product types, but researchers concentrated only on a few of them, namely journal articles (which alone account for 87.2% of the total number of submitted publications), contributions in books, and monographs.

Journals involved in the VQR are hence identified in the Directory of Open Access Journal (DOAJ), which is the second data source6 used in the paper. The Directory of Open Access Journal (DOAJ) is the main extensive index of Open Access journals from around the world. According to this source, a journal is considered as OA if “the copyright holder of a scholarly work grants usage rights to others using an open license (Creative Commons or equivalent) allowing for immediate free access to the work and permitting any user to read, download, copy, distribute, print, search, or link to the full texts of articles, crawl them for indexing, pass them as data to software, or use them for any other lawful purpose7”. DOAJ contains about 19,000 journals (of which 13,081 without APCs) coming from 131 countries, and over 8,000,000 articles written in 80 languages. Basic inclusion criteria determine which OA journals can be indexed by DOAJ: journals must be actively publishing scholarly research and must demonstrate a publishing history of more than one year or have published at least 10 articles (and not less than 5 per year). The primary target audience should be researchers or practitioners. DOAJ data are free available and downloadable. Dataset contains information about the journal (i.e., ISSN, type of peer review, language, publisher, country of publisher), the license and the APC amount, if requested.

Figure 1: Research products by type of publication and Area of evaluation (GEV).

Immagine che contiene grafico Descrizione generata automaticamente

Source: Authors’ elaboration on ANVUR (2022) data

OA distribution according to the scientific field, geographic area and the indexation in the DOAJ list

According to the institutions that were responsible for submitting products for evaluation in the VQR exercise, overall, 54.1% of all products were openly accessible at the end of the VQR. Of the remaining, slightly less than 21.8% were not openly accessible because publishers did not allow open access; 22.3% were not supported by public funds, at least to the extent that law would require for open access; just 1.5% were still under editorial embargo when VQR was concluded. However, the VQR rules allowed the same product to be submitted by different institutions, albeit with certain limitations (see ANVUR, 2022). If we, therefore, remove such duplications and look at products net of duplicates, the share of open access publications decreases slightly from around 54.1% to 51.7%.

Table 1: Open access availability in the VQR 2015-19.

Open access information according VQR Call With duplicates Without duplicates
n % n %
Open access distribution 97,986 54.1 80,398 51.7
Under embargo 2,732 1.5 2,424 1.6
Editor does not allow OA distribution 39,458 21.8 35,614 22.9
Not financed with public funds 40,349 22.3 36,472 23.4
No information available 755 0.3 671 0.4
Total number of products 181,280 100.0 155,579 100.0

Source: Authors’ elaboration on ANVUR (2022) data

Figure 2 shows the distribution of OA publications within the 17 VQR areas; the area of Physics presents the largest share of OA publications, followed by Mathematics and Computer science. More generally, it is the so-called STEM areas that present the largest shares of OA publications, followed by Life sciences disciplines. This mode of dissemination is less common in the areas of Social Sciences and Humanities, reaching the lowest level in Law studies, followed by Political and Social sciences and Business studies.

Figure 2: Availability of open access publications by VQR area (GEV).

Immagine che contiene grafico Descrizione generata automaticamente

Source: Authors’ elaboration on ANVUR (2022) data

Regarding the geographical distribution of OA publications, Figure 3 shows the existence of a quite significant territorial divide. In particular, the share of OA publications is higher in the Northern and Central regions, while it remains lower in Southern Italy and the two main Islands (Sicily and Sardinia).

Figure 3: Availability of open access publications by geographic area.

Immagine che contiene grafico Descrizione generata automaticamente

Source: Authors’ elaboration on ANVUR (2022) data

Table 2 focus on the relationship between OA journal articles submitted for evaluation and their presence in journals indexed in the DOAJ database, that can be interpreted as a proxy of Gold OA; the data shows that only one-fifth of OA articles submitted for the VQR are published in journals indexed in DOAJ8, while most OA articles is not published in journals comprised in the list (including Green OA). Most of the articles published in DOAJ journals charge a fee for article processing; no relevant differences in this sense emerge on a geographic basis.

Table 2: Open access availability and the DOAJ list

Articles Open access availability
Open access No open access
n % n %
Published in DOAJ Journals 15,757 20.6 456 0.8
Of which: in Journals charging APC 12,900 16.9 288 0.5
Not in DOAJ9 60,781 79.4 55,749 99.2
Total 76,538 100.0 56,205 100.0

Source: Authors’ elaboration on ANVUR (2022) data and DOAJ (2023) data

4. Open access availability and evaluation results

As stated in section 1, OA can increase the dissemination of scientific results; consequently, it may have a positive effect on publication impact, as measured by its influence on the international scientific community and the number of citations received by a paper, and by this way it may positively influence evaluation outcomes. In the following, we provide some first evidence about the relationship among OA availability and evaluation results; more specifically, Figure 4 compares the categories and corresponding scores assigned by the different GEVs during the evaluation of research products for both OA and non-OA publications. In STEM and, to a lesser degree, in Life sciences areas, scores are slightly higher for OA than for non-OA publications: this is especially true for Physics, Mathematics and Computer sciences, Biology and Psychology; on the other hand, no significant differences emerge in Social Sciences and Humanities. Notwithstanding the fact that there are several other factors that may influence evaluation, these results provide some first evidence about the relationship among OA and evaluation results.

Figure 4: Comparison of assessment categories and scores among areas

Immagine che contiene testo, statico, strumento di scrittura, matita Descrizione generata automaticamente Immagine che contiene grafico Descrizione generata automaticamente

Source: Authors’ elaboration on ANVUR (2022) data

Figure 5 finally relates evaluation outcomes in the different VQR areas to being published in a journal included in the DOAJ database. The data shows that the average score is higher for OA articles published in journals not indexed in DOAJ with respect to those included in the list and to non-OA articles; however, Social Sciences are an exception, since non-OA articles show better evaluation results than OA ones. Physics is the only area where publications included in DOAJ perform clearly better that non-OA articles. It should be noted that, on average, 80% of the VQR-assessed products published in scientific journals listed in DOAJ, according to the information found in this database, have been charged a fee for publication.

Figure 5: Comparison of average scores among areas of articles by presence in DOAJ database.

Immagine che contiene grafico Descrizione generata automaticamente

Source: Authors’ elaboration on ANVUR (2022) data and DOAJ (2023) data

Main findings and conclusions

The most recent Italian evaluation exercise, referred to the period 2015-19, recorded for the first time if a publication submitted for evaluation was available in Open Access mode; according to available data, 51.7% of publications submitted for evaluation are openly accessible. Differences in OA availability however differ substantially among scientific and geographic areas: in STEM areas, the share of open access publications varies between over 80% in Physics and around 52% in Chemistry. On other hand, Social sciences and Humanities show the lowest shares of OA publications: in Law studies less than 25% of publications are open access. At the geographic level, Universities located in the South of the country show a much lower share of open access publications, which never exceed 50% of the total number of publications submitted for evaluation. However, among open access articles as defined in VQR, only 20% is indexed in DOAJ, the main extensive index of Gold Open Access Journals from around the world.

We also find that open access publications obtained on average higher ratings in the STEM areas, whilst there seems to be no significant differences in other scientific areas, except for Classics, Philology, Literary Studies and Art History, where non-OA publications obtained a slightly higher average rating. Moreover, OA articles not indexed in DOAJ received on average higher ratings than those included in DOAJ in all the scientific areas (only exceptions being Physics and Law studies).

The analysis presented in the paper concerns with a relatively small subset of publications of Italian scholars, those that have been submitted for evaluation in the VQR 2015-19; a possible future extension of the study may imply the analysis of the diffusion of OA practices in Italy looking at a more comprehensive dataset of publications of Italian researchers. Further future developments of this work intend to investigate the association between evaluation outcomes and the type of articles published in open access, especially in the case of scientific journals with APC. However, based on available information, we may conclude that OA has a positive effect on evaluation results in the fields where the impact is more closely related to the count of the citations received; there is also preliminary evidence that regional inequalities, in terms of income and availability of technological infrastructures, may have penalised Universities based in the South of the country with respect to the access to OA facilities, particularly when the payment of an APC is requested.

5. References

ANVUR, (2022). Rapporto finale ANVUR. Statistiche e risultati di compendio – Valutazione della Qualità della Ricerca 2015-2019 (VQR 2015-2019), 21 luglio 2022. https://www.anvur.it/wp-content/uploads/2022/07/VQR-2015-2019_Rapporto_Finale_EC_21luglio2022.pdf

Archambault É, Amyot D, Deschamps P, Nicol AF, Provencher F, Rebout L, Roberge G. (2014). Proportion of open access papers published in peer-reviewed journals at the European and worldlevels–1996–2013. European Commission.

Gargouri, Y., Larivière, V., Gingras, Y., Carr, L., & Harnad, S. (2012). Green and gold open access percentages and growth, by discipline. arXiv preprint arXiv:1206.3664.

Harnad S., Brody T., Vallières F., Carr L., Hitchcock S., Gingras Y., Oppenheim C., Stamerjohanns H., Hilf E. R. (2004), The Access/Impact Problem and the Green and Gold Roads to Open Access, Serials Review, Volume 30, Issue 4, 2004, p. 310-314, ISSN 0098-7913, https://doi.org/10.1016/j.serrev.2004.09.013.

Fox, M., & Hanlon, S. M. (2015). Barriers to open access uptake for researchers in Africa. Online Information Review.

Hicks, D. (2012). Performance-based university research funding systems, Research Policy, Volume 41, Issue 2, 251-261, https://doi.org/10.1016/j.respol.2011.09.007.

Klebel, T., & Ross-Hellauer, T. (2022). The APC-effect: stratification in open access publishing.

Langham-Putrow, A., Bakker, C., & Riegelman, A. (2021). Is the open access citation advantage real? A systematic review of the citation of open access and subscription-based articles. PLoS One, 16(6), e0253129.

Lu, R., Zhao, X., Li, J., Niu, P., Yang, B., Wu, H., ... & Tan, W. (2020). Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding. The lancet, 395(10224), 565-574.

McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B.A., Ram, K., Soderberg, C.K., Spies, J.R., Thaney, K., Updegrove, A., Woo, K.H., Yarkoni, T. (2016). How open science helps researchers succeed. elife, 5, e16800.

Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., ... & Haustein, S. (2018). The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles. PeerJ, 6, e4375.

Robinson-Garcia N, Costas R, van Leeuwen TN. Open Access uptake by universities worldwide. PeerJ. 2020;8: e9410. pmid:32714658

Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. (2016). The academic, economic and societal impacts of Open Access: an evidence-based review. F1000Research, 5, 632.

Zhu, Y. (2017). Who support open access publishing? Gender, discipline, seniority and other factors associated with academics’ OA practice. Scientometrics, 111, 557-579.


Open science practices

Data used in our research was derived from evaluations of scientific products performed during the last national assessment exercise. The list of all the publications evaluated in the VQR 2015-19 is available on the ANVUR website: https://www.anvur.it/attivita/vqr/vqr-2015-2019/elenco-dei-prodotti-e-dei-casi-di-studio-valutati/. Data concerning evaluation results are also available at the aggregate level at the following link: https://www.anvur.it/wp-content/uploads/2022/07/VQR-2015-19_Rapporto_ANVUR_tabelle_EC_21luglio2022.xlsx. DOAJ data are also publicly available at the following link: https://doaj.org/csv. Evaluation results at the individual level are disclosed only to the researcher to which the evaluation has been associated and are not currently available according to the rules of the evaluation exercise.

Acknowledgments

We would like to thank the components of Governing Board of ANVUR for their suggestions and input throughout the research process. We also thank all the ANVUR officials that have participated to the VQR project. Their insights and expertise were instrumental in shaping the direction of this project.

Author contributions

Conceptualization, T.C., M.M., S.S.; Data curation, S.S.; Formal Analysis, S.S.; Software, S.S.; Writing – original draft, T.C., S.S.; Writing – review & editing, T.C., M.M.

Competing interests

The authors have no conflicts of interest to declare. All co-authors have seen and agree with the contents of the manuscript and there is no financial interest to report. We certify that the submission is original work and is not under review at any other publication.

Funding information

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.


  1. The list is available here: https://www.anvur.it/attivita/vqr/vqr-2015-2019/elenco-dei-prodotti-e-dei-casi-di-studio-valutati/↩︎

  2. https://www.budapestopenaccessinitiative.org/↩︎

  3. https://www.anvur.it/wp-content/uploads/2021/07/VQR-Call_Update-25.09.pdf↩︎

  4. This total does not include double counting of monographies, as required by VQR Call.↩︎

  5. Area 1 Mathematics and computer sciences; Area 2 Physics; Area 3 Chemistry; Area 4 Earth sciences; Area 5 Biology; Area 6 Medicine; Area 7 Agricultural and veterinary sciences; Area8a Architecture; Area 8b Civil engineering; Area 9 Industrial and information engineering; Area 10 Classics, Philology, Literary Studies, Art History; Area 11a History, Philosophy and Pedagogy; Area 11b Psychology; Area 12 Law; Area 13a Economics and Statistics; Area 13b Business Studies; Area 14 Political and Social Sciences.↩︎

  6. DOAJ data is available at the following url: https://doaj.org/csv. Downloaded 5th of April, 2023.↩︎

  7. https://doaj.org/apply/guide/↩︎

  8. Slightly less than 1% of the journal articles submitted in VQR were declared non-open access by the institutions even though they were published in journals present in DOAJ. Likely, this is a result of either an incorrect understanding by the institutions of the definition of open access provided by the VQR Call, or an inability to find the open access information for these products.↩︎

  9. This category includes articles published on journals currently in DOAJ but not at the time of publishing.↩︎

Figures (6)

Publication ImagePublication ImagePublication ImagePublication ImagePublication ImagePublication Image
Submitted by21 Apr 2023
Download Publication
ReviewerDecisionType
User Avatar
Hidden Identity
Minor Revision
Peer Review
User Avatar
Hidden Identity
Minor Revision
Peer Review