The growth of international collaboration: Collaboration Category Normalised Citation Impact (Collab CNCI) and implications for responsible research evaluation

Ross W. K. Potter^*

^*ross.potter @clarivate.com

0000-0002-1338-5910

Institute for Scientific Information, Clarivate, UK

International collaboration is now integral to academic research, with the Collaboration Category Normalised Citation Impact (Collab CNCI) indicator formulated to specifically account for this. Here, Collab CNCI is used independently, as well as in concert with other CNCI variants (the standard method and fractional counting), to derive and showcase important and otherwise hidden insights into three countries’ (Australia, China Mainland, Sri Lanka) research output. By deconstructing output into different collaboration types, as well as analysing data at both national and institutional levels, highly multilateral papers are shown to directly influence and in some cases (Sri Lanka) dominate a country’s CNCI performance. Such information is critical to be able to fully understand and responsibly compare research performance and consequently drive better informed policy and funding decisions.

1. Introduction

Academic research has become increasingly, particularly internationally, collaborative since the 1980s (e.g., Narin et al., 1991; Adams, 2013) and continues to do so (Adams et al. 2019a). Collaboration can be driven by factors including knowledge transfer, equipment access and financial aid and is generally seen as a positive (e.g., Hicks & Katz, 1996; Katz & Martin, 1997) and in some cases a necessity (e.g., COVID-19, CERN). However, collaboration can muddy the waters of researchers’ individual contributions (particularly early career researchers: Bikard, Murray & Gans, 2015), especially as highly multilateral collaborations tend to be more highly cited (e.g., Narin et al., 1991, Glänzel & Schubert, 2004, Adams et al., 2019a).

Citations are seen as a first order approximation of research influence (Garfield, 1955), i.e., the more citations the more influential the work, but vary widely between research fields (e.g., Garfield, 1979; Waltman & van Eck, 2013) necessitating normalisation. The Category Normalised Citation Impact (CNCI) indicator offers this normalisation by scaling citations by publication year, document type, and Web of Science subject category. With world average benchmarked at a value of 1.0, CNCI is considered a standard indicator for national and institutional comparisons (Jappe, 2020) and a key measure by many policymakers. However, CNCI, when calculating averages for an entity (e.g., country), is becoming increasingly skewed by highly multilateral papers with exceptional and erratic citation counts (e.g., Adams et al., 2022) that could disproportionately influence research policy and decision making. Consequently, various methods of assigning credit to authors (i.e., accounting for collaboration) have been formulated (e.g., full counting, fractional counting, first-author counting), discussed in detail (e.g., van Hooydonk, 1997; Egghe, Rousseau, & van Hooydonk, 2000; Gauffriau & Larsen, 2005; Huang, Lin, & Chen, 2011; Aksnes, Schneider, & Gunnarsson, 2012; Waltman & van Eck, 2015) and excellently summarised (Gauffriau, 2021). No strong preference has emerged in favour of any method, possibly because continuing problems are evident for all. 

Consequently, the Institute for Scientific Information formulated its own variant of CNCI – Collaboration, or ‘Collab’ CNCI (Potter, Szomszor and Adams 2020, 2022) – with the reasoning that if collaboration is now integral to research, then it should be incorporated directly into analyses when normalising citation counts. Here, Collab CNCI is showcased to demonstrate how it can be used independently, as well as in concert with other CNCI variants, to derive important and otherwise hidden insights into countries’ research output and therefore drive better informed policy and funding decisions.

2. Methods

Web of Science Core Collection article data covering 2009-2018 inclusive were used. Institutions and countries of each author were extracted from address metadata. CNCI values were calculated using the standard method of normalisation (henceforth referred to as standard CNCI) by publication year, document type, and Web of Science subject category (one of 254 categories ranging from acoustics to zoology and applied at the journal level). In cases where articles were assigned to multiple subject categories, the average CNCI of all subject categories was calculated and then assigned to the article. Collaboration CNCI follows the standard CNCI approach with citation counts normalised by publication year, document type and subject category but, crucially, has an additional normalisation that considers the type of collaboration. Five collaboration types cover domestic (single institutional and multi-institutional) and international bilateral, trilateral and quadrilateral plus (i.e., four or more countries) collaboration, with the type calculated based on address metadata. This approach means, for example, international bilateral articles are only compared to other international bilateral articles. Fractional CNCI was calculated at the author level following Waltman & van Eck (2015). This method assigns credit to each entity (author, country etc.) on an article based upon the total, deduplicated number of each, which is then multiplied by the article’s CNCI to calculate the fractional CNCI value. A country’s mean CNCI was then calculated by summing the CNCIs on all articles which that country was present and dividing this by the sum of its total fractional CNCI value. See Waltman & van Eck (2015) for a more thorough description and methodology. These three CNCI variants were then compared over the ten-year period, with the ‘improvement’ in each calculated as the difference in their value between 2009 and 2018. Data for 30 Chinese institutions were taken from Potter, Szomszor & Adams (2022) where they were chosen semi-randomly to represent a mixture of high, medium and low article output institutions.

3. Results

Figure 1 illustrates how CNCI values calculated using the standard, collaboration and fractional approach compare for three different research economies. For Australia (a well-established research economy with a gross domestic expenditure on research and development (GERD) of 1.80% in 2019), fractional and collaboration methods produce similar results despite their very different approaches. Both these values are lower than that of standard CNCI and appear to have stagnated. For China Mainland (a recently developed research economy; GERD of 2.40% in 2020), the standard and collaboration approaches are almost identical, suggesting that CNCI performance is not collaboration dependent. Importantly, China Mainland’s research has continually increased its performance over the period with all three indicators above world average in 2018. For Sri Lanka (a developing research economy; GERD of 0.13% in 2018), standard CNCI provides far higher values (peaking above 2.0 in 2015) and is more variable than the other approaches; normalising by collaboration type or author credit sees values become less erratic, though they drop below world average for each year analysed. Consequently, while Sri Lanka outperforms both Australia and China Mainland over several years by standard CNCI, it no longer does when normalising by collaboration type or author credit. Sri Lanka’s standard CNCI values are, therefore, likely driven by a concealed factor requiring a different analytical approach.

Figure 1: A comparison of standard, collaboration and fractional CNCI values for three countries over a period of 10 years. A value of one represents world average.

Figure 2 deconstructs the article output for China Mainland and Sri Lanka by collaboration type, providing an alternative view. Both countries illustrate that total cites and, consequently, CNCI values tend to increase as research becomes more (internationally) collaborative (boxplots to the left). The data, however, also illustrate the disparity between article and citation share for each collaboration group (bar plots to the right). China Mainland’s research is overwhelmingly domestic (~75% of articles), likely due to its large research economy, and accounts for 65% of all its citations. Its article and citation share for each collaboration type are broadly comparable. Conversely, Sri Lanka’s research is mainly internationally collaborative (62% of articles), likely due to its small research economy, with that output responsible for ~90% of all its citations, particularly international quadrilateral plus articles. These citations to highly multilateral articles are likely responsible for the elevated standard CNCI values seen in Figure 1.

Figure 2: Article and citation data (2009-2018) for two countries deconstructed by collaboration type (dom:single – domestic single institution; dom:multi – domestic multi-institutional, int:bilat - international bilateral; int:trilat – international trilateral; int:quad+ – international quadrilateral plus). White squares on boxplots represent the mean, the vertical black line within the boxes the median. Note different scales.

A country’s CNCI is driven by its own institutions, so these must be analysed to understand national trends. Figure 3 plots standard, collaboration and fractional CNCI values deconstructed into the five collaboration types for the University of Peradeniya, the largest university in Sri Lanka, over a 10-year period. The standard CNCI plot illustrates again that, in general, more collaborative research produces a higher CNCI; fractional CNCI also tends to follow this with international quadrilateral plus plotting the highest and domestic single the lowest. However, when using Collab CNCI, the same order is not necessarily maintained, with domestic output bettering internationally collaborative work in some years (e.g., domestic single institutional articles in 2009 and 2018). Most striking, however, is the variability and range of international quadrilateral plus articles, reaching an astonishing standard CNCI value of 100 in 2012. The international quadrilateral plus group using collaboration and fractional CNCI follow a similar same trend, reaching values far exceeding other collaboration types, though far lower than standard CNCI. Highly multilateral article output from this university is, therefore, likely driving the spikes in Sri Lanka’s CNCI indicators.

Figure 3: Standard, collaboration and fractional CNCI for University of Peradeniya, Sri Lanka, broken down by collaboration type over a period of 10 years. Note different scales.

For China Mainland, the similarity between all three indicator variants, particularly standard and collaboration (Figure 1) should also be analysed at an institutional level to understand these relationships. Figure 4, therefore, plots the change in collaboration and standard CNCI between 2009 and 2018 for 30 Chinese institutions. The institutions have an almost one-to-one relationship (gradient of 0.96) between the two variants, with only one institution having a negative change (State Oceanic Administration). The largest increase is for Northwestern Polytechnical University, Xi’an (0.78, 0.74).

4. Discussion

Results demonstrate the benefits of using CNCI variants in concert (Figure 1). Sri Lanka’s standard CNCI performance, given its variance and the country’s research capacity, could be anomalous (when comparing to China Mainland and Australia) and result in ill-informed policy making. The two indicators considering collaboration provide a different interpretation, while those for Australia and China are generally consistent with their respective standard CNCI behaviour. However, it is only by analysing data by collaboration type (Figure 2) as well as at the institutional level (Figure 3) that these results can be fully understood. The national spikes for Sri Lanka in 2012 and 2015 match those for the University of Peradeniya. Analysis of the university’s output shows that it was involved in several global disease papers published in the Lancet containing tens of countries and hundreds of authors. These papers (Lozano et al., 2012; Naghavi et al., 2015; Vos et al., 2015) are exceptionally highly cited. Given the global nature, in terms of authorship and content, it is no surprise that these articles have received thousands of citations and heavily skewed Sri Lanka’s standard CNCI values. Such articles, while valuable for Sri Lanka’s participation in global health research, should not therefore be considered when assessing Sri Lanka’s overall performance as they are clear outliers.

Figure 4: Collab CNCI improvement against standard CNCI improvement (over the period 2009-2018) for 30 Chinese institutions. The institutions are fit through the origin by the equation (shown in blue): y = 0.960*x + 0.

Australia’s CNCI performance may mirror that of Sri Lanka to some degree. Its increase in standard CNCI, compared to stagnation in fractional and collaboration CNCI, may be due to an internationally collaborative boost on globally relevant research which, when normalised by collaboration or author contributions, disappears. Performance interpretation can vary based on the available data and any summary information must be used with caution (Szomszor et al., 2021). Therefore, as with the Sri Lanka example, investigation of Australia’s research output by collaboration type is required to assess whether an international boost is present which may alter evaluation of Australia’s overall performance.

At the institutional level for China Mainland there appears to be no appreciable effect in CNCI change when normalising CNCI by collaboration type (Figure 4). This may be because, unlike Sri Lanka, China’s research is heavily domestically focused, so the effects of any collaboration on exceptionally cited highly multi-lateral articles (i.e., 1.6% quadrilateral plus articles accounting for 4.4% of total citations) is likely limited. Additionally, article output and citation share for each collaboration type are comparable (within a few percentage points), so no one group is likely to have a dominant effect.

The sample of Chinese institutions also illustrates that most improved their CNCI values over the period. For CNCI to increase, citations must be accumulated at a greater rate compared to other articles published in the same year and subject category. An increase in citation count could be due to Chinese citation practices, namely: citing their fellow citizens more often (Bakare & Lewison, 2017; Shehatta & Al-Rubaish, 2019) and having longer reference lists than world average (Stahlschmidt & Hinze, 2018). Irrespective of how citations are received, Chinese research performs well (above world average as of 2018) regardless of the type of research collaboration; China Mainland, unlike Sri Lanka, does not need to rely on highly multilateral articles to boost its performance.

5. Conclusion

Given the increasingly collaborative nature of research, Collab CNCI offers an alternative version of CNCI (and other counting methods) that explicitly considers collaboration. Importantly, Collab CNCI retains a clear origin, building upon the well-established and understood, standard CNCI method. Additionally, comparisons show that the standard and fractional approaches can be used to complement the collaboration approach (Potter, Szomszor and Adams, 2022; Potter & Kovač, 2023) further strengthening analysis and understanding.

A potential criticism of Collab CNCI is that, like the standard and fractional approach, it only offers a single value snapshot of an entity’s performance. However, the use of collaboration types allows a detailed deconstruction of an entity’s research output for in-depth comparisons between collaboration types as well as peer institutions and other countries. Visual representations of the collaboration types also allow multi-author effects to be easily recognised providing context and insights that are not possible with single indicator values.

The interpretation of collaboration type is a further argument in favour of profiling publication portfolios (Adams et al., 2019b) rather than relying on the single, summary metric of CNCI. A profiling method could be applied at several levels including institution and country, or even to research funders, to achieve greater, more responsible, understanding of research activity to better inform and drive policy decisions.

Open science practices

Data used in this study were sourced from Clarivate’s Web of Science, which is accessible to academic researchers in the UK under licence from the Joint Information Services Committee and in other countries through separate licencing agreements.

Acknowledgments

R.W.K.P acknowledges fellow members of the Institute for Scientific Information for their contributions to discussions and previous publications on citation indicators.

Competing interests

R.W.K.P. is employed by Clarivate, the owner of Web of Science.

References

Adams, J. (2013). The fourth age of research. Nature, 497, 557–560. doi.org/10.1038/497557a

Adams, J., Pendlebury, D., Potter, R. & Szomszor, M. (2019a). Multi-authorship and research analytics. London: Clarivate Analytics.

Adams, J., McVeigh, M., Pendlebury, D. & Szomszor, M. (2019b). Profiles, not metrics. London: Clarivate Analytics

Adams, J., Pendlebury, D. & Potter, R. (2022). Making it count: Research credit management in a collaborative world. London: Clarivate.

Aksnes, D.W., Schneider, J.W. & Gunnarsson, M. (2012). Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods. Journal of Informetrics, 6(1), 36-43. doi.org/10.1016/j.joi.2011.08.002

Bakare, V. & Lewison, G. (2017). Country over-citation ratios. Scientometrics. 113, 1199-1207.  doi.org/10.1007/s11192-017-2490-z

Bikard, M., Murray, F., & Gans, J.S. (2015). Exploring trade-offs in the organization of scientific work: Collaboration and scientific reward. Management Science, 61(7), 1473–1495. https://doi.org/10.1287/mnsc.2014.2052

Egghe, L., Rousseau, R. & van Hooydonk, G. (2000). Methods for accrediting publications to authors or countries: consequences for evaluation studies. Journal of the American Society for Information Science and Technology, 51(2), 145–157. doi.org/10.1002/(SICI)1097-4571(2000)51:2<145::AID-ASI6>3.0.CO;2-9

Garfield, E. (1955). Citation indexes for science. Science, 122(3159), 108–111. doi.org/10.1126/science.122.3159.108

Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359-375. doi.org/10.1007/BF02019306

Gauffriau, M. (2021). Counting methods introduced into the bibliometric research literature 1970–2018: A review. Quantitative Science Studies, 2(3), 932–975. doi.org/10.1162/qss_a_00141

Gauffriau, M. & Larsen, P. (2005). Counting methods are decisive for rankings based on publication and citation studies. Scientometrics, 64, 85-93. doi.org/10.1007/s11192-005-0239-6

Glänzel, W. & Schubert, A. (2004). Analyzing scientific networks through co-authorship. In Moed, H. F., Glänzel, W. and Schmoch, U. (Eds.), Handbook of Quantitative Science and Technology Research: The Use of Publication and Patent Statistics in Studies of S&T Systems, (pp 257-276). Dordrecht: Kluwer Academic Publishers.

Huang, M.‐H., Lin, C.‐S., Chen, D.‐Z. (2011). Counting methods, country rank changes, and counting inflation in the assessment of national research productivity and impact. Journal of the American Society for Information Science and Technology, 62, 2427-2436. doi.org/10.1002/asi.21625

Hicks, D. & Katz J.S. (1996). Science Policy for a Highly Collaborative Science System. Science and Public Policy. 23, 39-44. doi.org/10.1093/spp/23.1.39

Jappe, A. (2020). Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005–2019. PLoS ONE, 5(4), e0231735. doi.org/10.1371/journal.pone.0231735

Katz, J.S. & Martin, B.R. (1997). What is research collaboration? Research Policy, 26, 1-18. doi.org/10.1016/S0048-7333(96)00917-1

Lozano, R. et al. (2012). Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet, 380 (9859), 2095-2128. doi.org/10.1016/S0140-6736(12)61728-0

Naghavi, M. et al. (2015). Global, regional, and national age-sex specific all-cause and cause-specific mortality for 240 causes of death, 1990-2013: a systematic analysis for the Global Burden of Disease Study 2013. Lancet, 385(9963), 117-171. doi.org/10.1016/S0140-6736(14)61682-2

Narin, F., Stevens, K. & Whitlow, E.S. (1991). Scientific co-operation in Europe and the citation of multinationally authored papers. Scientometrics, 21, 313–323. doi.org/10.1007/BF02093973

Potter, R.W.K. & Kovač, M.G. (2023). Tracking Category Normalized Citation Impact (CNCI) changes: Benefits of combining standard, collaboration and fractional CNCI for performance evaluation and understanding, 19^th International Conference on Scientometrics and Informetrics, abstract #126.

Potter, R.W.K., Szomszor, M. & Adams, J. (2020). Interpreting CNCIs on a country-scale: The effect of domestic and international collaboration type, Journal of Informetrics, 14(4), 10175. doi.org/10.1016/j.joi.2020.101075.

Potter, R.W.K., Szomszor, M. & Adams, J. (2022) Comparing standard, collaboration and fractional CNCI at the institutional level: Consequences for performance evaluation. Scientometrics, 127, 7435-7448. doi.org/10.1007/s11192-022-04303-y.

Shehatta, I. & Al-Rubaish A.M. (2019). Impact of country self-citations on bibliometric indicators and ranking of most productive countries. Scientometrics. 120, 775-791. doi.org/10.1007/s11192-019-03139-3

Stalschmidt, S. & Hinze, S. (2018). The dynamically changing publication universe as a reference point in national impact evaluation: A counterfactual case study on the Chinese publication growth. Frontiers in Research Metrics and Analytics. 3(30). doi.org/10.3389/frma.2018.00030

Szomszor, M. et al. (2021). Interpreting bibliometric data, Frontiers in Research Metrics and Analytics, 5: 628703. doi.org/10.3389/frma.2020.628703

van Hooydonk, G. (1997). Fractional counting of multiauthored publications: Consequences for the impact of authors. Journal of the Association for Information Science and Technology, 48, 944-945. doi.org/10.1002/(SICI)1097-4571(199710)48:10<944::AID-ASI8>3.0.CO;2-1

Vos, T. et al. (2015). Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990-2013: a systematic analysis for the Global Burden of Disease Study 2013. Lancet, 386(9995), 743-800. doi.org/10.1016/S0140-6736(15)60692-4

Waltman, L. & van Eck, N.J. (2013). A systematic empirical comparison of different approaches for normalizing citation impact indicators. Journal of Informetrics, 7(4), 833-849. doi.org/10.1016/j.joi.2013.08.002

Waltman, L., & van Eck, N.J. (2015). Field-normalized citation impact indicators and the choice of an appropriate counting method. Journal of Informetrics, 9, 872–894. doi.org/10.1016/j.joi.2015.08.001