The development and application of a bibliometric strength, potential and risk analysis for research strategy in a University Medical Center

The development and application of a bibliometric strength, potential and risk analysis for research strategy in a University Medical Center

Rik Iping^*, Thed van Leeuwen^**, Ed Noyons^*** and Adrian Cohen^****

^*r.iping@erasmusmc.nl

0000-0002-5405-0535

Research Intelligence & Strategy unit, Research Policy & Advice, Erasmus MC University Medical Center Rotterdam, the Netherlands

^** leeuwen@cwts.leidenuniv.nl;

0000-0001-7238-6289

Center for Science & Technology Studies (CWTS), Leiden University, the Netherlands

^*** noyons@cwts.leidenuniv.nl;

0000-0002-3103-2991

Center for Science & Technology Studies (CWTS), Leiden University, the Netherlands

^**** a.cohen@erasmusmc.nl

Research Intelligence & Strategy unit, Research Policy & Advice, Erasmus MC University Medical Center Rotterdam, the Netherlands

Abstract

This paper describes the development of a bibliometric strength, potential and risk analysis tool, and its (potential) applications for research strategy. We focus specifically on the path leading up to development, our motivations and strategic needs. Furthermore we highlight the co-creation of the tool together with bibliometric experts and discuss the methodology behind the tool, how it works and initial feedback on how insights from the tool can be applied for research strategy.

1. Introduction

1.1 Setting

Erasmus MC is one of the largest University Medical Centers in The Netherlands. There are over 45 departments covering fundamental biomedical sciences, clinical sciences and public health research. The broad coverage of disciplines allows developments and innovations in research to be directly translated and applied into clinical practice, and societal applications, and at the same time translate questions from the clinic and society back into new research. In order to keep track of the quality and impact of research and strategic research choices, there are several factors that are regularly monitored such as the ability to attract external funding, PhD tracks, talent management and succession planning, academic networks and positions by using research intelligence analyses.

1.2 Historic use of bibliometry in Erasmus MC

Since the mid-90s Erasmus MC has had a partnership with the Center for Science and Technology Studies (CWTS) to perform bibliometric analyses on the scientific output at the level of departments and research schools. Bibliometric analyses revolve around a number of aspects of academic performance, such as output numbers, scientific impact as can be measured via citations, as well as visibility in the top of the literature, uncitedness, disciplinary embedding, and scientific cooperation. Scientific impact is expressed as comparing the received number of citations to a certain expected value, which functions as average on a global scale (Waltman et al. 2011). This comparison to an average and within a publication year is called normalization. For a long time, the Journal Subject Categories, as known from the Web of Science core collection, were used as a basis to compute a so-called world average level to compare the citations of publications to, within certain disciplines. These categories are journal based, and journals could be assigned to multiple categories, making citation-based comparisons complex. Another problem was that these categories are sometimes very large and heterogeneous, while others are smaller and more homogenous, and citation behaviour in disciplines within the subject categories could vary greatly. This problem was especially noticeable in small or niche fields, for instance plastic surgery (Iping et al. 2021), medical ethics or rare diseases (Zampeta et al. 2022). Around 2019 the CWTS and visiting researchers came up with an algorithm that could cluster publications based on direct citation relations (Waltman et al. 2020). This algorithm can make clusters of publications that have a link in their content (based on where they cite to and are cited from). These clusters are more suitable to be used as averages to compare citations of publications to, because they create a more level playing field, and reflect a more homogenous body of literature. Erasmus MC has embraced this new way of calculating citation impact and uses it to assess the scientific impact of the publications of departments.

1.3 The changing landscape of applied bibliometry

Bibliometric analyses have been scrutinized ever since they came into existence, but they rose to great popularity because these are very strong tools to quantitatively assess scientific impact. There are many examples of flawed uses of bibliometric analyses (such as h-index, and using journal based metrics, such as the Journal Impact Factor) (e.g. Waltman & van Eck 2012, Brito & Rodriguez Navarro 2021, Moed 2002). For some years, people have been advocating the responsible use of metrics (e.g. Leiden Manifesto (Hicks et al, 2015), Wilsdon 2015, DORA, Recognition & Rewards, COARA). A form of responsible use is using normalization in bibliometric calculations (internal contextualization), but even more important is the situatedness and interpretation of bibliometric analyses (external contextualization) (e.g. Iping et al. 2022). They only shed light on a limited portion of the context of what academic research comprises, and they can only be valued when the context is made clear by the interpreter to the receiver.

Under the pressure of changes in the way we recognize and reward academics, and assess academic research, bibliometric analyses are scrutinized, and it is sometimes advocated that they should be disregarded entirely, without offering validated alternatives. However, with the right context and interpretation bibliometry still holds much value when you recognize the limitations and provide the right context. The information is very usable to understand the dynamics of research fields, to understand what your colleagues and competitors are doing, who the most interesting parties to collaborate with are and simply to see how your research resonates in the scientific community, and if not, to explain why. For this purpose, the previously mentioned algorithmically defined clusters can be put to use.

2. The co-creation of a bibliometric strength, potential and risk analysis for research strategy

2.1 Analysing the demand - Structured interviews with department heads

The departments in Erasmus MC were provided basic bibliometric statistics at department level which they used to roughly monitor trends in overall citation impact. These statistics were now calculated using the CWTS cluster algorithm. In advisory talks with the departments heads they indicated that having the overall statistics could not provide them the insights required to actually use this information to steer or make strategic decisions. They lacked a general understanding on how their department scores were constructed. The research intelligence advisor was dealing with the same problem, not having enough in-depth information to provide the department heads strategic insights. Driven by these restrictions and the developments outlined earlier about the changing landscape of applied bibliometry, structured separate in-depth interviews were carried out among four department heads. In these interviews we used a standard set of questions probing into the current uses of the citation data at the level of the departments, and the demands and potential use of more in-depth data to support strategic decision making. Most of the feedback concerning the available citation data focused on the limited information it provided on the dynamics behind the citation score of a department. This score (the mean normalized citation score) is calculated by comparing the actual number of citations of an article to the mean number of citations of all articles from the same year in the same cluster (the expected value or world average). Though this score allows monitoring the citation impact trend of the total body of publications of a department, it lacks the power to show why a score is going up or down. The department heads unanimously agreed that potentially citation-based information on research performance could be used to support strategic decision making in research, but only if it could show the dynamics underlying the total scores of the departments. The main questions that could be distilled from the first round of interviews were:

• Can we break down the citation scores of a department on a lower level, showing the performance of groups in unbiased clusters?

• Can we use this information to identify strengths and weaknesses?

• Can we identify potentially interesting collaboration partners based on their impact in the clusters in which we are active?

In all the interviews, it was stressed by both the department heads and the research intelligence advisor conducting the interviews that the information coming from these analyses is just one dimension of the total portfolio of research performance of groups and individuals, and it should be treated as such. Our aim is to provide intelligence to support department heads in making better-informed decisions, but also to help researchers potentially improve their research by giving them strategic leads for collaboration or research focus.

The second stage of the co-creation process consisted of going back to the bibliometric experts from CWTS, during which the strategic questions we identified were discussed with the bibliometric experts from CWTS who subsequently transformed the citation analyses into a bibliometric Strength, Potential and Risk dashboard (SPR).

2.2 The methodology behind the bibliometric Strength, Potential & Risk analysis

The Strength, Promising and Risk (SPR) areas analysis, aims at classifying research foci of a unit to potentially feed into strategic decisions. The SPR analysis uses the CWTS publication-level classification system of an entire bibliographic database, e.g., Web of Science (WoS) (Traag et al., 2019). For the SPR analysis, each research area contains publications from around the globe within a number of years (in our case 2000 onwards). The publication set of a research unit (e.g., institute, university) is projected in this landscape of science. This means that a unit’s publications are positioned across research areas. Consequently, research areas may contain many, few or no publications from this unit. For each area there are a number of characteristics based on all publications within, e.g., growth over the years (volume per year), citation statistics, average number of authors per paper, and more. Besides these general statistics, information is available for subsets, e.g., a unit’s output. The characteristics at the area level at large can be considered as a benchmark to assess the performance of a unit within (like the way the mean normalized citation score mentioned before is calculated).

In the SPR analysis, we relate statistics of a unit to the statistics overall within each area in which it has publications.

3. The application of the bibliometric strength, potential and risk analysis for research strategy

3.1 Description of the SPR-tool

In the SPR-tool the user immediately gets an impression of the research portfolio of the department and the size of the different research areas, and it answers the first question we identified from our interviews ‘Can we break down the citation scores of a department on a lower level, showing the performance of groups in unbiased clusters?’. Because the clustering is algorithmic the list of clusters to which publications of a department are assigned to, can be long. Therefore, it makes sense to create a cut-off point, for example to only look at clusters with more than 5 publications in 4 years to count as substantial research area for a small or middle-size department, and maybe 10 or more for a large department. For each individual department parameters can be chosen to identify Strength, Potential and Risk areas. This functionality provides an answer to the second question identified from the interviews ‘Can we use this information to identify strengths and weaknesses?’. The parameters to choose from may differ per department, because they all differ in size and publication culture. The parameters can best be discussed together with the head of the department who has the most in-depth knowledge of the own department and how they would define a strength, potential or risk. Besides the fine-tuning of thresholds, the definition of S,P and R may differ from one context to the other. Having many publications in an area with high impact may be perceived as a strength, but considering the age of the staff, for instance, may also identify it as a risk area. A ‘potential’ can be defined as a cluster in which a department is increasingly publishing and where the citation-based impact is relatively high to very high. And a ‘risk’ could be a cluster in which a department publishes substantially, but the citation impact is lacking behind.

3.2 Translation to insights for research strategy

The SPR tool subsequently also helps in putting the information into context by providing an overview of the most prominent institutions world-wide also active in a cluster, the size of their output and PPtop10%, and if there is already collaboration with your own institution. This information provides the answer to the final question we identified from our interviews ‘ Can we identify potentially interesting collaboration partners based on their impact in the clusters in which we are active?’ Also the authors from the department active in the cluster are listed, and the individual publications assigned to the cluster and their citations are listed and can be accessed directly for the tool using the DOI.

The context that this information provides is essential to interpret a Strength, Potential or Risk. When a Risk is identified, a head of department wants to know what other institutions are doing in this cluster, and with which partners their researchers collaborate. They can also find comparable institutions (for instance other Dutch UMCs) in the list and see if how their activity is compared to their own. A citation score can be lower if a cluster is very large and dominated by exceptionally active and/or influential institutions. Then a Risk can be put into context, and it can be concluded that, compared to that highly competitive field, the impact is still significant. On the other hand, the information can also help to define a Potential area further or sharpen the parameters by comparing it to the total activity in the cluster. If a cluster remains a Risk area, the next step can be to discuss this with the researchers involved, by looking in depth at what their competitors are doing, if they are active in the right network, or if their focus can or should be adjusted. Also, an analysis can be made why certain publications are not or limitedly cited. A Risk area can be a reflection of a research line within a department which is discontinued, or an underperforming research line, but this can never be concluded solely by using citation-based information. This should be completed by other indicators (qualitative and quantitative), when a head of department evaluated her or his research groups. We have seen examples of an area that came out as a Risk, but the work was extremely important for clinical or medical practitioners in the field specifically in the Netherlands. This explained the limited number of (international) citations, but because of the context and interpretation this research line would never actually be labelled a Risk.

The four department heads involved in the in-depth interviews were also involved in the evaluation of the first version of the SPR dashboard. We asked them up front what their expectations were, and which research lines they would list as Strengths, Potential and Risks. In most cases the information from the SPR tool corresponds well with the image that the heads of department have of their research lines, but certainly not in all cases. Sometimes they were surprised by a cluster in which the citation-based impact was high. A concrete example was a research area in which a professor by special appointment was active. The head of department was not directly aware of the high impact of their work in the field and decided to assign two researchers to expand the research in this line and embed it better in the department. Another example concerned an associate professor with two research lines. For his promotion to professor the department head was seeking more focus in his research. The tool clearly showed one research line had significantly higher citation impact. The associate professor confirmed that this line was also more feasible based on the funding and PhD students he had. Together they decided that he should focus his research on this specific topic, and his chair was also aimed in this direction, and finally approved by the Dean and board of the hospital. A third example came from a research group that wanted to organize an international scientific meeting and invite not only the established groups but also emerging groups with potential. We used the SPR tool, together with network analyses to find interesting groups. Several new collaborations came forth from this meeting.

4. Discussion

4.1 Novel applications of bibliometry

Bibliometric information can provide very valuable insights. Available and potential are already far more sophisticated than h-indices and impact factors. However, sophisticated bibliometry is often not simple to calculate and to interpret, and many people are unaware of the possibilities. It is often not available on the fly and is not provided in many of the available databases and (online) tools. But the added value that advanced bibliometry can offer is great. It can help researchers, research leaders, department heads and boards of directors to support strategic decision making by opening up the actual dynamics of citation practices within fields and identifying collaboration partners. Users can support decision making, substantiate evaluations or strengthen narratives for several accountability purposes. The SPR tool is an example of a novel application of bibliometry as a form of research intelligence.

4.2 The role of the research intelligence expert

In the process of developing the SPR tool, the role of the research intelligence advisor was crucial. The advisor functioned as a sort of broker between the layer of management and the bibliometrics experts to translate strategic demands into a technical description of a tool to be designed. And in the end, it is up to the research intelligence advisor to translate the advanced bibliometric information from the dashboard back in the form of answers on specific questions of the heads of department and the Dean. The research intelligence advisor should be someone that understands both worlds. Someone who is knowledgeable about bibliometry, about the entire research process, about the context in which research is performed and should be valued and evaluated, and about overall department research strategy and the choices department heads are faced with distributing scare resources. Without this full perspective there is a risk of not sending the right information or not receiving it the right way. Finally, the Research Intelligence advisor is someone that is aware of the broad context of research evaluation, movements like ‘ recognize and reward’ and ‘ Open science’, to be able to frame their strategic advice knowing there are many more dimensions of research than just publishing and citing.

Open Science practices

The tool described in this paper is not openly available because it contains sensitive organizational information. It can only be accessed by the advisor at this point, and as a next step, by the heads of department who can only view the information for their own department. Erasmus MC publication information is openly available through our institutional research portal. Added dimensions containing citation analyses cannot be openly shared because of vendor restrictions and for strategic considerations.

Acknowledgments

We would like to thank Irene Mathijssen, Lex Burdorf and Hans van Leeuwen (all Erasmus MC) for their reflections on the SPR tool and the manuscript.

Author contributions

RI conceptualized the idea, performed the interviews and guided the development

TvL and AC supervised the development

EN and TvL developed the SPR methodology and tool

RI wrote the original draft

TvL, AC and EN reviewed and edited the manuscript

Competing interests

All authors declare that they have no competing interests.

Funding information

We did not receive funding.

References

Brito, R. & Rodríguez Navarro, A. (2021). The inconsistency of h-index: A mathematical analysis. Journal of Informetrics, 15 (1), 101106. https://doi.org/10.1016/j.joi.2020.101106

Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), Article 7548. https://doi.org/10.1038/520429a

Iping, R., Cohen, A.M., Abdel Alim, T., van Veelen, M.C., van de Peppel, J., van Leeuwen, J.P.T.M., Joosten, K.F.M., Mathijssen, I.M.J. (2021). A bibliometric overview of craniosynostosis research development. European Journal of Medical Genetics, 64 (6), 104224. https://doi.org/10.1016/j.ejmg.2021.104224

Iping, R., Kroon, M., Steegers, C. & van Leeuwen, T.N. (2022). A research intelligence approach to assess the research impact of the Dutch university medical centres. Health Research Policy & Systems, 20, 118. https://doi.org/10.1186/s12961-022-00926-y

Moed, H. F. (2002). The impact-factors debate: The ISI’s uses and limits. Nature, 415(6873), 731. https://doi.org/10.1038/415731a

Traag, V.A., Waltman, L. & van Eck, N.J. (2019). From Louvain to Leiden: guaranteeing well-connected communities. Scientific Reports, 9, 5233. https://doi.org/10.1038/s41598-019-41695-z

Waltman, L., van Eck, N.J., van Leeuwen, T.N., Visser, M.S. & van Raan, A.F.J. (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5 (1), 37-47. https://doi.org/10.1016/j.joi.2010.08.001

Waltman, L., Boyack, K.W., Colavizza, G., van Eck, N.J. (2020). A principled methodology for comparing relatedness measures for clustering publications. Quantitative Science Studies, 1 (2), 691–713. https://doi.org/10.1162/qss_a_00035

Waltman, L. & van Eck, N.J. (2012), The inconsistency of the h-index. Journal of the American Society for Information Science and Technology, 63, 406-415. https://doi.org/10.1002/asi.21678

Wilsdon, J. (2015). The metric tide. SAGE Publications Ltd, https://doi.org/10.4135/9781473978782

Zampeta, F.I., Distel, B., Elgersma, Y., Iping, R. (2022). From first report to clinical trials: a bibliometric overview and visualization of the development of Angelman syndrome research. Human Genetics, 141 (12), 1837-1848. https://doi.org/10.1007/s00439-022-02460-x