International University Rankings and Changing University Functions: Underlining indicators to capture societal and entrepreneurial contributions of universities

Solanki Gupta^*, Anurag Kanaujia^**, Sujit Bhattacharya^***, Vivek Kumar Singh^****

^*solankigupta2@gmail.com, ^**anuragkanaujia01@gmail.com, ^****vivek@bhu.ac.in

0000-0002-5779-2948, 0000-0002-5813-7427, 0000-0002-7348-6545

Department of Computer Science, Banaras Hindu University, Varanasi, India

^***sujit_academic@yahoo.com

0000-0002-6769-8293;

CSIR- National Institute of Science Communication and Policy Research (NIScPR), New Delhi, India

Abstract: The international university rankings are now increasingly used for various purposes, including for funding decisions. Various stakeholders trust ranking assessments which are based on narrow set of criteria namely teaching, research, knowledge transfer, global outlook. However, the role of university in the contemporary scenario have evolved to include societal functions. This paper presents an analysis on the need for updating the ranking frameworks in the context of changing functions of the Universities. In depth literature review and thorough study of popular ranking frameworks (QS, THE and ARWU) were utilized for identifying areas whose inclusion may be beneficial towards addressing the new university functions. The study presents a case for inclusion of five criterion to expand the scope of university ranking framework besides their usual criticism.

Keywords: Institutional rankings, STI ecosystem, Higher education, Performance Assessment, Societal development.

Introduction

The university ranking frameworks since their origin in 1980s (US News & World Report¹), and wider adoption after the Shanghai league table in 2003 (Stergiou & Lessenich, 2014), have gained much importance among academic as well as administrative groups. Issued by a third party, these rankings make it easy to quantify the achievements of universities and compare them with each other. Thus, universities use the rankings to satisfy the public demand for transparency and information (Usher & Savino, 2006) and also to secure funding and resources. At least ten global and as many regional university ranking frameworks are currently active (Vernon, Balas & Momani, 2018).

An explanation for the success of rankings originates from their ability to make heterogeneous characteristics of universities comparable through classification, normalization and standardization (De Rijcke et al., 2016), thus allowing universities to take part in a global higher education market driven by a competition for expertise, reputation, students and money. In the recent times, the role of funding and reputation has become even more pronounced, for these determine the quantity and quality of students, researchers, and funding support received by the universities worldwide (Lathabai et al., 2021). Further, factors like advancement of research in fields of statistics and bibliometrics, internationalization, globalization and economization in higher education have contributed to popularisation of university rankings.

The popularity of university rankings is accompanied by growth in the amount of literature related to their significance, role and methodologies (Blanco-Ramírez & Berger, 2014; Thakur, 2007; Hazelkorn, 2007). These studies are classified into three broader categories, namely studies looking at the origins (epistemological explorations of university roles and their role in development of systems for rankings) (Slaughter, Slaughter & Rhoades, 2004; Bok, 2009; Radder, 2010; Shore, 2010), methodologies used for assessments (Huang, 2012; Van Raan, 2005; Moed, 2017) and studies about the influence and effects of ranking exercises on the university ecosystem (Thakur, 2007; Hazelkorn, 2007) (Table1).

Table 1. Studies coving different criteria of discussions of the university rankings

These studies, however, have not looked at the new roles taken up by universities with respect to the changing environment and university ecosystem. Universities are now focused towards social responsibilities, community engagement, societal contributions, innovation, technology transfer beyond commercialisation and sustainable development etc. The evolving functions of universities and the adoption of a ‘global and interdisciplinary’ knowledge system² have led to the development of new criteria which should be included in the assessment of university performance and outputs. Motivated by this research gap, the present study emphasises on the aspects of performance assessment, identifies some criteria to reflect the university’s functions in the evolving R&D systems, and provides suggestions for incorporation of these criteria in major ranking frameworks.

RQ: Do international rankings appropriately capture the functions of universities?

Methodology

This study conducts a qualitative review of the existing literature related to the university ranking frameworks and their approaches. In order to effectively look at the appropriateness of university rankings, it is necessary to explore the underlying frameworks used for evaluation of the university’s performance. Three popular ranking systems namely, The Academic Ranking of World Universities (ARWU)/ Shanghai, Quacquarelli Symonds (QS), and Times Higher Education (THE) are considered for this purpose. A comparison of the different criteria and respective indicators used by each ranking framework has been used for this study. These ranking systems utilise various strategies for evaluation of university performances, these include, the general key pillars of ranking systems namely – “teaching”, “research”, “knowledge transfer”, and “global outlook” (Table 2). These indicators have been seen with respect to the roles of universities in the current scenario.

Table 2: Criteria and weights of different indicators considered by ranking frameworks

The assessment exercise assigns different weights to each indicator in these criteria, which relates to the most common criticism of these ranking systems, i.e., assigning arbitrary weights to the indicators in various criteria (Jeremic et al., 2011; Buela-Casal et al., 2007; Van Raan, 2005) as well as the seemingly arbitrary selection of parameters for measuring broader criteria's (Billaut, Bouyssou, & Vincke, 2010; Huang, 2012). For instance, QS rankings use faculty/student ratio at 20% weightage whereas, THE rankings use 5 indicators at a total of 30% and ARWU uses 3 indictors at a total of 50% weightage for assessment of teaching criteria. Secondly, the methodology used for data collection and assessment is also debatable. For example, QS and THE utilize 50% (academic and employer reputation) and 33% (reputation surveys in teaching and research) survey data respectively. Multiple studies have concentrated on the issues of response rates, fairness and weights, uneven distribution of returned questionnaires, and regional bias in the computation of survey-based data. (Billaut, Bouyssou, & Vincke, 2010; Huang, 2012; Jovanovic et al., 2012; Docampo & Cram, 2014; Marginson, 2014). The other issues will focus on problems with the impact of data normalisation on rankings (Jeremic & Jovanovic-Milenkovic, 2014; Moed, 2017; Fauzi et.al, 2020) and inconsistencies between university ranks in different ranking frameworks (Dehon, McCathie, & Verardi, 2010; Saisana, d’Hombres & Saltelli, 2011; Marginson, 2014). However, there is an issue concerned towards relative focus of ARWU only on the specific dimensions and ignorance of dimensions relating to essential aspects of university functions. Teaching quality, graduate employability, social work, and internationalisation etc. are some of the ignored dimensions (Marginson, 2014).

The critical analysis of the three ranking systems brings forth some of the important shortcomings in their methodologies drawing from both existing literature and observation made through understanding of the systems. To put these observations in perspective of research question, the next section explores the idea of university ecosystem and the changing functions of universities with respect to changes in the socio-politico-economic environment.

Analysis

Changing Role of Universities

The traditional functions of universities are teaching and research³ i.e., only focused on Science-Science connect. The “science-science connect” refers to “sharing of ideas and resources within the knowledge ecosystem”. This calls for increased interaction and resource sharing between scientific institutions and knowledge workers so that an optimal usage can be ensured. In their teaching activities, universities provide the education necessary for the overall development of the personality. Whereas, through their research activities universities contribute towards the existing body of theoretical knowledge and its application in resolving practical problems. A lot of this activity is geared towards the requirements of a selective few individuals/groups mainly academician and researchers. However, modern era requires significant contributions of science to society. This implies that scientists and researchers should interact more with society in a manner that they can understand the societal problems and needs, and work towards development of useful solutions. This is termed as “society-science connect” which calls for collaboration between universities and communities to identify problems and develop scientific and technological solutions for them. The interactions between scientists and society can facilitate two-way flow of facts, knowledge and ideas, which in turn can benefit both science and society (Delanty, 2001; Fagerberg, Landström, & Martin, 2012).

In the recent past the universities and the research institution have embraced their changing roles and now becoming centres of innovation and entrepreneurship (Mok, 2012). The most recent examples of this change have been seen in development of technological solutions in healthcare, medicinal products, societal change programmes etc. (Bornmann, 2013). Universities are collaborating with various stakeholders namely industry, government bodies and entrepreneur (Etzkowitz & Zhou, 2017). This UIG collaboration is highly relevant for assessing the quality and capability of universities in the contemporary scenarios (Rupika & Singh, 2016). However, the university ranking systems do not provide necessary importance to this aspect. Out of the three frameworks only one i.e., THE WUR considers Industry income as an indicator with only 2.5% weightage in overall score.

Further, one of the most common criticisms of university research has been its disconnect from the society. More often than not research conducted in universities remains at the proof-of-concept level (TRL1- TRL3)⁴. There is increasing call for promoting research into actionable technology/solution and the university across the globe have started focused towards this, for example, Oxford/AstraZeneca⁵ Vaccine for Covid-19.

Although these perspectives are now gaining their importance in the major roles of university but these perspectives are not captured by the existing ranking framework. For instance, ARWU considers (1) number of publications, (2) achievements of faculties but does not consider the contributions of universities at local levels such as participation in government programmes for entrepreneurship development, technology business incubators, character building for students etc. (Astebro & Bazzazian, 2011). Next section highlights some of the other indicators/criteria’s that are not yet considered in the evaluation of universities performance.

Criteria and indictors which are missing from existing university ranking system

Figure 1: Criteria’s / indictors missing from existing university ranking system

1. Open Science:

A major part of research and development activities in universities are public funded by government agencies. In this context, there has been an increasing call for making the research results available for public use. Open Science practices help to reduce resource demand by making tools and other resources readily available to the researchers. Hence it promotes sustainable research (Govaart, Hofmann & Medawar, 2022). It is also crucial to promote more inclusive research practices by allowing access to data for all⁶. Universities adopting practices for making Science more transparent, reproducible and accessible. But, none of the ranking frameworks include indicators which estimate these activities.

2. Innovation:

Universities in the changing environment are taking up new functions in the national innovation activities. Many countries have started promoting individual level as well as institutional level initiatives to promote manufacturing, infrastructure, societal services and environmental services (Boulton, 2011).

1. University Innovation Centres have been promoted across the world in order to support new technologies from individual entrepreneurs and researchers. For example, Indian government has promoted programs like Atal Innovation Mission and Start-up India; Australian government formulated policies like NISA (National Innovation and Science Agenda) to enhance the local innovation activities.

2. Technology partnerships with overseas universities have also been an increasing among the universities⁷. Bilateral academic collaboration agreements between north-south, north-north, and south-south are common and are also measured in many existing rankings⁸⁹. However, technology partnerships are complex and difficult to capture, as their outputs are not quantified in terms of articles, rather by development of varied resources.

3. Unable to capture severely fragmented research ecosystem (such as specialized institutions focused towards particular field like agriculture, architecture, pharmaceuticals, entrepreneurship and so on).

3. Knowledge Communication & Diffusion, and Social Outreach

University rankings do not give emphasis to the activities done by universities towards the development of cognitive skills and similar learning outcomes (Byrne, 1998). Universities across the globe have shifted focus on Educational and Information dissemination initiatives for the local population in response to the movement for inclusive, sustainable and equitable growth (Larrán Jorge & Andrades Peña, 2017).

4. Societal/Nationally relevant project

The contemporary discourse on role of universities has largely shifted from universities being centres of knowledge creation (through Science-Science Connect) to platforms and facilitators for knowledge dissemination and application (Snellman, 2015; Olcay & Bulu, 2017). They have developed a more intricate mission of contributing towards society by-

1. Partnering with municipal corporations, and other city and state bodies to address local challenges such as water management, traffic, waste management, local area development etc. Natural disasters or calamities, GIS mapping of amenities and resources,

2. Collaboration and guidance with to newly created institutions, resource sharing and student exchange.

3. Facilitating regional development through dialogue and diplomacy.

5. Sustainable Development

Since, the adoption of SDGs, they have become central agenda for activities of major institutions such as universities, government bodies, ministries and civic societies etc. Among these universities hold key-position as an influencer of opinion, sources of expertise in research and education on all sectors. For instance, Universities have started programs for teaching courses on sustainable development (Filho et al.,2022). For example- University College London (UCL)¹⁰, University of London¹¹ offer degree courses based on sustainable development. However, some other universities like University of Copenhagen, Columbia university introduces courses to sustainable development on several online educational platforms like Coursera¹².These offerings are prompted by the initiatives supported by the United Nations such as the Higher Education Sustainability Initiative, the Principles of Responsible Management Education initiative, and the Sustainable Development Solutions Network (SDSN Australia/Pacific 2017).

1. Contribution to development of local ecosystem for sustainable development. Captured in THE Impact Rankings since 2019.

2. Policy interventions in the local governance byelaws. Such as norms for water conservation, waste management, electricity usage, agricultural consultancy to local farmers etc.

Possible approach for including alternative assessment criteria

In the five proposed criteria, indicators for measurement need to be identified. Some emerging ranking agencies like QS, THE, SCIMago are using some indicators (Table 4) for area specific rankings.

Table 4. List of three rankings which cover some of the criteria for changing university functions

In these ranking frameworks, innovation is measured by parameters such as number of patent applications, the number of publications cited in patents, and so on. Other criteria such as, societal impact is evaluated using newly developed parameters of Altmetric mentions, number of web links, and so on¹³. Recent studies have also proposed scientometric parameters to estimate impact on SDGs (Vinuesa et al., 2020, Singh, Kanaujia & Singh, 2022). Similarly, percentage of articles published as bronze, Gold, green or hybrid access can be useful to capture the criteria of Open access. However, more research is needed to determine which parameters are more likely to be used to measure each the five proposed criterion. It is also to be noted that these are separate rankings (Table 4) which do not address the core issue with widely used university rankings. The following section draws insights from the analysis and presents a discussion about potential improvements in the ranking systems.

Conclusion

The quickly evolving technology landscape is reshaping the functions of many institutions and introducing new functions to their repertoire. In case of universities, the approach of focusing on development of individual capacities of organisations has been superseded by the opportunities for interactive learning and collaborative research and education between inter and cross-disciplinary institutions. The focus across different domains has been on developing multiple knowledge and resource sharing mechanisms, encouraging international collaboration and exchange programmes. The adoption of sustainable development paradigm has shifted the focus of developmental activities on regional challenges and engagement of the local stakeholders. Universities are uniquely placed to lead the cross-sectoral implementation of the SDGs, providing an invaluable source of expertise in research and education on all sectors of the SDGs, in addition to being widely considered as neutral and influential players. While the focus of this article is on the functions of universities, it is acknowledged that think tanks and other institutions involved in the production and communication of knowledge also have an important role in advancing the SDG agenda. A set of criteria for the evaluation of changing university functions have been suggested. Innovation, social outreach, societal/nationally relevant projects undertaken by the university, and the dimension of sustainable development if included in the university ranking can further improve the inclusiveness and applicability of ranking systems.

The adoption of these criteria in the popular ranking frameworks will: (a) broaden the assessment framework hence shading the narrow view of university function, and (b) field-specific criteria like SDG, Innovation, and entrepreneurship provide fair opportunity to domain-specific/ specialized institutions/ universities. Hence, this radical change will help to redistribute R & D resources among institutions which may otherwise had been overshadowed by the legacy of the large university with established alumni and formal as well as informal linkages with the stakeholders in STI ecosystem. These changes will truly bring about a shift from trust-based funding to performance-based funding in Science and Technology and related system.

References

Astebro, T., & Bazzazian, N. (2011). Universities, entrepreneurship and local economic development. Handbook of Research on Entrepreneurship and Regional Development: National and Regional Perspectives, 252-333.

Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? An MCDM view. Scientometrics, 84(1), 237-263.

Blanco-Ramírez, J. B., and Berger, G. (2014). Rankings, accreditation, and the international quest for quality: Organizing an approach to value in higher education. Quality Assurance in Education, 22(1).

Bok, D. (2009). Universities in the Marketplace. In Universities in the Marketplace. Princeton university press.

Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for information science and technology, 64(2), 217-233.

Boulton, G. (2011). University rankings: Diversity, excellence and the European initiative. Procedia-Social and Behavioral Sciences, 13, 74-82.

Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349-365.

Byrne, J. V. (1998). Outreach, engagement, and the changing culture of the university. Journal of Higher Education Outreach and Engagement, 3(2), 3-8.

Dehon, C., McCathie, A., & Verardi, V. (2010). Uncovering excellence in academic rankings: A closer look at the Shanghai ranking. Scientometrics, 83(2), 515-524.

Delanty, G. (2001). The university in the knowledge society. Organization, 8(2), 149-153.

Docampo, D., & Cram, L. (2014). On the internal dynamics of the Shanghai ranking. Scientometrics, 98(2), 1347-1366.

Etzkowitz, H., & Zhou, C. (2017). The triple helix: University–industry–government innovation and entrepreneurship. Routledge.

Fagerberg, J., Landström, H., & Martin, B. R. (2012). Exploring the emerging knowledge base of ‘the knowledge society’. Research Policy, 41(7), 1121-1131.

Fauzi, M. A., Tan, C. N. L., Daud, M., & Awalludin, M. M. N. (2020). University rankings: A review of methodological flaws. Issues in Educational Research, 30(1), 79-96.

Goglio, V. (2016). One size fits all? A different perspective on university rankings. Journal of Higher Education Policy and Management, 38(2), 212-226.

Govaart, G. H., Hofmann, S. M., & Medawar, E. (2022). The sustainability argument for open science. Collabra: Psychology, 8(1), 35903.

Hazelkorn, E. (2007). The impact of league tables and ranking systems on higher education decision making. Higher education management and policy, 19(2), 1-24.

Huang, M. H. (2012). Opening the black box of QS World University Rankings. Research Evaluation, 21(1), 71-78.

Jeremic, V., & Jovanovic-Milenkovic, M. (2014). Evaluation of Asian university rankings: position and perspective of leading Indian higher education institutions. Current Science, 1647-1653.

Jeremic, V., Bulajic, M., Martic, M., & Radojicic, Z. (2011). A fresh approach to evaluating the academic ranking of world universities. Scientometrics, 87(3), 587-596.

Johnes, J. (2018). University rankings: What do they really show?  Scientometrics, 115(1), 585-606.

Jovanovic, M., Jeremic, V., Savic, G., Bulajic, M., & Martic, M. (2012). How does the normalization of data affect the ARWU ranking? Scientometrics, 93(2), 319-327.

Kokko, H., & Sutherland, W. J. (1999). What do impact factors tell us? Trends in Ecology & Evolution, 14(10), 382-384.

Larrán Jorge, M., & Andrades Peña, F. J. (2017). Analysing the literature on university social responsibility: A review of selected higher education journals. Higher Education Quarterly, 71(4), 302-319.

Lathabai, H. H., Nandy, A., & Singh, V. K. (2021). x-index: Identifying core competency and thematic research strengths of institutions using an NLP and network based ranking framework. Scientometrics, 126(12), 9557-9583.

Leal Filho, W., Salvia, A. L., Vasconcelos, C. R. P., Anholon, R., Rampasso, I. S., Eustachio, J. H. P. P., ... & Sharifi, A. (2022). Barriers to institutional social sustainability. Sustainability Science, 1-16.

Leimu, R., & Koricheva, J. (2005). What determines the citation frequency of ecological papers?. Trends in ecology & evolution, 20(1), 28-32.

Liu, N. C., Cheng, Y., & Liu, L. (2005). Academic ranking of world universities using scientometrics-A comment to the “Fatal Attraction”. Scientometrics, 64(1), 101-109.

Marginson, S. (2014). University rankings and social science. European journal of education, 49(1), 45-59.

Moed, H. F. (2017). A comparative study of five world university rankings. In Applied Evaluative Informetrics (pp. 261-285). Springer, Cham.

Mok, K. H. (2012). The quest for innovation and entrepreneurship: The changing role of university in East Asia. Globalisation, Societies and Education, 10(3), 317-335.

Mussard, M., & James, A. P. (2018). Engineering the global university rankings: Gold standards, limitations and implications. IEEE Access, 6, 6765-6776.

Olcay, G. A., & Bulu, M. (2017). Is measuring the knowledge creation of universities possible?: A review of university rankings. Technological Forecasting and Social Change, 123, 153-160.

Pusser, B., & Marginson, S. (2013). University rankings in critical perspective. The journal of higher education, 84(4), 544-568.

Radder, H. (Ed.). (2010). The commodification of academic research: Science and the modern university (pp. 231-58). Pittsburgh: University of Pittsburgh Press.

Rijcke, S. D., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2016). Evaluation practices and effects of indicator use—a literature review. Research evaluation, 25(2), 161-169.

Rupika, A. U., & Singh, V. K. (2016). Measuring the university–industry–government collaboration in Indian research output. Current Science, 1904-1909.

Saisana, M., d’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university rankings and policy implications. Research policy, 40(1), 165-177.

Shore, C. (2010). Beyond the multiversity: Neoliberalism and the rise of the schizophrenic university. Social Anthropology/Anthropologie Sociale, 18(1), 15-29.

Singh, A., Kanaujia, A., & Singh, V. K. (2022). Research on Sustainable Development Goals: How has Indian Scientific Community Responded?. Journal of Scientific & Industrial Research, 81(11), 1147-1161.

Slaughter, S., Slaughter, S. A., & Rhoades, G. (2004). Academic capitalism and the new economy: Markets, state, and higher education. JHU Press.

Snellman, C. L. (2015). University in knowledge society: Role and challenges. Journal of System and Management Sciences, 5(4), 84-113.

Stergiou, K. I., & Lessenich, S. (2014). On impact factors and university rankings: from birth to boycott. Ethics in Science and Environmental Politics, 13(2), 101-111.

Thakur, M. (2007). The impact of ranking systems on higher education and its stakeholders. Journal of Institutional Research, 13(1), 83-96.

Usher, A., & Savino, M. (2006). A World of Difference: A Global Survey of University League Tables. Canadian Education Report Series. Online Submission.

Van Leeuwen, T. N., Moed, H. F., & Reedijk, J. (1999). Critical comments on Institute for Scientific Information impact factors: a sample of inorganic molecular chemistry journals. Journal of Information Science, 25(6), 489-498.

Van Raan, A. F. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133-143.

Vernon, M. M., Balas, E. A., & Momani, S. (2018). Are university rankings useful to improve research? A systematic review. PloS one, 13(3), e0193762.

Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., ... & Fuso Nerini, F. (2020). The role of artificial intelligence in achieving the Sustainable Development Goals. Nature communications, 11(1), 233.

Wong, B. B., & Kokko, H. (2005). Is science as global as we think? Trends in ecology & evolution, 20(9), 475-476.