The international University rankings are now increasingly used for various purposes, including for funding decisions. Various stakeholders use ranking assessments which are based on narrow set of criteria namely teaching, research, knowledge transfer, global outlook. This set of criteria have been criticised in different contexts. This paper presents an analysis on the need for updating the ranking frameworks in the context of changing functions of the Universities. A brief review of the popular ranking frameworks (QS, THE and ARWU) are presented. This is followed by a discussion on some new criteria and their evaluation for inclusion in the ranking framework in order to capture the changing university functions.
International University Rankings and Changing University Functions: Underlining indicators to capture societal and entrepreneurial contributions of universities
Solanki Gupta1, Anurag Kanaujia1, Sujit
Bhattacharya2, Vivek Kumar Singh1
0000-0002-5779-2948, 0000-0002-5813-7427, 0000-0002-6769-8923,
0000-0002-7348-6545
1Department of Computer Science, Banaras Hindu University, Varanasi, India
2CSIR- National Institute of Science Communication and Policy Research (NIScPR), New Delhi, India
Abstract
The international University rankings are now increasingly used for various purposes, including for funding decisions. Various stakeholders use ranking assessments which are based on narrow set of criteria namely teaching, research, knowledge transfer, global outlook. This set of criteria have been criticised in different contexts. This paper presents an analysis on the need for updating the ranking frameworks in the context of changing functions of the Universities. A brief review of the popular ranking frameworks (QS, THE and ARWU) are presented. This is followed by a discussion on some new criteria and their evaluation for inclusion in the ranking framework in order to capture the changing university functions.
Keywords: Institutional rankings, STI ecosystem, Higher education, Performance Assessment, Societal development.
The university ranking frameworks since their origin in 1980s (US News & World Report1), and wider adoption after the Shanghai league table in 2003 (Stergiou & Lessenich, 2014), have gained much importance among academic as well as administrative groups. Issued by a third party, these rankings make it easy to quantify the achievements of universities and compare them with each other. Thus, universities use the rankings to satisfy the public demand for transparency and information (Usher & Savino, 2006) and also to secure funding and resources. At least ten global and as many regional university ranking frameworks are currently active (Vernon, Balas & Momani, 2018).
An explanation for the success of rankings originates from their ability to make heterogeneous characteristics of universities comparable through classification, normalization and standardization (De Rijcke et al., 2016), thus allowing universities to take part in a global higher education market driven by a competition for expertise, reputation, students and money. In the recent times, the role of funding and reputation has become even more pronounced, for these determine the quantity and quality of students, researchers, and funding support received by the universities worldwide (Lathabai et al., 2021). Further, factors like advancement of research in fields of statistics and bibliometrics, internationalization, globalization and economization in higher education have contributed to popularisation of university rankings.
The popularity of university rankings is accompanied by growth in the amount of literature related to their significance, role and methodologies (Blanco-Ramírez & Berger, 2014; Thakur, 2007; Hazelkorn, 2007; ALİ, 2022). These studies are classified into three broader categories, namely studies looking at the origins (epistemological explorations of university roles and their role in development of systems for rankings) (Slaughter, Slaughter & Rhoades, 2004; Bok, 2009; Radder, 2010; Shore, 2010; Brankovic, Hamann & Ringel, 2023), methodologies used for assessments (Huang, 2012; Van Raan, 2005; Moed, 2017) and studies about the influence and effects of ranking exercises on the university ecosystem (Thakur, 2007; Hazelkorn, 2007) (Table1).
Table 1. Studies coving different criteria of discussions of the university rankings
These studies, however, have not looked at the new roles taken up by universities with respect to the changing environment and university ecosystem. Universities are now focused towards social responsibilities, community engagement, societal contributions, innovation, technology transfer beyond commercialisation and sustainable development etc. The evolving functions of universities and the adoption of a ‘global and interdisciplinary’ knowledge system2 have led to the development of new criteria which should be included in the assessment of university performance and outputs. Motivated by this research gap, the present study emphasises on the aspects of performance assessment, identifies some criteria to reflect the university’s functions in the evolving R&D systems, and provides suggestions for incorporation of these criteria in major ranking frameworks.
RQ: Do international rankings appropriately capture the changing functions of universities?
This study conducts a qualitative review of the existing literature related to the university ranking frameworks and their approaches. In order to effectively look at the appropriateness of university rankings, it is necessary to explore the underlying frameworks used for evaluation of the university’s performance. Three popular ranking systems namely, The Academic Ranking of World Universities (ARWU)/ Shanghai, Quacquarelli Symonds (QS), and Times Higher Education (THE) are considered for this purpose. A comparison of the different criteria and respective indicators used by each ranking framework has been used for this study. These ranking systems utilise various strategies for evaluation of university performances, these include, the general key pillars of ranking systems namely – “teaching”, “research”, “knowledge transfer”, and “global outlook” (Table 2). These indicators have been seen with respect to the roles of universities in the current scenario.
Table 2. Criteria and weights of different indicators considered by ranking frameworks
The assessment exercise assigns different weights to each indicator in these criteria, which relates to the most common criticism of these ranking systems, i.e., assigning arbitrary weights to the indicators in various criteria (Jeremic et al., 2011; Buela-Casal et al., 2007; Van Raan, 2005) as well as the seemingly arbitrary selection of parameters for measuring broader criteria's (Billaut, Bouyssou, & Vincke, 2010; Huang, 2012). For instance, QS rankings use faculty/student ratio at 20% weightage whereas, THE rankings use 5 indicators at a total of 30% and ARWU uses 3 indictors at a total of 50% weightage for assessment of teaching criteria. Secondly, the methodology used for data collection and assessment is also debatable. For example, QS and THE utilize 50% (academic and employer reputation) and 33% (reputation surveys in teaching and research) survey data respectively. Multiple studies have concentrated on the issues of response rates, fairness and weights, uneven distribution of returned questionnaires, and regional bias in the computation of survey-based data. (Billaut, Bouyssou, & Vincke, 2010; Huang, 2012; Jovanovic et al., 2012; Docampo & Cram, 2014; Marginson, 2014). The other issues highlighted include problems with the data normalisation (Jeremic & Jovanovic-Milenkovic, 2014; Moed, 2017; Fauzi et.al, 2020) and inconsistencies between university ranks in different ranking frameworks (Dehon, McCathie, & Verardi, 2010; Saisana, d’Hombres & Saltelli, 2011; Marginson, 2014). Teaching quality, graduate employability, social work, and internationalisation etc. are some of the ignored dimensions in ARWU rankings (Marginson, 2014).
To put the observations from this analysis in perspective of research question, the next section explores the idea of university ecosystem and the changing functions of universities with respect to changes in the socio-politico-economic environment.
The traditional functions of universities are teaching and research3 i.e., only focused on Science-Science connect.4 Whereas, personality development through education and knowledge creation through research are the goals of the teaching. The modern era requires significant contributions of science to society from universities. This implies that scientists and researchers should interact more with society in a manner that they can understand the societal problems and needs, and work towards development of useful solutions. This is termed as “society-science connect” which calls for collaboration between universities and communities to identify problems and develop scientific and technological solutions for them. The interactions between scientists and society can facilitate two-way flow of facts, knowledge and ideas, which in turn can benefit both science and society (Delanty, 2001; Fagerberg, Landström, & Martin, 2012).
In the recent past the universities and the research institution have embraced their changing roles and now becoming centres of innovation and entrepreneurship (Mok, 2012). The most recent examples of this change have been seen in development of technological solutions in healthcare, medicinal products, societal change programmes etc. (Bornmann, 2013). Universities are collaborating with various stakeholders namely industry, government bodies and entrepreneur (Etzkowitz & Zhou, 2017). This UIG collaboration is highly relevant for assessing the quality and capability of universities in the contemporary scenarios (Rupika & Singh, 2016). However, the university ranking systems do not provide necessary importance to this aspect. Out of the three frameworks only one i.e., THE WUR considers Industry income as an indicator with only 2.5% weightage in overall score. Further, one of the most common criticisms of university research has been its disconnect from the society. More often than not research conducted in universities remains at the proof-of-concept level (TRL1- TRL3). There is increasing call for promoting research into actionable technology/solution and the university across the globe have started focused towards this, for example, Oxford/AstraZeneca Vaccine for Covid-19.
Although these perspectives are now gaining their importance in the major roles of university but these perspectives are not captured by the existing ranking framework. For instance, ss noted in analysis the criteria for knowledge transfer, societal engagement, contributions of universities at local levels such as participation in government programmes for entrepreneurship development, technology business incubators, character building for students etc. (Astebro & Bazzazian, 2011) are not included in any of the university rankings. The next section highlights some criteria’s that are supported by existing literature and are not yet considered in the evaluation of universities performance.
Open Science:
A major part of research and development activities in universities are public funded by government agencies. In this context, there has been an increasing call for making the research results available for public use. Open Science practices help to reduce resource demand by making tools and other resources readily available to the researchers. Hence it promotes sustainable research (Govaart, Hofmann & Medawar, 2022). It is also crucial to promote more inclusive research practices by allowing access to data for all5. Universities adopting practices for making Science more transparent, reproducible and accessible. But, none of the ranking frameworks include indicators which estimate these activities.
Innovation:
Universities in the changing environment are taking up new functions in the national innovation activities. Many countries have started promoting individual level as well as institutional level initiatives to promote manufacturing, infrastructure, societal services and environmental services (Boulton, 2011).
University Innovation Centres have been promoted across the world in order to support new technologies from individual entrepreneurs and researchers. For example, Indian government has promoted programs like Atal Innovation Mission and Start-up India; Australian government formulated policies like NISA (National Innovation and Science Agenda) to enhance the local innovation activities.
Technology partnerships with overseas universities have also been an increasing among the universities6. Bilateral academic collaboration agreements between north-south, north-north, and south-south are common and are also measured in many existing rankings78. However, technology partnerships are complex and difficult to capture, as their outputs are not quantified in terms of articles, rather by development of varied resources.
Unable to capture severely fragmented research ecosystem (such as specialized institutions focused towards particular field like agriculture, architecture, pharmaceuticals, entrepreneurship and so on).
Knowledge Communication & Diffusion, and Social Outreach
University rankings do not give emphasis to the activities done by universities towards the development of cognitive skills and similar learning outcomes (Byrne, 1998). Universities across the globe have shifted focus on Educational and Information dissemination initiatives for the local population in response to the movement for inclusive, sustainable and equitable growth (Larrán Jorge & Andrades Peña, 2017).
Societal/Nationally relevant project
The contemporary discourse on role of universities has largely shifted from universities being centres of knowledge creation (through Science-Science Connect) to platforms and facilitators for knowledge dissemination and application (Snellman, 2015; Olcay & Bulu, 2017). They have developed a more intricate mission of contributing towards society by-
Partnering with municipal corporations, and other city and state bodies to address local challenges such as water management, traffic, waste management, local area development etc. Natural disasters or calamities, GIS mapping of amenities and resources,
Collaboration and guidance with to newly created institutions, resource sharing and student exchange.
Facilitating regional development through dialogue and diplomacy.
Sustainable Development
Since, the adoption of SDGs, they have become central agenda for activities of major institutions such as universities, government bodies, ministries and civic societies etc. Among these universities hold key-position as an influencer of opinion, sources of expertise in research and education on all sectors. For instance, Universities have started programs for teaching courses on sustainable development (Filho et al.,2022). For example- University College London (UCL) 9, University of London10 offer degree courses based on sustainable development. However, some other universities like University of Copenhagen, Columbia university introduces courses related to sustainable development on several online educational platforms like Coursera11.These offerings are prompted by the initiatives supported by the United Nations such as the Higher Education Sustainability Initiative, the Principles of Responsible Management Education initiative, and the Sustainable Development Solutions Network (SDSN Australia/Pacific 2017).
Contribution to development of local ecosystem for sustainable development. Captured in THE Impact Rankings since 2019.
Policy interventions in the local governance byelaws. Such as norms for water conservation, waste management, electricity usage, agricultural consultancy to local farmers etc.
In the five proposed criteria, indicators for measurement need to be identified. Some emerging ranking agencies like QS, THE, SCIMago are using some indicators (Table 4) for area specific rankings.
Table 4. List of three rankings which cover some of the criteria for changing university functions
New Ranking Frameworks | Included Ranking Criteria | Weightage Assigned |
---|---|---|
QS Star Rankings |
|
N/A |
SciMago Research and Innovation Rankings |
|
30% 20% |
THE Impact Rankings by SDGs |
|
26% each for Top Three SDGs 22% for SDG 17 |
QS Sustainability Rankings |
|
N/A |
In these ranking frameworks, the percentage of articles published as bronze, gold, green or hybrid access can be useful to capture the criteria of Open access. Similarly, innovation is measured by parameters such as number of patent applications, publications cited in patents, and collaboration activities. Other criteria such as, societal impact is evaluated using newly developed parameters of Altmetric mentions, number of web links, and so on12. Recent studies have also proposed scientometric parameters to estimate impact on SDGs (Vinuesa et al., 2020, Singh, Kanaujia & Singh, 2022). However, more research is needed to determine which parameters are more likely to be used to measure each the five proposed criterion. It is also to be noted that these are separate rankings (Table 4) which do not address the core issue with widely used university rankings. The following section draws insights from the analysis and presents a discussion about potential improvements in the ranking systems.
The quickly evolving technology landscape is reshaping the functions of many institutions and introducing new functions to their repertoire. In case of universities, the approach of focusing on development of individual capacities of organisations has been superseded by the opportunities for interactive learning and collaborative research and education between inter and cross disciplinary institutions. The focus across different domains has been on developing multiple knowledge and resource sharing mechanisms, encouraging international collaboration and exchange programmes. The adoption of sustainable development paradigm has shifted the focus of developmental activities on regional challenges and engagement of the local stakeholders. Universities are uniquely placed to lead the cross-sectoral implementation of the SDGs, providing an invaluable source of expertise in research and education on all sectors of the SDGs, in addition to being widely considered as neutral and influential players. A set of criteria for the evaluation of changing university functions have been suggested. Innovation, social outreach, societal/nationally relevant projects undertaken by the university, and the dimension of sustainable development if included in the university ranking can further improve the inclusiveness and applicability of ranking systems.
The adoption of these criteria in the popular ranking frameworks will -
broaden the assessment framework and hence more capable of capturing the changing functions of the universities,
allow appropriate inclusion of field specific criteria like SDG, innovation, entrepreneurship etc. which in turn will help in making the university rankings more representative.
There is a well-established need for making the university ranking more broad-based, inclusive and representative. This has become more important in the context of changes in the functions and activities carried out by the universities in the present time. Inclusion of the criteria suggested may help in making the university rankings more appropriate and relevant.
References
ALİ, M. G. A. (2022). A General Perspective about Institutional Rankings, Ranking Framework, Benefits of Rankings and Ranking Methodological Flaws and Best Approach for Being A World Class Institution. International Journal of Educational Research Review, 7(3), 157-164.
Astebro, T., & Bazzazian, N. (2011). Universities, entrepreneurship and local economic development. Handbook of Research on Entrepreneurship and Regional Development: National and Regional Perspectives, 252-333.
Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? An MCDM view. Scientometrics, 84(1), 237-263.
Blanco-Ramírez, J. B., & Berger, G. (2014). Rankings, accreditation, and the international quest for quality: Organizing an approach to value in higher education. Quality Assurance in Education, 22(1).
Bok, D. (2009). Universities in the Marketplace. In Universities in the Marketplace. Princeton university press.
Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for information science and technology, 64(2), 217-233.
Boulton, G. (2011). University rankings: Diversity, excellence and the European initiative. Procedia-Social and Behavioral Sciences, 13, 74-82.
Brankovic, J., Hamann, J., & Ringel, L. (2023). The institutionalization of rankings in higher education: continuities, interdependencies, engagement. Higher Education, 1-13. https://doi.org/10.1007/s10734-023-01018-8
Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349-365.
Byrne, J. V. (1998). Outreach, engagement, and the changing culture of the university. Journal of Higher Education Outreach and Engagement, 3(2), 3-8.
Dehon, C., McCathie, A., & Verardi, V. (2010). Uncovering excellence in academic rankings: A closer look at the Shanghai ranking. Scientometrics, 83(2), 515-524.
Delanty, G. (2001). The university in the knowledge society. Organization, 8(2), 149-153.
Docampo, D., & Cram, L. (2014). On the internal dynamics of the Shanghai ranking. Scientometrics, 98(2), 1347-1366.
Etzkowitz, H., & Zhou, C. (2017). The triple helix: University–industry–government innovation and entrepreneurship. Routledge.
Fagerberg, J., Landström, H., & Martin, B. R. (2012). Exploring the emerging knowledge base of ‘the knowledge society’. Research Policy, 41(7), 1121-1131.
Fauzi, M. A., Tan, C. N. L., Daud, M., & Awalludin, M. M. N. (2020). University rankings: A review of methodological flaws. Issues in Educational Research, 30(1), 79-96.
Goglio, V. (2016). One size fits all? A different perspective on university rankings. Journal of Higher Education Policy and Management, 38(2), 212-226.
Govaart, G. H., Hofmann, S. M., & Medawar, E. (2022). The sustainability argument for open science. Collabra: Psychology, 8(1), 35903.
Hazelkorn, E. (2007). The impact of league tables and ranking systems on higher education decision making. Higher education management and policy, 19(2), 1-24.
Huang, M. H. (2012). Opening the black box of QS World University Rankings. Research Evaluation, 21(1), 71-78.
Jeremic, V., & Jovanovic-Milenkovic, M. (2014). Evaluation of Asian university rankings: position and perspective of leading Indian higher education institutions. Current Science, 1647-1653.
Jeremic, V., Bulajic, M., Martic, M., & Radojicic, Z. (2011). A fresh approach to evaluating the academic ranking of world universities. Scientometrics, 87(3), 587-596.
Johnes, J. (2018). University rankings: What do they really show? Scientometrics, 115(1), 585-606.
Jovanovic, M., Jeremic, V., Savic, G., Bulajic, M., & Martic, M. (2012). How does the normalization of data affect the ARWU ranking? Scientometrics, 93(2), 319-327.
Kokko, H., & Sutherland, W. J. (1999). What do impact factors tell us? Trends in Ecology & Evolution, 14(10), 382-384.
Larrán Jorge, M., & Andrades Peña, F. J. (2017). Analysing the literature on university social responsibility: A review of selected higher education journals. Higher Education Quarterly, 71(4), 302-319.
Lathabai, H. H., Nandy, A., & Singh, V. K. (2021). x-index: Identifying core competency and thematic research strengths of institutions using an NLP and network based ranking framework. Scientometrics, 126(12), 9557-9583.
Leal Filho, W., Salvia, A. L., Vasconcelos, C. R. P., Anholon, R., Rampasso, I. S., Eustachio, J. H. P. P., ... & Sharifi, A. (2022). Barriers to institutional social sustainability. Sustainability Science, 1-16.
Leimu, R., & Koricheva, J. (2005). What determines the citation frequency of ecological papers?. Trends in ecology & evolution, 20(1), 28-32.
Liu, N. C., Cheng, Y., & Liu, L. (2005). Academic ranking of world universities using scientometrics-A comment to the “Fatal Attraction”. Scientometrics, 64(1), 101-109.
Marginson, S. (2014). University rankings and social science. European journal of education, 49(1), 45-59.
Moed, H. F. (2017). A comparative study of five world university rankings. In Applied Evaluative Informetrics (pp. 261-285). Springer, Cham.
Mok, K. H. (2012). The quest for innovation and entrepreneurship: The changing role of university in East Asia. Globalisation, Societies and Education, 10(3), 317-335.
Mussard, M., & James, A. P. (2018). Engineering the global university rankings: Gold standards, limitations and implications. IEEE Access, 6, 6765-6776.
Olcay, G. A., & Bulu, M. (2017). Is measuring the knowledge creation of universities possible?: A review of university rankings. Technological Forecasting and Social Change, 123, 153-160.
Pusser, B., & Marginson, S. (2013). University rankings in critical perspective. The journal of higher education, 84(4), 544-568.
Radder, H. (Ed.). (2010). The commodification of academic research: Science and the modern university (pp. 231-58). Pittsburgh: University of Pittsburgh Press.
Rijcke, S. D., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2016). Evaluation practices and effects of indicator use—a literature review. Research evaluation, 25(2), 161-169.
Rupika, A. U., & Singh, V. K. (2016). Measuring the university–industry–government collaboration in Indian research output. Current Science, 1904-1909.
Saisana, M., d’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university rankings and policy implications. Research policy, 40(1), 165-177.
Shore, C. (2010). Beyond the multiversity: Neoliberalism and the rise of the schizophrenic university. Social Anthropology/Anthropologie Sociale, 18(1), 15-29.
Singh, A., Kanaujia, A., & Singh, V. K. (2022). Research on Sustainable Development Goals: How has Indian Scientific Community Responded?. Journal of Scientific & Industrial Research, 81(11), 1147-1161.
Slaughter, S., Slaughter, S. A., & Rhoades, G. (2004). Academic capitalism and the new economy: Markets, state, and higher education. JHU Press.
Snellman, C. L. (2015). University in knowledge society: Role and challenges. Journal of System and Management Sciences, 5(4), 84-113.
Stergiou, K. I., & Lessenich, S. (2014). On impact factors and university rankings: from birth to boycott. Ethics in Science and Environmental Politics, 13(2), 101-111.
Thakur, M. (2007). The impact of ranking systems on higher education and its stakeholders. Journal of Institutional Research, 13(1), 83-96.
Usher, A., & Savino, M. (2006). A World of Difference: A Global Survey of University League Tables. Canadian Education Report Series. Online Submission.
Van Leeuwen, T. N., Moed, H. F., & Reedijk, J. (1999). Critical comments on Institute for Scientific Information impact factors: a sample of inorganic molecular chemistry journals. Journal of Information Science, 25(6), 489-498.
Van Raan, A. F. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133-143.
Vernon, M. M., Balas, E. A., & Momani, S. (2018). Are university rankings useful to improve research? A systematic review. PloS one, 13(3), e0193762.
Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., ... & Fuso Nerini, F. (2020). The role of artificial intelligence in achieving the Sustainable Development Goals. Nature communications, 11(1), 233.
Wong, B. B., & Kokko, H. (2005). Is science as global as we think? Trends in ecology & evolution, 20(9), 475-476.
https://www.usnews.com/education/best-global-universities/rankings↩︎
By global and interdisciplinary knowledge system, we refer to the extensive network of institutions and individuals working together on knowledge production across disciplinary boundaries. For instance, use of Artificial Intelligence tools is as common in medical sciences, environmental sciences and physical sciences as much as in engineering.↩︎
http://www.professor-frithjof-kuhnen.de/publications/agricultural-colleges/2.htm↩︎
The “science-science connect” refers to “sharing of ideas and resources within the knowledge ecosystem”. This calls for increased interaction and resource sharing between scientific institutions and knowledge workers so that an optimal usage can be ensured.↩︎
Open science: enabling a more inclusive, collaborative, transparent world of research (elsevier.com)↩︎
DWIH (dwih-netzwerk.de): https://www.dwih-netzwerk.de/en/↩︎
IFCPAR/CEFIPRA: http://www.cefipra.org/↩︎
Indo-Korea Science & Technology Centre (ikst.res.in): https://ikst.res.in/↩︎
https://www.ucl.ac.uk/short-courses/search-courses/educating-sustainable-development-schools-and-universities↩︎
https://www.london.ac.uk/courses/sustainable-development↩︎
https://www.coursera.org/courses?query=sustainable%20development↩︎
Methodology (scimagoir.com)↩︎
Gupta, S., Kanaujia, A., Bhattacharya, S. & singh, V. (2023). International University Rankings and Changing University Functions: Underlining indicators to capture societal and entrepreneurial contributions of universities [preprint]. 27th International Conference on Science, Technology and Innovation Indicators (STI 2023). https://doi.org/10.55835/644281c46972d54225e8f684
No reviews to show. Please remember to LOG IN as some reviews may be only visible to specific users.