Evaluating international strategic partnerships between universities

Carole de Bordes*, Leonie van Drooge**, Rhodri Leng***, Rodrigo Liscovsky***, Mayline Strouk***, Niki Vermeulen***

^*c.a.m.de.bordes@cwts.leidenuniv.nl

0000-0002-1773-0259

Centre for Science and Technology Studies (CWTS), University of Leiden, The Netherlands

^** leonie@lvd-impact-evaluatie.nl

0000-0002-7177-4586 

LvD Impact & Evaluation. Utrecht, The Netherlands

^** Rhodri.Leng@ed.ac.uk; rliscovs@ed.ac.uk; m.strouk@sms.ed.ac.uk; niki.vermeulen@ed.ac.uk

0000-0002-4861-826X; 0000-0002-3240-550X; 0000-0002-9164-4578; 0000-0001-8080-855X

Science, Technology and Innovation Studies (STIS), University of Edinburgh, United Kingdom

in collaboration with Mara Constantinescu (Leiden University), Thommy Gatling (University of Sydney), Johanna Kolhinen (University of Helsinki), Derek MacLeod (University of Edinburgh) and other Evaluate project partners from University of Copenhagen and University College Dublin.

Abstract: The University of Edinburgh is involved in a range of strategic partnerships, and noticed a lack of consistent, aligned evaluation practices. It proved a challenge to assess whether to enter into an agreement with a potential partner, or whether a strategic partnership indeed delivered its expectations, let alone to discuss this with their strategic partners. They invited five of their strategic partners to join hands to develop an evaluation framework, based on state-of-the-art literature leading to clear assessments of strategic partnerships. Leiden University’s CWTS and the University of Edinburgh's Science, Technology and Innovation Studies (STIS) were asked to contribute to the project given expertise in research governance and assessment, including collaboration and internationalization. We decided to co-create the framework with the international officers, which resulted in an evaluation framework. This looks unlike anything imagined, but it is changing the way in which evaluation thinking is integrated in practice.

Introduction

How to evaluate strategic partnerships? International officers of six universities asked this simple yet challenging question. Researchers collaborate across borders and continents. Students go on exchange and go study abroad. Nothing new so far. Yet formalised international strategic partnerships between universities are more recent. These more formal agreements between universities include both research and education and cover a range of departments. As they are expected to contribute to strategic goals and have great impact, the question of evaluation becomes prominent. However, it soon became clear that there is no precedent for the evaluation of strategic partnerships and that also the term of strategic partnership has different meanings. Given the lack of literature on the evaluation of internationalisation arrangements, the project team decided to develop a framework from scratch. It chose a co-creative approach to make sure that the framework was embedded in both theory and practice. Scholars from Leiden University and the University of Edinburgh guided staff in international offices of the six universities through the evaluation of a specific strategic partnership. The scholars then iteratively developed the framework, and adjusted it based on the feedback and responses of the international office staff. Only through mutual exploration of the practice of international strategic partnerships it became possible to find ways to mobilise existing insights from the evaluation literature and integrate this in a meaningful way. As such, the creation of an evaluation framework for strategic partnerships between universities was an experiment in co-creation, developing evaluation in practice with those using the framework. By working on the development of the framework from the start, via a case study approach, we made sure that the resulting framework fits existing practice and can more easily be integrated in ongoing partnership work. This has enhanced the capability for implementation and allowed project participants to communicate results and spread evaluative thinking.

The co-creation approach we took in developing the evaluation framework fits with current calls for reforming evaluation culture (CoARA, 2022). First of all, we steered away from purely quantitative approaches. Although some universities had a good overview of investments on the one hand, and outputs in terms of publications and funding on the other hand, we agreed that good evaluation requires more than measuring what can be quantified. We started by introducing the evaluative cycle, showing how good evaluation practice is integrated throughout the life cycle of a partnership, and that a range of methods can be used to answer evaluative questions, developing mixed-method approaches (Better Evaluation 2023). In line with this, we connected to current shifts in evaluation culture, moving away from accountability towards a formative role for evaluation (Molas-Gallart et al. 2021, Dinges et al. 2020). Moreover, we introduced evaluation with the partner, so the “prime goal of evaluation shifts from accountability to communication between partners – regarding goals and research design - and to mutual learning” (Spaapen 2015, Joly and Matt 2022). As such, the development of our framework contributes to novel approaches to evaluation policy that challenge current practices. In fact, our process approach can be viewed as an experiment, and we hope it can inspire the development of other types of frameworks and contribute to policy learning.

In this paper, we will lay out the development process of the evaluation framework, step by step. Thereby it is important to note that we did not have a clear vision and outline of these steps at the start, but we developed them in interaction and through ongoing discussions with all partners involved (six universities), continuously adapting our approach when needed. At the beginning of the project, there was quite a big knowledge gap between the academics and international officers participating in the project. The international officers did not know much about evaluation, but the academics involved had no clear idea about the ways in which partnerships are built and managed, which is key when developing an evaluation framework for those partnerships. Therefore, it took some time to establish common ground and vocabularies, but we were able to increase mutual understanding and learning throughout the project, which was also crucial in designing the development process. Moreover, we established an international advisory board, with whom we discussed our approach, progress, and results.

The paper follows the different development phases: we started with defining international strategic partnerships, surveying relevant literatures. This was complemented by a case study approach, which provided good insight into the implementation and evaluation of strategic partnerships. This led to the development of a framework presented in the EVALUATE handbook, launched during the 2022 annual conference of the International Association of Universities (IAU) in Dublin.¹

1. Defining international strategic partnerships

While there is a wide body of knowledge on evaluation in general, there is not any work available on evaluation of international strategic partnerships in universities. Moreover, literature on all aspects of international strategic partnerships is scarce, as the term is not well defined and established in the literature. In fact, a broad and straightforward search query to find literature on “strategic partnerships” returns 449 articles, reviews, and book chapters, but many of these are not relevant for our purposes, or at least only indirectly so, such as the large literature on strategic partnerships between private firms and between nation states.

Only 19 documents are classified into categories associated with higher education or Higher Education Institution (HEI) research management, but these 19 papers neither cite one another nor do they reference the same literature, reflecting that these papers have very different subject matter to one another. Only 11 papers appeared to concern activities related to Higher Education Institution-International Strategic Partnerships (HEI-ISPs), with seven focusing on describing and analysing particular partnerships between HEIs or between sets of HEIs in different regions and only four papers making contributions that have a general scope that can feed into the EVALUATE project (Kristensen & Karlsen, 2018; Woodfield, 2018; Arrowood & Hitch, 2016; Otieno & Otieno, 2016).

As such, we continued the literature review by (i) examining the complexities involved in defining strategic partnerships and (ii) surveying the academic literature on strategic partnerships through more complex search techniques. Through this, we found a great diversity of different arrangements between HEIs that are currently labelled under the "strategic partnership" banner, including student mobility partnerships, international branch campuses, policy advocacy arrangements, and research arrangements. Furthermore, there is a plethora of different strategic objectives underlying the inception of different partnerships, such as enhancing market competiveness, addressing particular societal goals, and empowering students. This heterogeneity in both types of arrangement and their different strategic objectives represents a significant challenge in evaluating "strategic partnerships" under a common framework, but also for surveying the relevant evidence base that can inform such a framework.

To this end, we systematically retrieved a large body of potentially relevant literature to feed into the development of the EVALUATE framework and mapped this via citation network analysis into prominent research topics (see figure 1). We follow this general overview with a more in-depth summary of literature on the main topics relevant for our work: (iii) internationalisation, (iv) mobility, and (v) sustainability. However, based on our literature review, we recommend that the term "strategic partnership", while possibly rhetorically useful, ought to be more carefully specified in evaluation; the specific activities of strategic partnerships and their particular strategic objectives ought to be the focus.

Figure 1: Citation network of literature on activities associated with HEI-ISPs (n=6,175; m=41,243) Nodes are coloured by cluster membership as determined via the Leiden algorithm (Q=0.524 | 11 clusters). Nodes are sized by their in-degree – the total number of citations from other nodes in this network. Lin-log ForceAtlas 2 is the layout algorithm, which positions nodes close to other nodes to which they an edge, and into clusters of nodes that share a high density of cross citation.

Developing a framework on the go: a case study approach

As strategic international partnerships come in different shapes and forms, we had doubts from the start about the feasibility to develop a simple framework, with straightforward guidelines on data and a clear assessment as result. According to the definition developed by our partners, strategic partnerships include both research and education. However, it soon became clear that some of their strategic partnerships are research only. In addition, according to their definition, strategic partnerships are between universities, while we also heard about partnerships with a variety of partners, including local authorities. Also, a strategic partnership should be university wide. Yet several of the partnerships are focused on one or two topics or fields only. And regarding use and governance: we noticed that some universities wanted to use evaluation to gain insight into the partnership and develop recommendations for improvement, whereas in other universities the evaluation was going to be used to inform decisions.

We heard a lot about the strategic importance of partnerships, little about its implementation and management, and even less about their evaluation. We realized that whatever framework we would develop, it would differ substantively from the expectations of a comprehensive evaluation tool. Instead, we felt we needed to address simple implementation and evaluation questions, and we wanted to ensure the framework could be used in different governance systems, and at any time in a partnership (before, during, towards the end). We therefore started to think of a framework consisting of a set of questions and were inspired by examples such as the Societal Readiness Thinking Tool and the Toolbox Policy Evaluation (in Dutch).

In order to develop a useful evaluation framework, we proposed to develop the framework on the go, based on case studies, inspired by the ACCELERATE project which developed a framework for impact. Each of the six partners chose an existing strategic partnership to evaluate. As a result, we collected a diverse portfolio of cases such as evaluations of a first partnership; a research collaboration focused on two topics and two universities; a research collaboration of one university with a variety of partners in a specific region; the portfolio of partners in one continent and university wide collaborations covering research, education as well as professional staff.

We offered guidance throughout the case study and suggested to use a “logbook” and keep notes of any changes in the evaluation, new insights, eye-openers, questions, etc.

We organised four meetings with each partner, planning to address different topics in every meeting. First, the history, rationale, and context of the partnership and the reason to evaluate, resulting in a clear evaluation question. Secondly, data collection, followed by the analysis of the data and the interpretation of the evidence collected, and the formulation of the assessment of the partnership. In practice the flow was iterative, and new topics arose every time. Composing the evaluation question was not easy at all and required many reformulations. The history and rationale of a partnership were addressed in subsequent meetings. The implementation of the partnership and the roles and responsibilities were brought up more than once. And when several partners wanted to reach out to researchers and students, we spent ample time talking about qualitative research methods such as interviews and surveys.

According to their own account, our partners lost confidence when they first started working on their case. They grappled with the complexity of contextual factors and the variety of perspectives of students, researchers, administrators, and the partner university. However, after four online meetings, they regained confidence when we finally met in person for the very first time (i.e. COVID). They presented and discussed their evaluations, thought along with each other, and advised on next steps. Several mentioned that it was all about asking the right questions and they asked each other the right questions indeed.

In the end, the framework we developed together is basically a series of questions, about the partnership and its evaluation. Most importantly we ask about the specific evaluation (What is the evaluation about? What is the central question?) (see figure 2). These questions are intended to guide the design of the evaluation, with a focus on the evaluation process. Once a useful and meaningful evaluation is contextualized, we also specifically asked about the context of the evaluation (Why evaluate? What is the goal?). Other questions concern the partnership and its context (What is the history? What are the targets?). They are intended to provoke thought on the history and goals of the partnership and its relationship with other policies. The focus is not on evaluation as such, but on information that can be relevant to an evaluation. When we presented the framework, our partners recognised the questions and accepted it as a useful framework, that provided both guidance and flexibility as not all questions are always relevant.

Figure 2: The central evaluation question, excerpt from the handbook

2. Conclusion and lessons learned

The EVALUATE project has been a learning journey for all. The governance philosophy of the various universities, the partnership activities, the goals of the partnerships and the relation to university strategies differ between the partners and cases. The lack of consistent and aligned evaluation practices was confirmed throughout the project. Consequently, partners realised that a rigid framework with clear measures or benchmarks is not realistic or useful. Evaluation is best integrated from the start to the end of partnerships. If integrated well, evaluation is a cyclical activity returning in every phase of the partnership, underpinning decisions, and new actions.

The project to co-create an evaluation framework was innovative in a number of ways. Firstly, the framework provides a solution for university staff working in the areas of partnership development and evaluation. This is an area of need, given recent and current prioritisation of international partnerships in both institutional, national and international strategies that concern university education and research. In addition, the co-creation with a project team consisting academic evaluation specialists and international officers was key. The combination of academic and professional inquiry enabled the project to apply robust academic methodologies and scrutiny alongside professional experience and expertise in the field. In addition, the project was complimentary to a wide range of other initiatives and can support the higher education sector to forge greater understanding of the impact of university partnerships. For example, the UK Research Excellence Framework (REF) highly values the impact of research. The impact of research often happens through partnerships – and this project provides a framework to understand the value of partnerships.

Perhaps the most interesting chapter of the handbook is Lessons Learnt from the EVALUATE project, written by the international officers. While they initially struggled with the diverse perspectives, they now acknowledge the variety and suggest taking these into account and use participative methods. They provide recommendations on such diverse topics as data collection, the use of evaluation and the implementation of strategic partnerships. They question unrealistic expectations and bold claims, such as turning students into “global citizens” or using research to address “global challenges”. And they advise thinking about evaluation as an opportunity to build capacity and involve and inspire partners. They also charachterise their own process of change in respect to evaluation, from first awakening to building awareness to maturing appreciation (see figure 3).

Figure 3: Example of lessons learned from the handbook: changing understanding of the use of evaluation in three phases.

We hoped that talking about evaluation of strategic partnerships would lead to changes in the management and implementation of those partnerships. And indeed, when we met for the second time, we noticed how our partners were changing their practice. Several had discussed expectations and intentions with their partner universities, as well as within their own university. Something they hardly did before. Moreover, they wanted to reach out to other universities outside the consortium and use the experience, and the framework, for these partnerships as well.

But we need to be realistic, about expectations of partnerships as well as the skills and time involved in evaluation. Evaluation questions can look deceptively easy, but are often hard to compose and answer. It requires multiple attempts and iterations over time to arrive at the right level of understanding. Some universities have a team with evaluation experts that support the development of institutional evaluation capacity and skills. Making time for evaluation, practising it regularly, is the best way to increase insight in the functioning of international strategic partnerships and their value.

In general, many see partnerships as well, of strategic importance. They expect great contributions, however small the partnership is. And when it comes to the newest development of university alliances, expectations are even grander. Four universities involved in EVALUATE, including the universities where we are based, are part of Una Europa. The alliance believes a university is created by and for society and as such driven to be of relevance, impact, and high quality. It is a laboratory of the here and now, where creativity and experimentation unlock the hidden potentials of tomorrow. Who is against this creed? Not many, we guess. But what does it mean in practice? What is the evidence? How can this be assessed? Let’s talk about our partnership!

4. References

Arrowood, R. J., & Hitch, L. (2016), “The Centrality Of The Faculty Role In Transnational Partnerships: A Research Agenda”. In Blessinger, P. and Cozza, B. (eds.), University Partnerships For Academic Programs And Professional Development, vol. 7, Bingley: Emerald Group Publishing Ltd, pp. 39-54.

Better Evaluation (2023): https://www.betterevaluation.org

CoARA (2022): Agreement on Reforming Research Assessment https://coara.eu/agreement/the-agreement-full-text/

Dinges, M., Meyer, S & Brodnik C. (2020). "Key Elements of Evaluation Frameworks for Transformative R&I Programmes in Europe", fteval Journal for Research and Technology Policy Evaluation, 51: 26 - 40.

Joly, PB & Matt, M. (2022). Towards a new generation of research impact assessment approaches, The Journal of Technology Transfer, 47 (3): 621-631.

Kristensen, K., and Karlsen, J. (2018), “Strategies for internationalisation at technical universities in the Nordic countries”, Tertiary Education and Management 24(1): 19-33.

Molas-Gallart, J., Boni, A., Giachi, S. & Schot, J. (2021). A formative approach to the evaluation of Transformative Innovation Policies, Research Evaluation, 30: 431-442

Otieno, I. A., and Otieno, T. (2016). “Community College University Cross-Border Partnership Through Faculty Exchange”. In Blessinger, P., and Cozza, B. (eds.), University Partnerships For Academic Programs And Professional Development, vol. 7, Bingley: Emerald Group Publishing Ltd, pp. 181-199.

Spaapen, J. (2015) “A New Evaluation Culture Is Inevitable”. Organic Farming, 1 (1): 36 – 37.

Woodfield, S. (2018), “International HE in the UK: leveraging the policy-institution-expert nexus in uncertain times”, Journal of Higher Education Policy and Management 40(6): 629-647.

5. Open science practices

This paper is based on work that is already published: The EVALUATE framework and handbook: Harnessing the power of evaluation to build better international strategic partnerships between universities. Edinburgh: The University of Edinburgh.

6. Competing interests

Authors have no competing interests.

7. Funding information

EVALUATE is funded under ERASMUS+ Key Action 2.