Implementing ‘Gender in Research’ as Inclusive Excellence Indicator – Practices in peer review panels

Helene Schiffbänker

^{helene.schiffbaenker@joanneum.at}

https://orcid.org/0000-0001-6052-447X

POLICIES, Joanneum Research, Austria

1. Introduction

Gender fairness in research funding has been on the political agenda for decades. RFOs have implemented various policies to foster the equal inclusion of women in the science system and mitigate potential gender bias (Husu etal, 2022).

The request to address and incorporate GiRI is a rather new policy. The European Commission has focused on GiRI since 2003 and made the integration of the sex and/or gender dimension in knowledge production a policy target. Starting in Horizon2020 and pursued in Horizon Europe, all applicants are required to consider sex, gender and/or intersectional analysis, if relevant. In Gender Equality Plans (GEPs), GiRI is also a topic suggested to address. National funders, e.g. DFG, Swedish SRC, Irish SFI and Swiss SFN, have implemented similar policies. This way GiRI becomes an element of excellence/scientific quality that needs to be addressed by applicants, and assessed by reviewers, reflecting what is highlighted in the scoping report of the reform of the research assessment system: “Revised, or potentially new, criteria, tools and processes appropriate for assessing quality could be explored alongside peer review” (EC 2021).

The GiRI policy addresses gender bias in knowledge production and builds on research revealing that not addressing sex and gender in the research approach might lead to weak(er) research findings and innovations (Håkansson/Sand, 2021; Gibney/Schiebinger, 2020).

Research broadly demonstrated how gender analysis has enhanced scientific knowledge production (Gendered Innovations, n.d.). For example studies indicate different gendered symptoms, diagnosis and treatment of heart attacks: Women with acute coronary syndrome were less likely to experience typical chest pain, but rather atypical symptoms, such as shortness of breath, nausea, and vomiting (van Oosterhout etal, 2020) and were therefore less likely to be diagnosed and to receive guideline-recommended treatments (Khraishah etal, 2021).

Schiebinger/Schraudner (2011) have argued that “fixing knowledge by incorporating gender analysis into basic and applied research (...) assure(s) excellence and quality in research outcomes” (p. 154). Excellence as major concept for assessing proposals being interwoven with gender, however, has been critically discussed (Husu/de Cheveigné, 2010), pointing out that assessment criteria reflect the norm of the ideal, male scientist (Rees, 2011; Van den Brink/Benschop, 2012; O’Connor/Barnard, 2021; Schiffbänker etal, 2022a).

GiRI is an approach to broaden the concept of excellence by addressing gender in the production of knowledge and innovation when developing research questions, in constructing research samples or when collecting, analysing or interpreting data. RFOs define excellence/scientific quality and what research should be funded, and in this way translate political targets into policies. They “are responsible for promoting excellent research that benefits all of society (..) and make research more responsive to social needs” (Hunt etal, 2022, p. 1492). When pushing the incorporation of GiRI, RFOs address two stakeholder groups: First, they encourage applicants to consider gender in their research, and secondly, they instruct reviewers regarding the assessment of GiRI in research proposals.

The latter is the starting point for our paper: We analyse how formal GiRI policies have been applied in practice. Specifically, we investigate how this policy is discussed in peer review panels and what reviewers look at when they assess how applicants have incorporated the gender dimension in their research proposals.

On a more general level, we show how the implementation of a new policy is perceived and which challenges emerge when RFOs revise their formal assessment criteria. We identify support factors facilitating the implementation of a new policy in practice. This can serve as a baseline for discussing indicators to measure how GiRI was taken up and thus foster a broader, more inclusive understanding of scientific excellence. Furthermore, learnings for other new or revised policies in the context of responsible research assessment (RRA) can be retrieved, as fostering “diversity, equity and inclusion” is part of RRA. Analysing how the concept of excellence can be assessed in a broader perspective by focusing on gender can also provide relevant insights for developing an intersectional analysis.

We analysed the data to answer following research questions:

• How are the formal GiRI policies designed in selected RFOs and how is the formal process to assess GiRI organised?

• How do reviewers apply this policy in practice? How is GiRI linked to the assessment of scientific excellence?

• Which supporting and hindering factors for assessing GiRI can be identified?

• What can RFOs learn for the development of indicators?

• What learnings can be found from the implementation of this new excellence indicator to be applied for other RRA aims and policies?

We report preliminary findings from the ongoing H2020 research project GRANteD that aims to identify potential gender bias in the allocation of research grants. Case studies were conducted in five national RFOs, involving interviews and observations of peer review panels. GiRI was addressed when discussing the assessment criteria and various policies to mitigate gender bias.

2. Data

To study potential gender bias in the allocation of research grants, five national RFOs¹ were selected to be studied in depth. In each RFO one funding instrument was studied in more detail.

In all RFOs, semi-structured interviews were conducted with staff members to learn about the grant allocation procedures in place. In some RFOs, remote reviewers and/or panel chairs and members were also interviewed. Most interviews were conducted online, all were recorded and transcribed. Additionally, panels were observed in two RFOs. Observation data provide complementary information on panel practices. The data was collected between December 2020 and November 2022.

Table 1. Data

Data was analysed by qualitative content analysis (Mayring 2000), using MAXQDA software.

Only in three of the five RFOs, formal policies on GIRI were in place which could be analysed for implementation in practice.

3. Results

3.1 The process for assessing GiRI varies between RFOs

GiRI assessment processes differ between RFOs. While in some RFOs, the excellence of the applicant and the research proposal are primarily assessed by remote reviewers whose scoring (and additional written assessment) is the basis for panel discussions (RFO1, RFO2), in others panel members do all assessment work (RFO3). Thus, as the assessment of excellence in general, also GiRI is assessed by different actors. While in some RFOs, GiRI is primarily assessed by remote reviewers, in others it is discussed and negotiated in panels (for all findings see Schiffbänker etal, 2023).

3.2 Formal GiRI policies differ

Our analysis also shows that how the formal policies to assess GiRI are designed and applied in practice in panels and by remote reviewers differs between the three RFOs. One RFO provides remote reviewers with the instruction to “focus only on the relevance of gender in their research content and innovation” (RFO2). Remote reviewers assess GIRI, like other criteria, with a score from 1–4 and provide an additional written assessment. Another RFO provides similar instructions: “The principal investigator is required to address any relevant gender-specific and/or gender-related elements inherent in their research questions and/or research design. Please assess whether their treatment of these components is adequate“ (RFO1). The gender relevance is further specified: “Positioning and reflecting on the research approaches planned for the project in terms of sex-specific and gender-related issues, for instance: Is the research approach likely to produce sex-specific and gender-related findings? If so, what findings? How and where are these integrated into the research approach?” (RFO1). Here remote reviewers assess how adequately GiRI – as one of seven criteria – was addressed, using a 5-point scale. A third RFO has recently introduced a question in the application guidelines asking applicants whether a gender or diversity perspective is applied in the project, which is then discussed in disciplinary panels.

3.3 Various assessment practices

When interviewing remote reviewers about their practices of assessing GiRI and what they are looking at, very different ideas came up. Interestingly, GiRI was mainly linked to the numerical representation of women. Interviewees referred primarily to the team composition and stated that they are aware that the number of female researchers in teams is low, particularly in STEM. In one RFO, the focus of the GiRI policy had been broader the year before, also addressing team composition, which led to the reviewers mixing up the former (including gender in team composition) and recent policies (focusing on gender in research content more narrowly). Reviewers from countries with an affirmative action tradition (e.g. US) argued that they are used to take gender into account by pushing more women into research, similar to other underrepresented groups and ethnic minorities being included in the science system. For them, gender balance in teams was a well-known and well-practiced policy. Thus, when asked about gender in research, they referred to the representation of women in research teams. Furthermore, other informants argued that GiRI did not play any role in the grant assessment, as only very few women applied (in specific fields). Here, GiRI was linked to the representation of women applicants.

Thus, while the representation of women researchers was addressed frequently, reviewers only rarely referred to research content in a narrow sense. Furthermore, when observing panel meetings, we noticed that very limited reference was made to GiRI at all. In one disciplinarily mixed panel, GiRI was picked up twice: In the first case, the sex-balanced composition of mice samples in a Natural Science application was highlighted (this discussion even made some panel members laugh, indicating the limited awareness for GiRI), in another case, a non-binary gender concept in a Social Science application was positively highlighted, pointing to an advanced understanding of integrating gender in the research content. This illustrates how gender competence can differ between panel members and how reviewers’ disciplinary background might be relevant. Indeed, we found that the knowledge about how to operationalise GiRI was limited. In particular, reviewers in Life Sciences and STEM fields stated that they had not thought about gender in research in their discipline before, arguing, e.g.: “Typically in my area of research we don't think about that at all, so that made me think for a while” (remote reviewer, RFO3).

Reviewers in STEM fields reported that GiRI was only picked up by few applicants, while most applicants denied that gender was relevant in their proposals. The reviewers did not question this ‘non-relevance’, but argued that applicants are the experts in assessing the gender relevance in their research proposal correctly. The applicants are thus supposed to decide if, where and how to integrate the gender dimension in the research proposal. In contrast, reviewers perceive their task in either assessing the reliability of what is included in the research proposal or entering in the review form that the gender dimension was not addressed.

The potential of GiRI was evaluated in different ways: Some reviewers rated GIRI as an innovative and pioneering measure and a chance for reviewers to learn, become more gender-aware and feed this learning back to their universities. This way GiRI policy in research funding can have a direct impact also in universities. In contrast, panel members argued that reviewers do not like to be told by RFOs how to assess scientific excellence/merit. Moreover, it was remarked that GiRI could potentially restrict academic freedom. A panel member (who claimed to actively support gender equality) argued that this policy might reflect ideological positions: “I think what makes them problematic is that they express some kind of political interference with academia and research which I think is both a little bit un-transparent and also potentially problematic to have such kind of thresholds or ideologically tinted dimensions” (panel member, RFO3).

3.4 Intersectionality

As gender in knowledge production intersects with other social categories (ethnicity, age), GiRI needs to be broadened accordingly. However, no further dimensions were linked to GiRI when panel members or remote reviewers discussed and assessed this policy. More inclusiveness requires a broader set of data, e.g. on ethical background (Boytchev, 2023), and more indicators to be able to adopt policies accordingly (Woods etal, 2021).

4. Discussion of learnings

When analysing assessment practices in RFOs, some factors supporting or hindering the implementation of the GiRI policy in practice were identified, which can be beneficial for RFOs and other stakeholders aiming for RRA.

4.1 Clear definition of the GiRI policy

We found that reviewers have various ideas regarding GiRI policy. They had difficulties in distinguishing between gender in research content and gender representation in research teams.

This points to a lack of clarity among reviewers on what is covered by GiRI policy, corresponding to diverging formal implementation concepts and political aims. The definition and focus of GiRI as well as the way it should be assessed varies among RFOs. As many reviewers are active in several RFOs, it might be difficult to apply the appropriate assessment practice in each RFO. It might be easier if GiRI would be defined, measured and assessed similarly across RFOs. Further, we found that RFOs have adjusted their policy from one year to the next. A clear, uniform definition of the policy would help to develop a common understanding and align the assessment of this concept. RFOs need to make clear what applicants are expected to report and what reviewers are expected to assess to avoid individual standards (Schiffbänker etal, 2022b).

4.2 Communicating the rationale and benefits of GiRI

The assessment of gender in the content of research projects and innovation was not always comprehensible. Some interviewees were not familiar with the policy and its application in research proposal assessment. Others had a lack of understanding regarding the purpose and political aim of the policy, e.g. how GiRI is linked to the concept of scientific excellence and why research quality benefits from considering a sex or gender dimension. Due to this lack of understanding, reviewers might not apply the GiRI policy (appropriately).

Explaining the underlying arguments, demonstrating current shortcomings to be overcome or potential bias factors helps developing an understanding of the rationale behind the GiRI concept. Presenting good practice examples from various scientific fields might help reviewers to develop awareness and creativity on how to assess GiRI optimally.

4.3 Building capacities: Gender awareness and competence

Our data illustrates that the GiRI policy is difficult to understand without gender awareness or knowledge. Gender awareness means that reviewers (at least to some extent) are aware of the relevance that gender plays as a structural dimension in the science system, but particularly in their research fields. Here, it might be relevant that in multidisciplinary panels different levels of gender knowledge can be found, while in some disciplinary panels (STEM fields), gender relevance and awareness are more difficult to identify. Gaining more gender awareness, however, enables reviewers to assess whether gender was addressed adequately in grant proposals in a more informed way. When assessing the scientific excellence of a proposal, reviewers are asked to assess the applicants’ adequate consideration of GiRI. This way reviewers are (only) expected to focus on adequacy. When an applicant argues that the proposed research topic does not have gender relevance, the reviewer assesses the adequacy of this statement and does not engage in discussions about GIRI directly. Nevertheless, to be able to decide if a research proposal addresses gender adequately, at least some gender competence is needed.

Given the limited time resources of reviewers, efficient formats for developing this competence need to be identified, such as preparatory sessions prior to panel meetings or self-checks for remote reviewers. In medical research, online training with evidence-based capacity-building methods have been developed. Pre- and post-training evaluation shows that most participants increased their knowledge on how to integrate sex and gender in grant proposals (Tannenbaum/van Hoof 2018).

4.4 Developing indicators

Indicators are needed to monitor and evaluate the quality and quantity of GiRI assessment. To avoid confusion among reviewers, RFOs should agree on common indicators covering different layers: reviewer training, quality of remote reviewers’ assessment, extent to and manner in which panels address and assess GiRI, quality of GiRI in applications. Clear accountability for a continuous implementation should be defined.

4.5 Addressing GiRI along the funding cycle

Finally, we want to stress that RFOs are able and encouraged to address and support the implementation of GiRI along the whole funding cycle. Our findings on practices of panels and reviewers feed into recent research on this (Fritch, etal 2022; Hunt etal, 2022).

Table 2. GiRI in the funding cycle (practice level)

Source: own table, based on Fritch etal, 2022

4.6 Learnings for responsible research assessment

Analysing how GiRI has been implemented in practice shows that when a new policy is formally implemented, reviewers should be familiar with the underlying rationale and capable to understand its aim. Attention should be paid to remote reviewers who are less connected and have no access to collective learning. RFOs could provide support as implementing a formal policy does not mean that all reviewers are aware of its meaning and significance.

When implementing or revising a new policy, indicators should be defined, and applicants as well as reviewers should know what is expected by the RFOs.

5. Bibliographic references

Open science practices

Data is not openly available as findings are still preliminary, as agreed in the Memorandum of Understanding with the RFOs.

Acknowledgments

HS acknowledges fellow colleagues for providing feedback and Liisa Husu, Helen Peterson and Angelika Sauer for also collecting and analysing data.

Competing interests

No competing interests.

Funding information

This paper has been produced within the H2020 project GRANteD – GRant AllocatioN Disparities from a gender perspective, funded by the European Union under grant agreement no. 824574.

References

Boytchev H. (2023). Diversity in German science: researchers push for missing ethnicity data. Nature, 616(7955), 22–24. https://doi.org/10.1038/d41586-023-00955-9

European Commission, Directorate-General for Research and Innovation (2021). Towards a reform of the research assessment system: scoping report, Publications Office, 2021.

Fritch R., Ivarsson, S., Persdotter, M. & Foley, C. (2022). Advancing the sex and gender dimension in the research and innovation funding process, Report from the FORGEN CoP workshop, https://www.genderportal.eu/resources/advancing-sex-and-gender-dimension-research-and-innovation-funding-process

Gendered Innovations (n.d.) https://genderedinnovations.stanford.edu/.

Gibney, E. & Schiebinger, L. (2020). How sex and gender analysis improves science. Nature, 588(7837), 209-209.

Håkansson, S. Y. & Sand, J. (2021). The Gender Dimension in Research and Innovation: Results from a global survey on research funding organisations. Gothenburg: Swedish Secretariat for Gender Research, University of Gothenburg.

Husu, L., Peterson, H., with Schiffbänker, H., Hearn, J., Sauer, A., Hock, M. & Walker, D. (2022). Synthesis Report on Contextual Factors, Gender Equality Policy Analysis and Gender Bias Risk Analysis. D5.1. of GRANteD project.

Husu, L., & de Cheveigné, S. (2010). Gender and gatekeeping of excellence in research funding: European perspectives. In Riegraf, B., Aulenbacher, B., Kirsch-Auwärter, E., & Müller, U. (Eds.) GenderChange in Academia (pp. 43-59).

Hunt, L., Nielsen, M., W. & Schiebinger, L. (2022). A Framework for Sex, Gender, and Diversity Analysis in Research, Science, Vol 377, Issue 6614, (pp. 1492-1495), DOI: 10.1126/science.abp9775.

Khraishah, H., Alahmad, B., Alfaddagh, A., Jeong, S. Y., Mathenge, N., Kassab, M. B., Kolte, D., Michos, E. D. & Albaghdadi, M. (2021). Sex disparities in the presentation, management and outcomes of patients with acute coronary syndrome: Insights from the ACS QUIK trial. Open Heart, 8(1), e001470. https://doi.org/10.1136/openhrt-2020-001470

Mayring, P. (2000). Qualitative content analysis. Forum: Qualitative Social Research, 1(2). Retrieved July 28, 2008, from http://217.160.35.246/fqs-texte/2-00/2-00mayring-e.pdf.

O’Connor, P. & Barnard, S. (2021). Problematising excellence as a legitimating discourse. In O’Connor, P. & White, K. (Eds.) Gender, Power and Higher Education in a Globalised World (pp. 47-69). Cham: Palgrave Macmillan.

Rees, T. (2011). The Gendered Construction of Scientific Excellence. Interdisciplinary Science Reviews, 36. https://doi.org/10.1179/030801811X13013181961437

Schiebinger, L. & Schraudner, M. (2011). Interdisciplinary approaches to achieving gendered innovations in science, medicine, and engineering. Interdisciplinary Science Reviews, 36(2), (pp. 154 – 167).

Schiffbänker, H., Sauer, A., Husu, L. & Peterson, H. (2023). Synthesis Report on gender bias in each core-RFO, GRANteD Deliverable D6.1

Schiffbänker, H., van den Besselaar, P., Holzinger, F., Mom, C. & Vinkenburg, C. (2022a). Gender bias in peer review panels – “The Elephant in the Room”. In Jenkins, F., Hoenig, B., Weber, S., Wolffram, A. (Eds.) Inequalities and the Paradigm of Excellence in Academia, London: Routledge.

Schiffbaenker, H., Haas, M. & Holzinger, F. (2022b). The gendered nature of independence in the context of research funding and excellence. SN Soc Sci 2, 275 (2022b).

Tannenbaum, C. & van Hoof, K. (2018). Effectiveness of online learning on health researcher capacity to appropriately integrate sex, gender, or both in grant proposals. Biol Sex Differ 9, 39 https://doi.org/10.1186/s13293-018-0197-3

Van den Brink, M. & Benschop, Y. (2012). Gender practices in the construction of academic excellence: Sheep with five legs. Organisation, 19(4), (pp. 507-524).

van Oosterhout, R. E. M., de Boer, A. R., Maas, A. H. E. M., Rutten, F. H., Bots, M. L. & Peters, S. A. E. (2020). Sex Differences in Symptom Presentation in Acute Coronary Syndromes: A Systematic Review and Meta‐analysis. Journal of the American Heart Association, 9(9), e014733. https://doi.org/10.1161/JAHA.119.014733

Woods, D., Benschop, Y. & van den Brink, M. (2021). What is intersectional equality? A definition and goal of equality for organizations. Gender, Work & Organization, 29(1) (pp. 92-109).