Project IntelComp: AI-assisted Research and Innovation Policy-Making

Project IntelComp: AI-assisted Research and Innovation Policy-Making

Jeronimo Arenas Garcia¹, Ioanna Grypari², Natalia Manola³, Paresa Markianidou⁴, Haris Papageorgiou⁵, Joseba Sanmartín Sola⁶, Lena Tsipouri⁷

¹jarenas@ing.uc3m.es; ²igrypari@athenarc.gr; ³natalia.manola@openaire.eu; ⁴paresa.markianidou@technopolis-group.com; ⁵haris@athenarc.gr; ⁶joseba.sanmartin@fecyt.es; ⁷lena.tsipouri@opix.ai

¹0000-0003-4071-7068; ²0000-0002-7485-1591; ³0000-0002-3477-3082; ⁵0000-0002-7352-2403; ⁷0000-0002-7454-6271

¹Universidad Carlos III de Madrid, Spain; ^2,5Athena Research and Innovation Center in Information Communication & Knowledge Technologies, Greece; ³OpenAIRE AMKE, Greece; ⁴Technopolis Group, Belgium; ⁶Fundación Española para la Ciencia y la Tecnología, Spain; ⁷OPIX P.C. and National and Kapodistrian University of Athens, Greece

IntelComp, a Horizon 2020 project, integrates AI-based systems and human-in-the-loop methodologies to create a comprehensive end-to-end platform for research and innovation (R&I) policy-makers and administrators. Prioritizing Open and FAIR data alongside Open, Transparent, and Reproducible methodologies, the project aims to enhance transparency, collaboration, and efficiency in R&I decision-making. This approach supports alternative research assessment methods, aligning with Open Science principles. IntelComp's technology-agnostic framework addresses diverse R&I decision-making requirements and adapts to specific thematic or national contexts. A use case of indicators for agenda setting in the domain of Energy in the EU is showcased through the STI Viewer, an interactive data and visualization platform created in the project.

1. Introduction

Research and innovation (R&I) activities are crucial to advancing digital transformation across society, boosting economic growth and development, enhancing the job market, and helping achieve the UN Sustainable Development Goals (SDGs). Both funding and policy-making for R&I are prioritized by various stakeholders, as evidenced by the global expenditure on research and innovation. The rapidly evolving nature and growing complexity of science, technology, and innovation (STI) activities necessitate an updated approach to tracking these efforts.

An essential aspect of this updated approach is the reforming of research evaluation practices in light of responsible research assessment and Open Science practices. By embracing these principles, policy makers and administrators can better align R&I activities with societal needs and values, while fostering a more transparent, collaborative, effective, and efficient research ecosystem.

IntelComp, a Horizon 2020 project, aims to assist policy makers and administrators in creating evidence-based policies by converting vast amounts of dynamic, multilingual, text-based and other heterogeneous data into actionable insights. The project begins with a set of technology-agnostic policy questions, refines them for specific domains, gathers multilingual datasets, and employs automated text analytics workflows. This process results in a set of indicators that depict various aspects of the policy cycle, from agenda setting to evaluation. These indicators are made available through the STI Viewer, an interactive, AI-assisted platform for monitoring, evaluation, and decision-making, which features business intelligence (BI) dashboards that allow for in-depth analysis across multiple dimensions, including:

• Sectors: Science, Technology, Industry, Human Resources, Society

• Geography: Europe, individual countries

• Research Areas: Domains, topics

• Actors: Research-performing organizations (RPOs), funding agencies

• Other facets: Time, etc.

2. Methodology

The monitoring and evaluation of STI activities must be driven by data that is relevant, comprehensive, and valid, encompassing key elements of big data, new data sources (including heterogeneous, unstructured, and structured data), and innovative tools for data collection, analysis, and visualization. It is essential that this process provides a 360^o view of R&I activities and their connections across various facets, while remaining automated, timely, sustainable, and transparent in order to ensure trustworthiness. A solid conceptual framework tailored to the needs of policy makers is also fundamental to achieving these goals.

Traditional surveys, case studies and focus groups cannot to provide a comprehensive view and must be combined with additional sources of data. To address these requirements, the IntelComp project employs AI-based systems, cutting-edge technologies in terms of computational power (such as High-Performance Computing) and advanced NLP techniques. Additionally, it adopts a human-in-the-loop approach, which begins with the development of a conceptual framework and policy questions, followed by refinement in specific domains, such as AI, climate, and cancer and is completed with community-led validation.

IntelComp emphasizes the use of Open (whenever possible) and FAIR Data, as well as Open, Transparent & Reproducible Methodologies. This approach allows for the discovery, linkage, and tracking of R&I activities within society, automation of processes, and provision of replicable assessments which is key for both short and long-term decision-making.

2.1 Policy Framework & Indicators

Thanks to the seminal contribution of K. Arrow (1962) public policies started focusing on the role of public support for R&I in the 1970s’. Globally accumulated evidence suggested at the end of the 20th century that sustainable productivity increases, and international competitiveness can only be achieved through investments in technology (Nelson and Winter, 1982, Fagerberg and Verspagen 2002). More recently the role of R&I for facing societal challenges (Mazzucato 2021) triggered increasing public intervention. Models and tools to monitor and assess this increasing public funding are co-evolving with R&I policies themselves. The initial linear model proved too simple to mirror reality and new models (Kline and Rosenberg, 1986) demonstrated that the path from basic research to wealth creation is too complex, sector and territory specific. The universal acceptance of the relevance of R&I policies is not accompanied by silver bullets for policies. Policy mixes must be tailored and constantly adapted to local circumstances, therefore information and data gathering facilitating thorough and timely monitoring and evaluation become an integral part of effective spending.

R&I evaluation started with theoretical models, surveys and case studies but then ICT allowed for new tools leading to more evidence-based policies. Standardizing these tools needs a solid conceptual framework to help understand what is needed, how it can be obtained and how it can be communicated. We addressed this need in two steps: We started by building a framework that can address all types of R&I evaluation needs and then applied this “agnostic” framework in specific thematic/national circumstances.

The innovation system can be decomposed into seven functions, namely (1) Entrepreneurial Activity, (2) Knowledge Creation, (3), Knowledge Diffusion through networks, (4) Guidance, (5) Market formation, (6) Resource Mobilization and (7) Creation of legitimacy/counteract resistance to change.

At the same time any (not just R&I) policy follows a 5-stages cycle, namely Agenda Setting (Definition of the problem(s) to address), Policy Formulation (Explore different courses of action), Policy Adoption (Make a choice), Policy Implementation (and Monitoring) and Evaluation. Based on the evaluation results the Policy Cycle starts again with a new Agenda Setting in a reasonable (usually 5-7 years) period.

By crossing these two dimensions we formulated the main questions of interest for policy makers:

Table 1: Conceptual Framework - Policy Questions

2.2 Indicators

A thorough data collection is the first step to respond to the questions above. Data for scientific publication, citations, patents, collaborations, industrial production and policy documents, among others are retrieved and stored. Using ontologies and NLP algorithms these data are then compiled to create indicators in the form of: Count-total; Count-fractional (in case of distributed input); Averages and Medians; Shares; Compound Annual Growth. These basic indicators can then be organized per country, per topic, per scientific field, per relevance (e.g., top publications, mostly cited patents etc.), per SDG or any combination of the above plus sophisticated ratios to respond to individual questions by policy makers.

2.3 From the theoretical technology-agnostic model to the IntelComp development of tools

The conceptual framework is useful for policy makers to formulate questions and conceive appropriate indicators for responding to their questions. IntelComp started with a long list of potential questions filling in the 28 cells of the matrix presented above. We came up with over a hundred questions and the corresponding indicators. The tech-agnostic approach was a starting point generating too many concepts, over one hundred interesting questions and even more potential indicators/measures. These technology-agnostic questions were used as a background to help policy makers select questions that fitted their needs.

Testing the framework, selecting indicators, and developing them using text analytics and machine learning can only be done in specific circumstances. Specific settings were selected for creating and testing the framework, namely:

• R&I Agenda Setting for the Energy and Agrifood sectors in the context of Climate Change in Greece and the EU

• R&I Agenda Setting for the domain of Artificial Intelligence in Spain and the EU

• Evaluation of a Horizon 2020 group of projects in Cancer

• Evaluation health-related projects and comparison across funders in France, EC, US

The development of the workflow is analysed below complemented with the first results on the Agenda Setting information for the Energy sector in the EU. IntelComp will complete its tools for the three selected domains but the workflow can be adapted to be used for any R&I domain, country and phase of the policy cycle.

3. IntelComp Workflows & The STI Viewer

The STI Viewer is an innovative, interactive platform designed to visualize and explore the wide range of indicators generated by the IntelComp project for research and innovation activities. By presenting complex data in an accessible and user-friendly manner, the STI Viewer empowers policy makers and administrators to make informed, data-driven decisions that promote growth and success in the R&I landscape.

The IntelComp project's structured approach not only supports evidence-based policy making, but also contributes to the reform of research evaluation practices by promoting alternative assessment methodologies and benefiting from open science practices. Here's how each step of the process aligns with these objectives:

1. Refining policy questions: By tailoring technology-agnostic policy questions to specific domains like climate change, AI, or cancer, IntelComp ensures that research evaluation focuses on topics relevant to society's needs and priorities, fostering a more accountable and pertinent assessment process.

2. Data collection: Collecting multilingual datasets from diverse sources, including research publications, patents, project reports, and funding data, enables a comprehensive and inclusive evaluation of R&I activities. IntelComp prioritizes the use of open datasets, as much as possible, without compromising the policy needs of the project. This approach aligns with open science practices by incorporating a wide range of research outputs and promoting transparency.

3. Automated text analytics: Utilizing advanced NLP techniques to process large volumes of heterogeneous data allows for more efficient and unbiased evaluation of research activities. This automation reduces the risk of human error and subjectivity, contributing to a more reliable and equitable assessment.

4. Indicator generation: The indicators generated by IntelComp, which cover various aspects of the policy cycle such as research output, funding distribution, collaboration networks, and societal impact, provide a holistic view of research activities. This comprehensive perspective supports a balanced assessment by considering multiple dimensions of research performance.

5. Interactive visualization platform: The AI-assisted platform and user-friendly BI dashboards enable stakeholders to easily explore, compare, and analyse the data, promoting informed decision-making and a more responsible evaluation process. Additionally, customizable views encourage transparency and openness in the evaluation process, which aligns with the principles of open science.

Some of the key features of the STI Viewer and IntelComp framework include:

• Multidimensional analysis: Users can analyse the data from different perspectives, such as by sector, geography, research area, actors, or time. This flexibility allows stakeholders to focus on the aspects most relevant to their interests and objectives.

• Comparative analysis: The platform enables users to compare indicators across countries, institutions, or research areas, which can reveal best practices, highlight areas of improvement, or identify potential collaboration opportunities.

• Trend analysis: By tracking data over time, the platform can identify emerging trends, monitor progress towards policy goals, and predict future developments in the R&I landscape.

• AI-assisted insights: The platform leverages artificial intelligence (AI) algorithms to provide users with recommendations and insights based on the data, helping them make more informed decisions and identify potential opportunities or challenges.

4. Use Case for Agenda Setting: Energy in Europe

4.1. Agenda Setting Policy Questions Driving the Indicators

Analysing and linking different data sources associated with a particular domain in a country can aid policymakers in gaining a comprehensive grasp of the field's present state and emerging patterns. Policymakers can use information obtained from scientific papers, patents, industry websites, and social requirements to recognize the most promising research topics, the organizations taking the lead in advancing such domains, and companies dedicated to R&D activities. Additionally, examining the supply of human capital skills, ESG measures, and policy documents from significant organizations may help identify the prospects and obstacles within the field. Based on this knowledge, policymakers can craft evidence-based policies that facilitate innovation, research, and sustainable development in the country's STI activities.

The Policy Questions

1. What are the emerging scientific fields and topics within the specific domain and country that relate to global societal challenges and sustainable development, and which institutions lead the development of these emerging topics?

2. What are the conventional, fashionable, and obscure topics within the domain and which organizations are leading their development?

3. Which companies engage in R&D activities within the domain, what are their primary focuses and how well are they performing in terms of both innovations and turnover?

4. What companies patent their innovations and hold trademarks within the domain, and what are their main sub-areas of focus in terms of innovative activity?

5. What are the most sought-after skills or qualifications within the domain?

6. Which higher education institutions produce graduates with these skills or qualifications?

7. What is the coverage of metrics in the ESG (Environmental, Social, and Governance) reports for the top companies in the domain, and how is their performance trending?

8. What are the key topics in policy documents from think tanks and other influential organizations within the domain?

9. Which regulations exist in the domain both domestically and from the EU regulatory framework?

Indicative Responses to the Questions

The project has already created over 30 indicators in five categories Scientific Production, Technological Production, Scientific Impact, Scientific Collaborations and Technological Impact, which can be used to respond to questions of policy makers or funding agencies and help taking decisions when setting the agenda for research at the beginning of a new policy cycle. We illustrated an example with two basic questions on the energy research agenda in the EU:

1. In which energy topics should the EU focus?

2. Identify the most attractive international collaboration.

Scientific production trends (per topic in the Energy domain in the EU) show the relative strength of topics in the EU, where the agenda can be expected to further reinforce topics where the EU is already strong and can ensure competitive advantage, for example. This can be identified by Scientific Production, Scientific Impact, Technological Production and Technological Impact per topic:

Figure 1: STI Viewer – Topics in Energy in EU – Production of Publications

Figure 2: STI Viewer – Topics in Energy in EU – Patents Production

Focusing on scientific and technological strength is insufficient, because quality/impact is more relevant than quantity, hence looking at citations per publication and patents per topic for patents of different technological value (enablers, adopters, pioneers, mavericks) complement the identification of strengths:

Figure 3: STI Viewer – Topics in Energy in EU – Publications Impact

Figure 4: STI Viewer – Topics in Energy in EU – Patents Impact

This is complemented by looking at the relative strengths of countries, since during the agenda setting choices on bilateral research agreements need to be made. In the case of energy, the best partnering could be with the core European countries, for example. Maps per Energy topic illustrate a more refined picture of where to seek collaborations:

Figure 5: STI Viewer – Energy in EU – Publications Production by Country

Figure 6: STI Viewer – Energy in EU – Publications Impact by Country

Additional questions, which can be answered with the indicators already developed help respond by country-organisation or domain to questions like: Which are the best performing organisations? Are there less experienced new entrants? Who are Pioneers in terms of Technological Value of patents?

5. Summary Discussion

The overview and use case presented above explain the conceptual approach and present an example of how IntelComp proceeded from the needs of policy makers to feeding them with policy relevant data for adopting their future agenda in the EU in the case of Energy. The IntelComp pipeline (from policy question to datasets to text analytics workflows to KPI estimation to visualizations) and tools developed can be adapted and tailored to respond questions for any phase of the policy cycle, function, R&I policy domain and geographical unit. Refined and more complex questions can be formulated and addressed using new combinations of the existing data.

Open science practices

The IntelComp project is built on open science practices, promoting transparency, accessibility, and reproducibility. The project has prioritized Open and FAIR input datasets, while datasets and software created in the project will become accessible via Zenodo, an open-access repository, after the project ends. The project advocates open software and source code, facilitating verification, collaboration, and platform improvement. When not conflicting with IPR, the code is added to GitHub for increased accessibility.

Author contributions

In this short paper, which presents the ongoing outputs of the IntelComp project, all authors have contributed equally to all aspects of the work.

Funding information

This paper is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101004870.

References

Arrow Kenneth J. (1962). The Economic Implications of Learning by Doing. The Review of Economic Studies, 29(3), 155–173. https://doi.org/10.2307/2295952

Fagerberg, J. & Verspagen, B. (2002). Technology-Gaps, Innovation-Diffusion and Transformation: An Evolutionary Interpretation. Research Policy, 31(8–9), 1291-1304. https://doi.org/10.1016/S0048-7333(02)00064-1

Kline, S.J. & N. Rosenberg (1986). An Overview of Innovation. In R. Landau & N. Rosenberg (eds.), The Positive Sum Strategy: Harnessing Technology for Economic Growth (pp. 275–305). Washington, D.C.: National Academy Press.

Mazzucato, M., (2021). Mission Economy: A Moonshot Guide to Changing Capitalism, Penguin UK

Nelson, R.R. & Winter, S.G. (1982). An Evolutionary Theory of Economic Change. Cambridge, MA: Harvard University Press.