Evolution of Scientific Evaluation Policy in China Over 1978-2021: A Bibliometric Analysis of Policy Documents

Houqiang Yu ^*, Yian Liang^** and Yue Wang ^**

^*yuhq8@mail.sysu.edu.cn

0000-0002-9241-6630

School of Information Management, Sun Yat-sen University, China

^** liangyian3@mail2.sysu.edu.cn; 595905250@qq.com

0000-0002-1234-1131; 0000-0001-6621-8780

School of Information Management, Sun Yat-sen University, China

Scientific evaluation in China is going through a transition period. In dealing with the misuse and alienation of quantitative indicators, China had initiated a top-down reform. To help better understand the scientific evaluation policy in China and provide reference for further reform, this study investigated the evolution and characteristics of scientific evaluation in China from a bibliometric perspective. In total, 343 policy documents from 1978 to 2021 were collected. Based on the temporal distribution of policy documents and timeline of landmark policy documents, four phases were identified. Text mining technique and social network analysis were used to identify the shift of policy focus over different phases. The four phases were characterized from three aspects: (1) The time span and number of relevant policy documents; (2) The focus of policy document content; and (3) The collaborative network of ministries that release the policy documents.

1. Introduction

In the era of big science, scientific resources are commonly allocated based on the result of scientific evaluation (Li, Mao, Zhang, Wang, Shen, & Huang, 2022). As an essential way for government to do scientific administration, scientific evaluation is an important tool for improving the governance capability (Wu, Wang, He, Huang, & Dong, 2021). China has played an increasingly important role in the global science community. Ever since its national science and technology conference convened in 1978, China has been dedicated to develop scientific evaluation system and achieved great progress. However, excessive reliance on quantitative evaluation has caused series of problems for scholarly communication (Quan, Chen, Shu, Lewandowski, & Paul-Hus, 2017) and brought negative effect on scientific innovation, and consequently been harmful for scientific development.

To advance the national innovation-based development strategy, it is urgent to deepen the reform of scientific evaluation system. In 2018, the Chinese central government released Opinions on Further Reform of Project Evaluation, Talents Evaluation, and Institution Evaluation (Short as Opinions), and pointed out that these three types of evaluation were the basis for improving scientific evaluation system. Following this document, the other ministries took a series of measures to improve the scientific research management and initiated the special action of Breaking the Four Sole Criteria (i.e., publication oriented, professional title oriented, education background oriented and academic awards oriented). These actions aroused broad attention and active responses from the entire society.

At this critical moment, it is enlightening to explore the evolution process of scientific evaluation system in China over the last 40 years. This study is aimed to answer the following question. What is the evolution process of scientific evaluation system in China from perspective of policy documents? Based on policy documents of scientific evaluation from 1978 to 2021, the evolution of scientific evaluation system in China will be divided into different phases. For each phase, the characteristics are analysed, and the collaboration between different ministries that release relevant policies are revealed.

2. Methodology

2.1. Data Collection

National policy documents about scientific evaluation were retrieved via two channels. Policy documents prior to 2011 were retrieved from tool books, including Overview of Science and Technology Policy in China 1949-2010, Selected Scientific Laws, Regulations and Policies 1985-2008, Selected Scientific Laws, Regulations and Policies 2009-2011. In total, 157 pieces of policy documents about scientific evaluation were collected. Policy documents after 2011 were retrieved from the policy document database of state council (http://www.gov.cn/zhengce/zhengcewenjianku/index.htm). The database has collected policy documents by various government ministries since January 2008. The database does not support full-text retrieval by multiple keywords. Therefore, relevant keywords were used to retrieve one by one. The used keywords were scientific evaluation/assessment (科技评价), evaluation/assessment of projects (项目评审), talents evaluation/assessment (人才评价), institutional evaluation/assessment (机构评价), breaking the four sole criteria (破四唯), publication oriented (唯论文), awards oriented (唯奖励), talents title oriented (唯帽子), education degree oriented (唯学历). It must be noticed that Breaking the Four Sole Criteria (i.e., publication oriented, awards oriented, talents title oriented and education degree oriented) was the landmark of reform in scientific evaluation policy since 2019 in China. The initial retrieval result consisted of 1,659 policy documents by ministries of state council and 1,170 policy documents by the state council. After careful scrutinization and manual filtering, 193 policy documents were identified to be highly relevant to scientific evaluation. After removing duplicates with the dataset of policy documents prior to 2011, 343 pieces of policy documents were harvested which spanned from October 1978 to December 2021.

2.2. Data analysis

In the first step, policy documents were classified into different phases. The dataset spanned 44 years. Based on the temporal trends of number of policy documents and key events of releasing the landmark policy documents, different phases were determined.

In the second step, keywords were extracted from the policy documents. Unlike scholarly papers, policy documents in China provide no keywords. To do further text mining, a framework was proposed for extracting relevant keywords from the policy documents: (1) The full text of policy documents was segmented into sentences. (2) According to the verb/noun synonym dictionary of common policy target sentence as illustrated in Table 1, sentences that include policy target were identified and filtered out. (3) Policy target sentences were processed including word segmentation and removal of stop words. (4) Using TF-IDF algorithm, the top 10 terms in the topic sentence set of each policy document were extracted as keywords. The results were manually checked.

Table 1 Examples of verb/noun in common target sentences

In the third step, keywords co-occurrence network in each phase was visualized using VOSViewer. Major topics were identified by clustering analysis. The focus of each topic was interpreted with the content of relevant policy documents.

In the fourth step, collaborative network of ministries that release policy document was constructed. For ministries with different names across historical stages, they were merged and presented using the contemporary name in the collaborative network. Pajek was used to do social network analysis.

3. Results

3.1. Temporal distribution of scientific evaluation policy documents in China

Figure 1 illustrated the temporal distribution of scientific evaluation policy documents in China from March 18^th, 1978 to December 31^st, 2021. The number of policy documents is fluctuating over the years. Combining the trend of number of policy documents and timeline of landmark policy documents, the evolution of scientific evaluation policy documents was classified into four phases. The details are shown in Table 1.

Figure 1 Temporal distribution of number of policy documents by the central government

Table 1 Different phases of policy documents of scientific evaluation in China from 1978 to 2021

Note: T.S. is time span, N.Y. is number of years, N.P.D. is number of policy documents

3.2. Evolution of scientific evaluation policy documents across different phases

3.2.1. The burgeoning phase (March 1978 to February 1985)

The burgeoning phase focused on how to boost the construction of socialism modernization via science and technology management. It was aimed to produce scientific outputs, cultivate talents and build the preliminary system for science and technology evaluation. As shown in Figure 2, three major topics were identified, namely the identification and awards for scientific achievements, science and technology institution management and evaluation of science and technology researchers.

Scientific evaluation system in this phase is centered on qualitative evaluation of scientific achievements and researchers. The process of evaluation is dominant by superior’s assessment and peer review. In this phase, the definition of science and technology evaluation was not manifested. The standards of evaluating various objects were not mature. Moreover, the whole country was in planned economy system, several problems were prominent in the scientific system, for example, the ossification in the management of science and technology.

Figure 2 Keywords co-occurrence map in the burgeoning phase (1978-1985)

3.2.2. The exploration phase (March 1985 to April 2003)

In this phase, Interim Procedures on Management of Scientific Evaluation, released by ministry of science and technology, manifested the definition of scientific evaluation for the first time. As shown in Figure 3, three major topics were identified, namely management of science funding, reform of scientific research institution and cultivation of scientific talents.

Influenced by the reform of science and technology system, this phase was featured with more diversified types of evaluation objects and methods. The focus was on evaluation of scientific plans and programs. The evaluation process began to adopt various quantitative indicators. The evaluation methods include superior’s evaluation, peer review and institutional evaluation and so on. In this phase, China was transforming from planned economy to market economy and prioritize economic development (Liu, Simon, Sun, & Cong, 2011). The economic benefits of projects, programs and achievements were key points in the scientific evaluation. However, problems also existed. The classification of scientific evaluation was unclear. The framework for peer review and reviewers’ reputation was immature. There lacked supervision and management of in-process and post evaluation.

Figure 3 Keywords co-occurrence map in the exploration phase (1985-2003)

3.2.3. The development phase (May 2003 to June 2018)

In the policy document Measure for Science and Technology Evaluation released by ministry of science and technology, the definition of scientific evaluation was further enriched. In this phase, number of relevant policy documents increased. These policy documents continued and extended the focus of the last phase, and simultaneously took reform measures on science and technology evaluation methods. As shown in Figure 4, three major topics were identified, namely management of scientific research institutions and plans, evaluation of science and technology innovation and achievements, cultivation and evaluation of scientific talents.

This phase was featured by the emphasis on the orientation to market needs and special attention were paid to scientific innovation. The adopted methods were mainly peer review and institutional evaluation. The standards of evaluation were regulated. The importance of classified evaluation was highlighted and for various types of objects the evaluation measures were improved. In this phase, China had undertaken socialist market economy. In order to optimize the allocation of scientific resources, the government conducted further macro-level guidance and supervision over the scientific evaluation practice and encouraged the development of intermediary organizations for scientific evaluation. However, with the increasingly used quantitative indicators, the evaluation system had several prominent problems, for example, utilitarianism in scientific activities and misuse of indicators.

Figure 4 Keywords co-occurrence map in the development phase (2003-2018)

3.2.4. The improvement phase (July 2018 to December 2021)

Despite that this phase had the shortest time span, the number of relevant policy documents was the highest, indicating the urgency and importance of further reform of scientific evaluation system. As shown in Figure 5, four major topics were identified, namely scientific research management, reform of professional title system, cultivation of skilled talents and university development.

This phase was featured with hitherto the most active and broad discussion about scientific evaluation system in China. Starting from the three types of evaluation, the evaluation principles put special emphasis on being problem orientated, using classified evaluation, and valuing moral evaluation. The misuse of quantitative indicators was extensively discussed and clarified. It was required to instead evaluate the quality, contribution, and value of representative works. The function of scientific evaluation in talents cultivation was emphasized. Via advocation and education, it was intended to improve the environment of scientific evaluation and establish the correct orientation of evaluation. Compared with previous phases, the key points of scientific evaluation system were more outstanding, the standards of evaluation were further improved and the reform measures were more thorough.

Figure 5 Keywords co-occurrence map in the improvement phase (2018-2021)

3.3. Evolution of collaborative network between ministries that release relevant policies

Figure 6 shows the collaborative network of ministries that release policy documents about scientific evaluation. According to the size and location of nodes, it can be identified the leading ministries and assistant ministries. The Figure uses abbreviations to present a clearer visualization. Table 2 shows the attributes of collaborative network in each phase. Network density reflects the closeness between nodes, and network centrality reflects the extent to which the network is concentrated to certain nodes. As seen from Table 2, collaboration grows over time, while the network centrality went through a changing process of from concentrated to scattered and back to concentrated.

Table 2 Attributes of collaborative network among different ministries that release policy documents of scientific evaluation

Figure 6 Evolution of collaborative network of ministries that release polices about scientific evaluation across different phases

As shown in Figure 6, in the foundation phase, the collaborative network is relatively sparse and percentage of collaborative policy documents was low. The state council and ministry of science and technology are the major ministries that release relevant policies. Each department was inclined to independently perform its own function of scientific evaluation. In the exploration phase, the connection between ministries became closer. The state council and ministry of science and technology were leading ministries, while the former tended to release policy document alone, the latter tended to collaborate with other ministries. In the development phase, the network became dense. Ministry of science and technology, the state council and ministry of finance are the leading ministries, while ministry of education, ministry of human resource and social security, national development and reform council are mainly the assistant ministries.

In the improvement phase, although percentage of collaboratively released policy documents was the highest, there were many involved ministries and consequently the network density declined. The high network centrality indicated the concentration of rights while different centers of rights emerged. The group with ministry of human resource and social security as the center is responsible for reform of professional title evaluation. The group with ministry of education as the center is responsible for amending the regulations for university development and scientific evaluation. Ministry of science and technology and ministry of finance are responsible for managing research funding.

4. Discussions

Over the nearly four decades, scientific evaluation system in China was featured by a top-down administration style. The central government and ministries release relevant polices, and local government and institutions are required to implement the documents. The best practices and accumulated experiences from specific institutions will then feedback the administration. While the current policies strongly require institutions to measure the quality, originality, and value of research, it is not a new idea. It is in fact always the desired goal of scientific evaluation. The difference lies in how it is implemented and trade-off has been made in difference periods.

The burgeoning phase (October 1978 to February 1985) was characterized by qualitative evaluation where superior’s evaluation played an important role. Peer review was also a major adopted method except that the procedure was immature and the process was inefficient. A few national ministries were involved and they tended to make policies alone. The exploration phase (March 1985 to April 2003) saw the rise of a wide range of quantitative indicators in scientific evaluation. China had turned from planned economy to socialism market economy. The economic benefits of projects, programs and achievements were key considerations. Quantitative indicators had significantly improved the efficiency, reliability, and fairness of scientific evaluation. China was also able to benchmark with the international performance. The development phase (May 2003 to June 2018) witnessed the ubiquitous use of quantitative indicators (Shu, Quan, Chen, Qiu, Sugimoto, & Vincent, 2020) and enriched scientific evaluation system. Consensus was reached about classified evaluation. Evaluation by third party institutions was encouraged. Criticism about the misuse of quantitative indicators and the alienation of evaluation direction became increasingly sharp. The improvement phase (July 2018 to December 2021) was dedicated to restore balance between qualitative and quantitative evaluation. Polices explicitly required the evaluation to be problem oriented, conduct classified evaluation and value moral assessment. The classified evaluation towards different type of researchers was improved and mandated, in order to guarantee that researchers can be devoted to truly important scientific affairs.

Over the years China was active in adopting the new methods and principles of scientific evaluation from the international community, and simultaneously contributed to the theories and practices of scientific evaluation. Based on the evolution process of scientific evaluation policies, the latest reform can draw conclusion that quantitative indicators would always be useful supplements for peer review. Instead of abandoning these indicators, emphasis should be paid on how to improve the metrics literacy of administrators. Many novel specific practices have come to the force in the improvement phase and are likely to be useful beyond China.

Open science practices

Dataset are archived at the author’s institution and can be available upon request to the authors.

Author contributions

Houqiang Yu: Conceptualization; Funding acquisition; Writing – review & editing

Yian Liang: Data curation; Formal Analysis; Writing – review & editing

Yue Wang: Formal Analysis; Visualization; Writing – original draft

Competing interests

The authors have no competing interests.

Funding information

This research is funded by Humanity and Social Science Foundation of Ministry of Education of China (22YJA870016), National Natural Science Foundation of China (72274227), and the Fundamental Research Funds for the Central Universities.

References

Li, Y., Mao, J., Zhang, L., Wang, D., Shen, S., & Huang, Y. (2022). How scientific research incorporates policy: an examination using the case of China’s science and technology evaluation system. Scientometrics, 127(9), 5283-5306.

Liu, F. C., Simon, D. F., Sun, Y. T., & Cong, C. (2011). China's innovation policies: evolution, institutional structure, and trajectory. Research Policy, 40(7), 917-931.

Quan, W., Chen, B., Shu, F., Lewandowski, D., & Paul-Hus, A. (2017). Publish or impoverish: an investigation of the monetary reward system of science in China (1999-2016). Aslib Journal of Information Management, 69(5), 486-502.

Shu, F., Quan, W., Chen, B., Qiu, J., Sugimoto, C. R., & Vincent L. (2020). The role of web of science publications in China’s tenure system. Scientometrics, 122.

Wu, J., Wang, K., He, C., Huang X., & Dong K. (2021). Characterizing the patterns of China’s policies against COVID-19: A bibliometric study. Information Processing & Management, 58(4), 102562.