The Potential of Meta-Analysis for Evidence-based Innovation Policy

: Three Research Cases in the South Korean Context

Juil Kim^*

^*juil@kistep.re.kr

0000-0001-7566-0503

Center for Growth Engine R&D Coordination, Korea Institute of S&T Evaluation and Planning (KISTEP), South Korea

The effectiveness of policy measures in innovation has been a subject of debate due to the heterogeneity of research results. To address this issue, this paper discusses the potential of meta-analysis as a methodology for evidence-based innovation policy by examining three previous Korean studies. The studies compare the effect size of direct and indirect government R&D support for SMEs, identify the key factors that promote technological innovation in SMEs, and determine various factors affecting the adoption of smart factories. The author introduces the results of three research cases and suggests that meta-analysis can be a useful tool for exploring higher-level evidence to establish innovation policies in Korea and beyond.

1. Introduction

R&D activities and scientific progress are widely recognized as drivers of economic growth (Romer, 1987; Aghion & Howitt, 1992; Grossman & Helpman, 1991). In line with this understanding, the Korean government invests approximately 30 trillion won annually in R&D. However, questions persist about the efficacy of these investments. To address these concerns, an evidence-based policy approach is crucial. Evidence-based policy is a rational problem-solving method that relies on accurate diagnosis and causal connections (Head, 2008). Evidence-based policies can justify policy decision-making, help select among various policy measures quickly, and enable rational policy-making that meets the expectations of the people (Davies, 2004). Therefore, an evidence-based approach is essential for promoting innovation policies and meeting the demands of our times.

Although many empirical studies have attempted to promote evidence-based policies in the area of innovation policy, there is a lack of studies that systematically synthesize these results. To address this limitation, this paper presents meta-analysis as a methodological alternative. Meta-analysis is a method that systematically integrates the results reported by existing empirical analysis studies and provides comprehensive results (Glass, 1976). By using meta-analysis, it is possible to quantitatively compare the effect sizes identified by various empirical studies and discover high-priority strategies. Given the heterogeneity of research results regarding the effects of various policy measures in the area of innovation policy, a meta-analysis that integrates and reviews these research results is particularly valuable.

In this paper, I present three studies that apply meta-analysis to the context of innovation policy in Korea. These studies provide meaningful insights regarding the comparison of policy measures, identification of technological innovation processes, and factors of manufacturing innovation.

2. Literature review and research issues

2.1. Evidence-based Policy

The term “evidence-based” was first used in medicine and healthcare to refer to the conscientious, clear, and deliberate use of the best available evidence in making care decisions for patients (Sackett et al., 1996). This concept was later adopted in the field of policy, following the explicit advocacy of evidence-based policy by the British government in the White Paper “Modernising Government” in 1999. Evidence-based policies emerged as a response to the need to advance innovation policies, given the necessity of considering economic and social priorities in research support policy and turning it into a practical direction that could make research results useful (Solesbury, 2001).

Evidence-based policy involves making policy decisions based on sufficient information by injecting the best evidence derived from survey research into policy development and implementation (Davies, 2004). This approach contrasts with opinion-based policy, which is based on subjectivity, ideology, prejudice, and speculation (Gray, 1997). Since innovation policy deals with the creation and diffusion of intangible knowledge, it can be difficult to measure and evaluate the process. An evidence-based approach can help address this difficulty by providing a systematic way to identify and analyze relevant data and information, thus informing policy decisions based on empirical evidence.

2.2. Meta-analysis

There have been several attempts to synthesize previous studies. Among them, the method of qualitative synthesis may involve too much subjectivity on the part of the researcher. Systematic literature reviews can fall prey to the majority fallacy. Meta-analysis was designed to overcome these limitations. Meta-analysis is a statistical methodology that systematically synthesizes the figures reported by empirical analysis studies and was first proposed by leading statisticians in the 1930s (Fisher, 1932; Pearson, 1934). The term “meta-analysis” was established by Glass (1976).

Meta-analysis is a statistical technique that analyzes a collection of analysis results reported by a number of individual studies for the purpose of integrating the research results. By using meta-analysis, it is possible to reach a more objective conclusion while overcoming the limitations of qualitative and subjective literature research, which is inevitably influenced by the bias of a specific researcher (Stanley, 2001).

Meanwhile, the meta-analysis methodology has been developed mainly by clinicians and statisticians and has been widely applied in the medical science field (Ioannidis, 2016). However, in principle, the meta-analysis methodology is not limited to randomized controlled trials (RCTs) that integrate several experiments and can provide superior evidence than individual approaches (Ahn & Kang, 2018; Uetani et al., 2009). Therefore, it is necessary to explore the possibility of meta-analysis in the field of innovation research and innovation policy.

Efforts to incorporate meta-analysis in technological innovation research began in earnest after Damanpour’s (1991) research in the 1990s. As a sub-topic, the influencing factors of innovation (e.g.: Büschgens, Bausch & Balkin, 2013; Mendoza, 2015), the relationship between innovation and performance (e.g.: Vincent, Bharadwaj & Challagalla, 2004; Li, Li & Gao, 2019), the effectiveness of government R&D support policies (e.g.: David, Hall & Toole, 2000; Donselaar & Koopmans, 2016), and other topics have been studied using meta-analysis. However, despite these attempts, efforts to improve the level of evidence-based innovation policy through meta-analysis have not yet gained sufficient trust in both the academic and policy areas.

2.3. Research issues

In this paper, three meta-analysis studies that provide implications for innovation policy are introduced. These studies were conducted in Korea and aim to advance evidence-based innovation policies. The three studies address the following research questions:

1. What is the comparative effectiveness of direct research funding versus indirect support through tax spending for government support of technological innovation in SMEs? Both direct and indirect support are useful policy tools, but it is necessary to quantitatively compare their effect sizes to determine the optimal policy mix.

2. What are the most significant internal and external factors that impact the technological innovation of SMEs? To answer this question, it is necessary to identify the specific factors that have the greatest influence on technological innovation and that should be prioritized by government innovation policies.

3. What are the factors that influence manufacturing companies to adopt smart factories? Smart factories can improve productivity in the manufacturing industry, and policies that encourage their adoption are important in countries such as South Korea. Through meta-analysis, it is possible to identify the factors that affect the adoption of smart factories and adjust innovation policies accordingly.

Examples of meta-analysis studies that address these research questions are presented, and their implications for promoting evidence-based innovation policies are discussed.

3. Method

Meta-analysis is a statistical method that collects and analyzes the effect size reported by individual studies. The meta-analysis study introduced in this paper collects the effect size of the correlation coefficient (r) and calculates the average value. Since the correlation coefficient (r) shows an asymmetric distribution, it is converted to Fisher’s Z to obtain the average effect size. However, when reporting this value, it is converted back to the correlation coefficient (r) (Shadish & Haddock, 1994). The formula for calculating Fisher’s Z and the formula for converting the mean effect size calculated with Fisher’s Z back to r are as follows (Borenstein et al., 2009):

\(Z = \ .5\ \times \ \ln\left( \frac{1 + r}{1 - r} \right)\) \(r = \ \frac{e^{2z} - 1}{e^{2z} + 1}\)

Studies that did not report the correlation coefficient (r) followed the method of integrating after converting the t-value or regression coefficient β to r. In this case, the formula is as follows (Wolf, 1986; Peterson & Brown, 2005):

\(r = \sqrt{\frac{t^{2}}{t^{2} + df}}\ \ (df:degree\ of\ freedom)\) \(r = \beta + .05\lambda\ \ \ \ (\lambda:\ 1\ if\ \beta \geq 0,\ 0\ if\ \beta < 0)\)

The variance and standard deviation of Fisher’s Z are calculated using the formula below:

\(V_{z} = \ \frac{1}{n - 3}\) \({SE}_{z} = \ \sqrt{V_{z}\ }\)

Studies subject to meta-analysis are analyzed with different numbers of samples. When calculating the mean effect size, different weights for each individual effect size are applied under the premise that the effect size calculated through more samples is closer to the overall average. The inverse variance weight (\(W_{i}\)) and weighted average (M) formulas applied at this time follow the following (Hedges & Olkin, 1985):

\(W_{i} = \frac{1}{V_{Y_{i}}}\) \(M = \ \frac{\sum_{i = 1}^{k}{W_{i}Y_{i}}}{\sum_{i = 1}^{k}W_{i}}\ \ \ \ \ \ \ \left( Y_{i}:effect\ size \right)\)

In accordance with Cohen (1988), the calculated effect size was interpreted as a small effect size when it was 0.1 or lower, a moderate effect size if it was around 0.3, and a large effect size if it was 0.5 or more.

4. Three Cases of Meta-analysis

4.1. Direct Support vs. Indirect Support

The South Korean government has implemented various policy instruments to promote R&D investment by SMEs, including direct research funding and indirect tax support. In 2021, the government provided 5.0 trillion won in direct support and 1.3 trillion won in indirect support (Ministry of Science and ICT & KISTEP, 2022). However, there is still debate about the effectiveness of these policies and how to strike the right balance between direct and indirect support.

Kim (2019) conducted a meta-analysis of 24 relevant studies published in domestic journals to examine the effectiveness of these policies. The meta-analysis revealed that, overall, indirect support (.192) had a higher crowding-in effect than direct support (.143). For large enterprises, the effect size of indirect support (.250) was more pronounced than that of direct support (.080). However, for SMEs, direct support (.124) was found to be more effective than indirect support (.098) in inducing R&D investments.

The study found that Korean SMEs receive a disproportionately high proportion of direct support for R&D, with a current ratio of 79.4 : 20.6 for direct and indirect support, respectively. This ratio is significantly higher than the 55.9 : 44.1 ratio of effect sizes resulting from the meta-analysis, suggesting that the actual financial investment in direct support is excessive. These findings suggest that the proportion of direct subsidies should be reduced, while tax support should be increased to promote SMEs’ technological innovation.

Table 1. The effect size of direct and indirect support for firms

K=number of effect size, ES=effect size, CI=confidence interval, p=significance level

Source: Kim (2019)

4.2. Antecedents of SMEs’ Technological Innovation

Studies on technological innovation in SMEs have grown both quantitatively and qualitatively. However, due to a lack of data, diversity of methodologies, and variables, each study has limitations in terms of generalizability. Therefore, a systematic and comprehensive statistical approach that draws upon numerous empirical studies can help SMEs and government policymakers make more informed decisions.

Kim, Kim & Park (2019) conducted a meta-analysis to comprehensively analyze the technological innovation process in SMEs. The study analyzed the antecedents of technological innovation and the relationship between technological innovation and management performance. The authors used 62,512 samples from 111 domestic empirical studies and obtained the following results: to improve the technological innovation of SMEs, internal cooperation (.571), innovation culture (.532), dynamic capabilities (.526), and absorptive capacity (.501) were found to be important antecedents. These results suggest that the government should strategically focus on these factors to promote technological innovation in SMEs.

Table 2. The effect size of antecedents on SMEs technological innovation

* P<.05, ** P<.01, *** P<.001, K=number of effect size, N=sample size, ES=effect size, CI=confidence interval, P=significance level, Q=total variance (Cochran’s Q value)

Source: Kim, Kim & Park (2019)

4.3. Factors for Adoption of Smart Factory

The adoption of AI and information technology in the manufacturing process is leading to an acceleration in the transition to smart factories. Intelligent smart factories are known to offer significant benefits in terms of productivity and sustainability. However, many industries are hesitant to introduce smart factories due to concerns around excessive costs and uncertainty.

Kim, Jeong & Park (2023) focused on the smart factory, which is a critical paradigm in the digital transformation of manufacturing, and conducted a meta-analysis to systematically integrate statistical results from existing empirical studies. The study established an integration model, which involved key factors related to smart manufacturing adoption and performance, based on an analysis of 42 Korean literature sources. The analysis revealed that the key factors for the adoption and continuous use of smart manufacturing were the network effect (.714), social influence (.672), finances (.628), performance expectancy (.627), facilitating conditions (.606), technological capabilities (.573), and entrepreneurship (.547). The government should consider these upper-level factors while promoting smart factory diffusion policies.

Table 3. The effect size of factors affecting and use of smart factories

K=number of effect size, N=sample size, ES=effect size, CI=confidence interval, P=significance level, Q=total variance (Cochran’s Q value)

* Note: ‘innovation resistance’ and ‘perceived risk’ were analyzed by switching the effect size sign: (–) → (+)

Source: Kim, Jeong & Park (2023)

5. Conclusion

5.1. The Potential of Meta-analysis for Evidence-based Innovation Policy

Despite various efforts to promote evidence-based innovation policies using different data and methodologies, opinions remain divided on their sufficiency. This study explored the possibility of meta-analysis as a methodology for evidence-based innovation policy. Meta-analysis, as a methodology that systematically integrates existing empirical evidence, can be useful for exploring higher-level evidence. The three studies introduced in this paper demonstrate how meta-analysis can be used to establish innovation policies in the Korean context.

The first study inferred a more effective financial portfolio by comparing the effect size of direct and indirect government R&D support for SMEs. The second study identified the government’s strategic focus areas by comparing priorities among various factors that promote technological innovation in SMEs. The third study compared various factors affecting the introduction of smart factories to identify focus areas in the smart factory diffusion policy. These research results provide evidence-based implications to innovation policy stakeholders who are trying to create optimal effects within a limited budget range.

5.2. Limitations and Future Study

The three studies presented in this article have already been published in Korean (Kim, 2019; Kim, Kim & Park, 2019) or are scheduled to be published in English (Kim, Jeong & Park, 2023). This paper’s purpose is to introduce meta-analysis as a possible methodology for evidence-based innovation policy by highlighting these previously published studies. However, it is important to note that there is no evidence indicating whether the results of the meta-analysis presented in this study were actually used in Korea’s innovation policy. This situation can be attributed to various factors, such as a lack of reliability of researchers, insufficient understanding of meta-analysis, and inadequate dynamism of government policy. However, the implications of these research results are introduced in this paper because they may be useful in other contexts beyond Korea.

Although meta-analysis has the potential to support evidence-based innovation policy, there are still many possibilities to be explored. Future discussions related to this topic can be expanded through various approaches, such as comparative verification of the effectiveness of specific policies across countries to identify optimal policy instruments for each context. It is also possible to compare how the policy context has changed before and after major external events such as the global financial crisis and pandemic. The potential of meta-analysis is that it can compare the priorities of various variables or events at a higher level, making it suitable for researchers and policy stakeholders who wish to advance innovation policies.

Open science practices

The meta-analysis data used in the three studies (Kim, 2019; Kim, Kim & Park, 2019; Kim, Jeong & Park, 2023) introduced in this article are personally held by the researcher and can be made available to interested parties upon request, subject to appropriate data sharing policies and regulations.

Competing interests

The author declare that there are no competing interests regarding the publication of this article.

Funding information

The three papers (Kim, 2019; Kim, Kim & Park, 2019; Kim, Jeong & Park, 2023) introduced in this study utilized the research results of KISTEP (The study numbers managed by KISTEP for each of the three studies are as follows: 2018-027; 2019-001; 2022-011), and were conducted with funding from the Ministry of Science and ICT.

References

Aghion, P. & Howitt, P. (1992). A model of growth through creative destruction. Econometrica, 60, 323-351. https://doi.org/10.2307/2951599

Ahn, E. J. & Kang, H. (2018). Introduction to systematic review and meta-analysis. Korean Journal of Anesthesiology, 71, 103-112. https://doi.org/10.4097/kjae.2018.71.2.103

Borenstein, M., Hedges, L. V., Higgins, J. P. T. & Rothstein, H. R. (2009). Introduction to Meta-Analysis. Chichester: John Wiley & Sons.

Bowen, F. E., Rostami, M. & Steel, P. (2010). Timing is everything: a meta-analysis of the relationships between organizational performance and innovation. Journal of Business Research, 63, 1179-1185. https://doi.org/10.1016/j.jbusres.2009.10.014

Büschgens, T., Bausch, A. & Balkin, D. B. (2013). Organizational culture and innovation: a meta-analytic review. Journal of Product Innovation Management, 30, 763-781. https://doi.org/10.1111/jpim.12021

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Science (2nd ed.). NJ: Lawrence Erlbaum Associates.

Correa, P., Andrés, L. & Borja-Vega, C. (2013). The impact of government support on firm R&D investments: a meta-analysis. Policy Research Working Paper 6532, World Bank.

Damanpour, F. (1991). Organizational innovation: a meta-analysis of effects of determinants and moderators. Academy of Management Journal, 34, 555-590. https://doi.org/10.2307/256406

David, P. A., Hall, B. H. & Toole, A. A. (2000). Is public R&D a complement or substitute for private R&D? a review of the econometric evidence. Research Policy, 29, 497-529. https://doi.org/10.1016/S0048-7333(99)00087-6

Davies, P. (2004). Is evidence-based government possible?. Presented at the 4th Annual Campbell Collaboration Colloquium, Washington D.C.

Dimos, C. & Pugh, G. (2016). The effectiveness of R&D subsidies: a meta-regression analysis of the evaluation literature. Research Policy, 45, 797-815. https://doi.org/10.1016/j.respol.2016.01.002

Donselaar, P. & Koopmans, C. (2016). The fruits of R&D: meta-analysis of the effects of research and development on productivity. Research Memorandum 2016-1, Netherlands: Faculty of Economics and Business Administration.

Evanschitzky, H., Eisend, M., Calantone, R. J. & Jiang, Y. (2012). Success factors of product innovation: an updated meta-analysis. Journal of Product Innovation Management, 29, 21-37. https://doi.org/10.1111/j.1540-5885.2012.00964.x

Fisher, R. A. (1932). Statistical Methods for Research Workers (4th ed.). Edinburgh: Oliver and Boyd.

Gaillard-Landinska, E., Non, M. & Straathof, B. (2015). More R&D with tax incentives? a meta-analysis. CPB Discussion Paper 309, CPB Netherlands Bureau for Economic Policy Analysis.

Gay, J. A. M. (1997). Evidence-Based Health Care: How to Make Health Policy and Management Decision. New York and London: Churchill Livingstone.

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8. https://doi.org/10.3102/0013189X005010003

Grossman, G. M. & Helpman, E. (1991). Innovation and Growth in the Global Economy. Cambridge, MA: MIT Press.

Head, B. W. (2008). Three lenses of evidence-based policy. The Journal of Public Administration, 67, 1-11. https://doi.org/10.1111/j.1467-8500.2007.00564.x

Hedges, L. V. & Olkins, I. (1985), Statistical Methods for Meta-Analysis. San Diego, California: Academic Press.

Henard, D. H. & Szymanski, D. M. (2001). Why some new products are more successful than others. Journal of Marketing Research, 38, 362-375. https://doi.org/10.1509/jmkr.38.3.362.18861

Ioannidis, J. P. A. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quarterly, 94, 485-514. https://doi.org/10.1111/1468-0009.12210

Kim, J. (2019), Direct support vs. indirect support: exploration of policy mix for R&D investment of SMEs. Journal of Technology Innovation, 27, 1-43. https://doi.org/10.14386/SIME.2019.27.1.1

Kim, J., Jeong, H. & Park, H. (2023). Key drivers and performances of smart manufacturing adoption: a meta-analysis. Sustainability, decision in process.

Kim, J., Kim, M. & Park, H. (2020). A meta-analysis on antecedents and consequences of technological innovation: focused on empirical analyses of South Korea’s SMEs. Asia Pacific Journal of Small Business, 42, 43-67. https://doi.org/10.36491/APJSB.42.2.3

Li, J., Li, X. & Gao, P.-B. (2019). A meta-analysis of the relationship between service innovation and firm performance. 6th International Conference on Management Science and Management Innovation (MSMI 2019), Advances in Economics, Business and Management Research, 84, 256-261.

Mandal, S. & Sharma, D. (2016). Antecedents and consequences of product innovation: a meta-analytic review. W.P. No. 2016-03-47, India: Indian Institute of Management Ahmedabad.

Mendoza, M. L. (2015). Innovation across types of organization: a meta-analysis. Suma de Negocios, 6, 108-113. https://doi.org/10.1016/j.sumneg.2015.08.010

Ministry of Science and ICT & KISTEP (2022). 2021 National R&D Project Research and Analysis Report, Institute 2022-001, Chungcheongbuk-do: Korea Institute of S&T Evaluation.

Montoya-Weiss, M. M. & Calantone, R. (1994). Determinants of new product performance: a review and meta-analysis. Journal of Product Innovation Management, 11, 397-417. https://doi.org/10.1111/1540-5885.1150397

Mueller, V., Rosenbusch, N. & Bausch, A. (2013). Success patterns of exploratory and exploitative innovation: a meta-analysis of the influence of institutional factors. Journal of Management, 39, 1606-1636. https://doi.org/10.1177/0149206313484516

Negassi, S. & Sattin, J.-F. (2014). Evaluation of public R&D policy: a meta-regression analysis. Working Paper No. 2014-09, Department of Economics, Alfred Lerner College of Business & Economics, University of Delaware.

Negassi, S. & Sattin, J.-F. (2016). Are public R&D subsidies effective? some evidence from a meta-analysis of the literature. Proposition de Communication – Congrés AEI 2017.

Pattikawa, L. H., Verwaal, E. & Commandeur, H. R. (2006). Understanding new product project performance. European Journal of Marketing, 40, 1178-1193. https://doi.org/10.1108/03090560610702768

Pearson, K. (1934). On a new method of determining “Goodness of fit”. Biometrika, 26, 425-442. https://doi.org/10.2307/2331988

Peterson, R. A. & Brown, S. P. (2005). On the use of beta coefficients in meta-analysis. Journal of Applied Psychology, 90, 175-181. https://doi.org/10.1037/0021-9010.90.1.175

Romer, P. M. (1987). Growth based on increasing returns due to specialization. American Economic Review, 77, 56-62.

Rosenbusch, N., Brinckmann, J. & Bausch, A. (2011). Is innovation always beneficial? a meta-analysis of the relationship between innovation and performance in SMEs. Journal of Business Venturing, 26, 441-457. https://doi.org/10.1016/j.jbusvent.2009.12.002

Sackett, D., Strauss, S., Richardson, W. & Haynes, R. (2004). Evidence-based Medicine: How to Practice and Teach EBM. New York and London: Churchill Livingstone.

Sarooghi, H., Libaers, D. & Burkemper, A. (2015). Examining the relationship between creativity and innovation: a meta-analysis of organizational, cultural, and environmental factors. Journal of Business Venturing, 30, 714-731. https://doi.org/10.1016/j.jbusvent.2014.12.003

Shadish, W. R. & Haddock, C. K. (1994). Combining estimates of effect size. In H. Cooper & and L. V. Hedges (Eds.), The Handbook of Research Synthesis (pp. 261-261). New York: Russell Sage Foundation.

Solesbury, W. (2001). Evidence based policy: whence it come and where it’s going. Working Paper 1, ESRC UK Centre for Evidence Based Policy and Practice.

Song, M., Podoynitsyna, K., van der Bij, H. & Halman, J. I. M. (2007). Success factors in new venture: a meta-analysis, Journal of Product Innovation Management, 25, 7-27. https://doi.org/10.1111/j.1540-5885.2007.00280.x

Stanley, T. D. (2001). Wheat from chaff: meta-analysis as quantitative literature review. Journal of Economic Perspectives, 15, 131-150. https://doi.org/10.1257/jep.15.3.131

Storey, C., Cankurtaran, P., Papastathopoulou, P. & Hultink, E. J. (2015). Success factors for service innovation: a meta-analysis. Journal of Product Innovation Management, 33, 527-548. https://doi.org/10.1111/jpim.12307

Uetani, K., Nakayama, T., Ikai, H., Yonemoto, N. & Moher, D. (2009). Quality of reports on randomized controlled trials conducted in Japan: evaluation of adherence to CONSORT statement. Internal Medicine, 48, 307-313. https://doi.org/10.2169/internalmedicine.48.1358

Vincent, L. H., Bharadwaj, S. G. & Challagalla, G. N. (2004). Does innovation mediate firm performance?: a meta-analysis of determinants and consequences of organizational innovation. TI:GER Student Working Papers, 6, Georgia Institute of Technology.

Wolf, F. M. (1986). Meta-Analysis: Quantitative Methods for Research Synthesis. Beverly Hills, California: Sage Publications.

Zou, T., Ertug, G. & George, G. (2018). The capacity to innovate: a meta-analysis of absorptive capacity. Innovation: Organization & Management, 20, 87-121. https://doi.org/10.1080/14479338.2018.1428105