Background

Estimating the macroeconomic effects of investments in public research and innovation is of central interest to policymakers, scientists and the general public alike. Many studies have attempted to provide quantifications of these returns, often finding substantial effects (Carree et al. 2014, Jones and Summers 2020, Comin 2021) in terms of GDP, employment, investment and tax revenues (Roy et al. 2021). Two very recent examples by Schubert (2021) and Allan et al. (2022) analyze the case of Fraunhofer as the biggest organization for applied research in Europe. One of the central findings of these studies was that increasing the Fraunhofer budget by one euro increases GDP by 21 euros.

Such monetarized estimates seem to suggest a high degree of precision. However, because modern economies are incredibly complex systems, estimating such macroeconomic effects is not a trivial undertaking and requires modelling assumptions and choices to be made, which may have more than a marginal effect on the results. This begs a number of questions: How precise are these estimates? How strongly do they depend on modelling assumptions? Can policymakers reasonably base potentially far-reaching budget decisions on such figures?

To answer these questions, it is important to have a better understanding of the methodological challenges that the scientists making such estimates have to face. Explaining these key challenges is the purpose of this blog-post, which aims to sensitize the readers as the recipients and users of such figures to the potentials and pitfalls of working with them. Specifically, I will argue that such figures should never be taken as precise estimates that are valid up to the last digit. Instead, provided that the studies are executed well, these figures indicate as good a benchmark as possible. It is therefore important that such studies conduct and present reasonable robustness checks of the key modelling assumptions. In this way, they give the readers an honest account of how much the results depend on the modelling assumptions and consequently where the limits to the interpretations lie.

The key challenge – reliance on observational data

Empirically determining the effects of public research on the economy has to rely on observational data ("field data"), i.e. data that is non-randomized unlike in clinical trials. Estimating causal effects based on field data requires great statistical care, the application of modern statistical methods and profound economic knowledge of the processes linking public research to economic outcomes. Based on the studies of Fraunhofer by Schubert (2021) and Allan et al. (2022), I will outline some of the key objections and challenges, and then explain how the studies have dealt with these issues:

Compared to other types of public investment, the Fraunhofer effect is unreasonably large: Studies estimating only the consumption or investment effects of public research find only up to 2 euros for each euro spent. However, these studies ignore knowledge effects and thus ignore innovation. Scientific studies have shown that the returns from innovation are substantial and typically exceed the original investment costs  by large multiples – ten to 25 times for reasonable parameterizations (compare, e.g., Jones and Summers 2020). The study on the economic value of the Fraunhofer-Gesellschaft (Schubert 2021) finds a cost-benefit ratio of 1:21, which is of a similar magnitude to that found by Jones and Summers (2020) for the economic value of R&D. Even larger effects were found by Comin (2021). Because all the studies mentioned build on completely different macroeconomic modelling approaches, the fact that their results are consistent in their magnitude is a strong indication of the resilience of the calculations.

Instead of causing higher GDP, Fraunhofer may be primarily located in economically strong regions. This argument refers to what econometricians call reverse causality. In this case, Fraunhofer does not cause GDP to increase, but is attracted to high GDP regions. To control for such problems, statisticians use complex statistical models to correct for such estimation biases. The studies by Schubert (2021) and Allan et al. (2022) employed a number of modern techniques to account for reverse causality, including but not limited to methods that create homogenized "control" regions that are similar in their economic characteristics. By applying this approach, it is possible to rule out the effect of Fraunhofer potentially selecting strong economic regions. Indeed, the results of this approach show that one euro of Fraunhofer spending increases GDP by 18 euros, which is slightly lower than the original result of 21 euros without this control. The studies therefore showed that locational choices do affect the results, but not so strongly that they invalidate the conclusion that Fraunhofer spending has substantial economic effects. After all, these effects are still at least 18 euros.

The effects may not result from Fraunhofer activities, but from unaccounted activities of universities in the region. This is an example of the failure to include relevant control variables. The result of this failure would be that an effect that should be attributed to the universities is empirically misattributed to Fraunhofer. In particular, Allan et al. (2022) address this problem by controlling for the regional presence of universities. This approach ensures that the estimated Fraunhofer effect is indeed the result of Fraunhofer itself and cannot be fully explained by universities in the same region. The effects for Fraunhofer are largely unchanged. This does not mean that universities do not matter – indeed, the opposite is true, as shown in Schubert and Kroll (2016). However, it does mean that the Fraunhofer effect of 21 euros is not due to misattribution resulting from not controlling for the presence of regional universities.

Large shares of the economic effects of Fraunhofer spill over to other regions and are thus unaccounted for. It is well known that a substantial share of the economic effects of public research is non-localized. Indeed, Schubert and Kroll (2016) provide consistent evidence of the role of spillovers in the case of universities. While the Fraunhofer study does not account for this effect, including spillovers would lead to even larger effects as these would logically be added to the estimated localized effects. The estimated localized GDP effects of Fraunhofer can therefore be considered as a lower bound. 

Conclusion

Estimates of the effects of public research rely on observational data, but the analysis of observational data comes with specific statistical challenges. Managing such challenges is key to ensuring the high quality and reliability of the studies. Robustness checks usually show that the estimated effects are not exact values, and provide lower and upper bounds for these effects in well-designed studies. These robustness checks should address important challenges such as reverse causality, locational choice or confounding mechanisms, which could lead to misattribution of the effects. In the studies by Schubert (2021) and Grant et al. (2022), the range was between 18 and 29. Clearly, this range is broad. However, even if the most conservative estimate is assumed, the effects are still very large and economically significant. It can therefore be assumed that the estimated returns of the Fraunhofer-Gesellschaft with its primary focus on applied research are economically plausible in size. Indeed, the results make it clear that application-oriented research as such, and the role of Fraunhofer in particular, have enormous economic potential in the given constellation of actors in Germany. State investment in this area therefore represents an essential contribution to the economic success of the German innovation system.