The GovLab recently hosted an Ideas Lunch featuring Julia Lane, an Institute Fellow and Senior Managing Economist at the American Institutes for Research. Lane spoke with us about the importance of measuring the results of public investment in research and development, and the many possibilities afforded by leveraging big data in this arena. The GovLab has examined the need for and current scarcity of evidence in decision-making practices in government (see our Index on Measuring Impact with Evidence here), and we were keen to hear Lane’s talk on this topic.
During the talk, Lane outlined the background behind the push to measure impact through research and development investments, some of the current challenges in the field, and a conceptual framework for empirical measurement. She also delved into the issues, challenges and potential of Big Data, and some thoughts about the future of efforts and initiatives in this space.
According to Lane, a new “science of science policy” is emerging that may offer compelling guidance for policy decisions and credible advocacy. She began her talk by reviewing the editorial “Wanted: Better Benchmarks” by John H. Marburger, former science advisor to President Bush, which pointed out the need for the scientific community to show the government why a nation should invest in science. The editorial raised some core questions, including how much should a nation spend on science? What kind of science? How much funding should come from private versus public sectors?
While the argument for investment in the sciences has been that such investments lead to job creation and innovation, impact has been hard to concretely document. A White House report, “A Federal Research Roadmap” found that “the data infrastructure is inadequate for decision-making”; the reasons for this finding, Julia explained, is that we don’t know what’s funded, who is funded, and we don’t know the impact.
To address some of these challenges, Lane discussed the potential of using Big Data to understand the impact of investment in the sciences. Some of the first steps in doing so, according to Lane, is to develop an empirical framework that is timely, generalizable and replicable, low cost and high quality. Moreover, she stressed the need for disambiguated data on individuals that is generated automatically, for example, through text mining, rather than manually filling out forms.
However, some of the challenges of using Big Data is that there is too much heterogeneous information in different formats across agencies, such as the lack of common identifiers or classifications for grants. However, Lane emphasized that by collaborating across agencies and by drawing on multiple data sources, researchers could use Big Data to understand the impact of research investment on subsequent science output in new and interesting ways. For example, a wide range of datasets could be combined to build a description of workforce dynamics and economic activity both at the level of an institution and larger scale, or to understand the impact of diversity in teams on productivity, or to trace and understand entrepreneurial activity.
Lane also took us through a conceptual framework for understanding the processes involved in investing in the sciences. She pointed out that it is a misconception that funding directly results in products; instead, funding is an intervention that is awarded to institutions, through which it reaches people, and then finally products. According to Lane, a theory of funding nurtures and sustains a group of individuals who transmit ideas over time, and that’s what you want to trace.
Some key takeaways from Lane’s talk:
- In order to answer key questions about investment in the sciences (such as where and how much to invest), it is vital to measure the impact of investment
- Currently, the data infrastructure to assess this investment is inadequate
- Big Data can be used to measure the impact of investment in a wide variety of ways; however, some current challenges must be overcome, such as the lack of common identifiers for grants, and the lack of collaboration across agencies
- One of the first steps in using Big Data to answer these questions is to develop an empirical framework that is timely, generalizable and replicable, low cost and high quality, as well as to use disambiguated data
- Overall, funding does not directly produce products in a vacuum: the impact of funding is far-reaching and while it is awarded to institutions, it also nurtures and sustains a wide range of individuals (not just principal investigators but their graduate students and more) and then also results in products