A new paper released by GovLab Research looks into “the promise and challenge of evaluating new practices of problem solving in governance, specifically citizen-engagement interventions.” The purpose of the paper is to inform those innovators in governance who are eager to start developing a deeper insight into what works.
The paper starts from the premise that:
While there is good reason to believe that breakthroughs may come from recent innovations such as community-based problem solving, behavioral economic insights about human behavior, or predictive analytic experiments, there are limited studies measuring exactly how productive it is to use these kinds of new governance techniques. Without a deeper understanding of whether, when, why and to what extent an intervention has made an impact, any initiative we design will be sub-optimal and will produce less than the desired results. If we are going to accelerate the rate of experimentation in governance and create more agile institutions capable of piloting new techniques and getting rid of ineffectual programs, we need research that will enable us to move away from “faith-based” engagement initiatives toward “evidence-based” ones.
Any evaluation is a complex and challenging task that requires judgment to choose how, what and when to measure along with the criteria for an intervention to be deemed successful. At the same time, there is an extensive and emerging body of knowledge on how to evaluate best. GovLab’s review of the existing literature on evaluation identified 3 key components behind every effort to measure:
- CONCEPTUAL FRAMEWORK: To determine an intervention’s impact, an evaluation must be based upon a concept or theory of change.
- METRICS AND INDICATORS: The selection of indicators and metrics for assessment is necessarily a value-based decision, since ultimately we measure what we deem important.
- METHODOLOGY: Once indicators have been established, data can be collected in a number of different ways.
These basic considerations of evaluation—conceptual framework, metrics and indicators, and methodology—are critical in assessing any societal intervention. The paper subsequently highlights particular considerations in applying each of these to assessments of citizen engagement and data-sharing in governance. A core challenge to measure impact in this space involves the absence of a logic model or conceptual framework making sense of the innovative intervention along with a wide diversity of (often unstated) goals underlying citizen engagement and data-sharing in governance.
The paper ends by focusing on four themes and questions that are important to examine further as we move forward in evaluating interventions in governance innovation:
Different horses for different courses. “Citizen engagement” comprises many different “means” to achieve many different “ends.” Depending on the context, citizens can play different roles: as providers of ideas and expertise (think of crowd sourcing, predictive analytics, grand challenges, prize-induced innovation, brainstorming, etc.); or as representatives of specific interests (in the context of participatory budgeting, citizen juries and deliberative polling). And the contexts of engagement may differ substantially—from post-conflict zones to gentrified city blocks. How do we provide answers to the questions we really should seek to answer: to achieve certain participatory objectives, what works, with whom, and under what conditions?
Improving people’s lives. Too often, the indicators used to measure citizen engagement are only meant to quantify the level of engagement—such as the number of people that participated or the volume of comments received. For citizen engagement to be meaningful and relevant, more effort is needed to answer the question: How do we start determining what the impact is on on people’s lives?—the ultimate benchmark of success.
We measure what we value. Indicators (particularly of the statistical kind) are sometimes presented as neutral or scientific tools of measurement. In fact, though, they are inextricably linked with values—i.e., we measure what we care about. Indicators are socio-political constructs. As such, we should consider whether and how to involve citizens in determining how citizen engagement is measured (possibly using new on-line mechanisms such as rating and feedback tools). How can we best engage with citizens to determine what success should look like and what to measure?
Experimentation in how we measure. While much experimentation in citizen engagement is taking place, experimentation in how we measure citizen participation is limited. Recent studies have shown that a mix of methodologies is needed to evaluate interventions in government. Reliance on the traditional gold standard of RCTs is not appropriate in many instances, and at the same time, relying solely on anecdotal evidence is no longer sufficient. But advances in both these areas hold out promise. In order to distill best practices and lessons learned, the field would benefit from increased experimentation using a variety of methods to understand the value that open government brings to people and the difference it makes in their lives. In particular, the use of big data to provide real time feedback that allows us to witness the impact of change to policy “as it happens” offers many opportunities for faster experimentation. How can we improve evaluation of governance innovation through increased experimentation in methods and practice, including the use of big data?
Ultimately, the paper is not meant to be a comprehensive review of current approaches to evaluating new governance initiatives but is intended to frame the issues involved, and suggest where work needs to be done to develop better assessments.
To solicit your engagement and debate we have made the paper available for everyone to annotate and invite you to also to expand upon the open bibliography included in the paper.