On April 19, the second day of the GovLab Experiment, participants took part in a variety of half-hour “horas” with specific themes to guide discussion. The “If Only We Knew…” session asked participants, “What are the most important research questions that, if answered, would radically transform our understanding of open government?”
Horas 1 and 3 discussed research questions revolving around determining when crowds are wise, how to foster that wisdom and how to develop metrics to support those findings.
Participants in Horas 1 and 3 were both focused on understanding the specific conditions in which knowledge obtained through crowdsourcing can outperform expert input. And, accordantly, the specific conditions in which crowdsourcing should not be the knowledge-acquisition method of choice. With a more strategic understanding to guide the deployment of crowdsourcing, we can not only get more value from crowdsourcing efforts, but also save time and resources by not directing questions to the crowd that are not well-suited to that type of information gathering. Of course, to gain this understanding, and more strategically deploy crowdsourcing efforts, meaningful metrics need to be developed for measuring the successes and failures of crowdsourcing. Of course, metrics can serve to do more than determine when crowdsourcing can be most effective, they can also provide guidance on data types and platforms that are best-suited for engaging and leveraging the wisdom of the crowd.
Below are illustrative examples of the challenges we are experiencing due to our current lack of understanding in this area, current attempts to find out how rectify this lack of understanding and some of the potential benefits of finding meaningful answers.
Challenges From Not Knowing
- James Surowiecki cites NASA’s Clickworkers experiment—which called on tens of thousands of amateurs to look at photos of Mars and identify and classify craters by age—in his article noting the failure of Reddit to crowdsource the Boston Marathon bombing manhunt. He notes that, “The problem from Reddit’s perspective, of course, is that [the Clickworkers] method of sleuthing would be far less exciting for users, and would probably generate less traffic, than its current free-for-all approach. The point of the ‘find-the-bombers’ subthread, after all, wasn’t just to find the bombers—it was also to connect and talk with others, and to feel like you were part of a virtual community. But valuable as that experience may have been for users, it also diminished the chances of the community coming up with useful information. Reddit has done an excellent job of being engaging. Now it needs to figure out if it wants to be effective.”
- David Eaves points to an example where Canada sold public census data to the private sector, “to help with planning where to locate stores or how to engage in marketing and advertising effectively.” That data was sold to the private sector due to a lack of recognition regarding the value it could potentially hold for “academics, non-profits and everybody else, for whom it should have been free, as it was in the US.” So instead of opening this data to the crowd—a crowd that was likely primed to create a variety of unexpected value—Canada received some amount of direct monetary investment and far less, far more predictable innovation and market activity by relying on traditional private actors.
Current Attempts to Find Out
- Panthea Lee, a principal at Reboot, is working to develop an impact assessment system for determining the true effects of open government projects, including open data initiatives for leveraging the skills and wisdom of the crowd, rather than focusing on simple metrics, like number of app downloads.
- GovLab produced a paper, “Toward Metrics for Re-Imagining Governance: The Promise and Challenge of Evaluating Innovations in How We Govern,” that explores the wide variety of efforts and frameworks to develop meaningful evaluation mechanisms for innovative governance projects. While there are a number of promising steps being taken, it is evident that no one has yet cracked the metrics nut.
Potential Benefits of Finding the Answer
- Finding the answer to these questions would lead to more unexpected uses for crowdsourcing, like the efforts of the Russian organization RosPil to hold “corrupt politicians and bureaucrats’ feet to the fire largely through internet-based crowdsourcing, whereby often-anonymous people identify requests for government-issued tenders that are designed to generate kickbacks.” Like India’s I Paid a Bribe, RosPil is using crowdsourcing to guide action in an area that isn’t immediately obvious where the crowd is in fact wise.
- Successfully and intelligently drawing on the wisdom, and funding, of crowds could help change our cities for the better. Rodrigo Nino hopes that the crowd in Bogota realizes, unlike the local real estate industry, that the city needs to develop vertically—i.e., through skyscrapers—rather than horizontally. If he’s right, and this is an area in which the crowd is wise and can have a meaningful effect on government policy and planning, it could revolutionize development in Bogota and beyond.