How Do We Know Who Knows What?

Screen Shot 2013-12-13 at 4.41.45 PMThis is a central question we are grappling with at the GovLab. We want to understand how people disclose their knowledge, experience, know-how and interests; how to make expertise searchable; and how to match relevant expertise with public challenges that need solving. Because we believe in the expertise of the public-at-large, we want to test whether government can engage with citizens on the basis of what they know — not about politics and civics — but what it takes to solve hard problems.

Traditionally, expertise is associated with credentials and institutional affiliations. We use credentials, such as a degree, as a proxy for expertise. The higher the rank of the institution that has accredited someone, the higher we rate her personal expertise. Credentials however only convey a tiny slice of the skills and expertise people possess.

Interests may be a more inclusive proxy for expertise. If I love sports, regardless of whether I have a degree in physiology or ever played for a baseball team, I might be more likely to know the ins and outs of the game. More precisely, while the baseball player can throw a cure ball, the ardent fan might do a better job of calling the plays. In fact, the avid fan might be more expert on the stats than the player. Similarly, the patient may have more expertise than the highly credentialed doctor about certain aspects of disease care and treatment because they are much more interested in finding something that works. While certain kinds of expertise require doing, the astute observer is often better poised to communicate what she knows.

Professor Panos G. Ipeirotis, Associate Professor and George A. Kellner Faculty Fellow at the Department of Information, Operations, and Management Sciences at Leonard N. Stern School of Business of New York University has been running experiments that get at the heart of this question. He spoke about his work at The GovLab Ideas Lunch (see his slides here). While on sabbatical at Google, Ipeirotis set out to develop approaches to “crowdsourcing all the knowledge in the world.” To that end, he designed to explore whether it was possible crowdsource in a predictable manner with knowledgeable users.

Using Google keywords, Ipeirotis advertised to potential participants based on their browsing habits the opportunity to participate in a fact-based knowledge quiz. For example, if you are on WebMD, you receive an ad asking you to take a health quiz on disease symptoms. He wanted to see if he could get good users not just clicks.

The system, called, automatically generates questionnaires using the structured data that Google Knowledge Graph maintains about people, music, books, songs, sports and other categories of information. For example, every person in Google Knowledge Graph has an associated set of standard attributes: birthday, birthplace, spouse, parents, children. These common data elements can be used to automate the creation of a quiz question about the spouses of famous people. “Who is the spouse of Barack Obama?” as well as a set of answer choices that are correct and incorrect.

Getting people to take the quiz eventually could create a way for lots of eyeballs to help Google improve the quality of information in its knowledge graph. But the real goal of the experiment currently running is to answer the question whether people who are interested in a topic are more knowledgeable about it? The answer is a resounding yes.

By targeting a request to participate health quizzes to Google users browsing on Web MD or the Mayo clinic, the project was able to increase the rate of participation by 10x. In addition to the quantity of the participation, volunteers from the targeted pool were also 20% more likely to answer correctly than those who found the quiz from other places. The self-selected pool also answered faster than those who were paid on Mechanical Turk or oDesk to answer the same questions for a fee.

The experiment of using Google ads to target people — what we at the GovLab call Crowdsourcing Wisely Not Widely — is yielding fruitful insights. confirms what we learned in Peer to Patent, namely that targeting works but must be combined with self-selection to drive performance. In other words, people see an ad but they aren’t required to participate and the results surpass hiring people to do the same work. Self-selection is more important than selection.

Once people found the quiz, showing them the correct answer was ten times more likely to result in someone participating again than simply showing them if their answer was correct. Showing people a leaderboard for the last few days of a current quiz, which conveyed performance that was achievable, improved ongoing participation over showing an all-time leaderboard that made it seem impossible to beat the best users. Not surprisingly but powerfully confirmed here, learning and competition are drivers of participation.

Finally, if browsing habits say something about interests then interests seem to be a powerful proxy for expertise. The WebMD surfer is better at answering health questions than the paid participants. This suggests that, instead of one-size fits all participation opportunities As we describe in our Research Agenda on Smarter Governance, we think that the same techniques an advertise users to target users, can help us to go beyond voting and connect people to new opportunities to participate in tackling important public challenges.

Just as reCaptcha repurposed Captcha to digitize old documents for the National Archives while administering the Turing test to shoppers, Ipeirotis’ experiments are getting people to answer quiz questions that are transforming our understanding of how to crowdsource wisely not just widely.

Author’s note: Panos’ research paper with the Quizz findings was accepted for presentation at the WWW2014 conference in Korea in April.

No comments yet.

Leave a Reply