Crowdsourcing a Meeting of Minds: Designing the Future of Work

As far as labor revolutions go, crowdsourcing may not seem like such a groundbreaking concept. But according to Michael Bernstein, assistant professor of computer science at Stanford University, crowdsourcing and computation have the potential to revolutionize the way we work and share skills. Bernstein visited the GovLab this month as part of our Ideas Lunch series to share his research on how expert crowdsourcing can be used to achieve complex and sophisticated projects.

Computers are already having a profound influence on our employment. Researchers estimate that in the future, 20 percent of our workforce could exist online, representative of approximately 45 million workers. This staggering number shows that computers are more than just another tool in our office to improve productivity. Rather, as Bernstein revealed, computers are becoming vast, powerful networks which connect us with others, best seen in apps like Uber.

According to Bernstein, there is great potential locked away in these computerized networks to radically transform how work is performed. Traditionally, crowdsourcing has been used to complete menial, micro-tasks, seen in projects like Amazon’s Mechanical Turk which primarily uses crowdsourced labor for image labeling, data collection and other non-expert tasks. For Bernstein, such an approach neglects the potential of crowdsourcing to achieve complex, interdependent goals by curating crowds of experts.

With fellow researchers at Stanford University, Bernstein investigated whether “flash-teams” of crowdsourced experts could achieve ambitious results, like designing a hi-fi prototype of an app or making a short animation in just one day. By recruiting workers through the website UpWork, and creating a web platform Foundry to manage workflows, Bernstein and his team found that flash-teams were able to achieve goals significantly faster than self-managed teams, with almost 50 percent fewer work hours expended.


Nevertheless, Bernstein pointed out that these flash-teams are limited in what they can do. Flash-teams need pre-defined workflows so that tasks can be delegated and guided, and only small teams can be involved on a single project. For larger, more complex projects, where workflows may evolve or be undefined, flash-teams are unable to deliver sufficient results.

Furthermore, there are considerable ethical challenges to such crowdsourced forms of labor. Research by Bernstein’s colleagues into collective action by crowd workers found that “the technical infrastructure [of crowdsourcing] actively disempowers workers”, and that new forms of computationally-empowered labor collectives are therefore needed to meet the needs of this distinct workforce. But experiments in delivering such a model to connect and spur advocacy among crowd workers, specifically through the web platform Dynamo—where workers could propose ideas, vote on these ideas, and then discuss and mobilize action—revealed some of the shortcomings of computerized crowdsourcing. Particularly, though the web is adept in gathering a vast array of people quickly, it is also just as easy for people to quickly disperse if they lose interest in the cause or encounter an obstacle. Trying to coordinate collective labor actions therefore becomes more difficult than simply providing a space for workers to share and discuss ideas online.

Bernstein’s research into the challenges and benefits of expert crowdsourcing continues to make exciting discoveries. For instance, a current project seeking to crowdsource research participants from across the world suggests that crowdsourcing can even help solve open-ended, messy and large-scale problems. There remains a vast array of untapped possibilities for computerized crowdsourcing to bring workers together to tackle complex and multifaceted problems.

Key Takeaways

  • There are four features afforded by computational crowdsourcing which make crowdsourced Flash Teams effective:
    1. Modularity of crowdsourcing means that team structures can be replicated and scaled across projects;
    2. Elastic work-flows allow tasks and team members to grow and shrink dynamically depending on the evolving needs of the project.
    3. Pipelining allows incomplete results to be passed down the timeline to proceeding workers on a project. This means the entire system adapts to missed deadlines or unexpected changes to a project.
    4. Creation by request means that synthetic teams can be created instantly depending on the project proposed. Tasks are also translated into time dependent ‘strips’ of action divided among team members.
  • Self-managed teams don’t work, often because they are inefficient and poorly coordinated, leading to frustrations among team members.
  • Computational crowdsourcing provides ‘light scaffolding’ enabling workers to be shepherded through tasks and for schedules and files to be shared between members through the workflow.
  • Flash-teams (mean time to finish is 13hr 2min) are significantly faster than self-managed teams (mean time to finish is 23hr47min), p=0.05
  • If crowdsourced Flash Teams are a new form of work collective, there is also a need for new forms of worker counterbalance.

All this allows us to glimpse at what the future of work might look like, and, according to Bernstein, we can expect crowdsourcing to achieve more complex and interdependent goals, to better advocate for pro-social outcomes, and to solve open-ended challenges.

About Michael Bernstein

Michael Bernstein is an Assistant Professor of Computer Science at Stanford University and member of the Human-Computer Interaction group. His research focuses on the design of crowdsourcing and social computing systems. This work has received five Best Paper awards and eleven honorable mentions at premier venues in human-computer interaction and social computing. Michael has been recognized as a Robert N. Noyce Family Faculty Scholar, and awarded the Sloan Fellowship, NSF CAREER award and the George M. Sprowls Award. He holds a bachelor’s degree in Symbolic Systems from Stanford University, and a master’s and Ph.D. in Computer Science from MIT.