The Future of Crowd Work
6 likes2,944 views
This document discusses the future of crowd work and proposes a vision for how crowd work could be improved. The authors envision crowd work involving more complex, creative tasks completed by distributed teams rather than just simple, independent microtasks. They propose a framework drawing from distributed organizations and computing to address challenges like task decomposition, team assembly, and workflow execution. The framework identifies 12 research challenges across crowd workers, crowd work design, and crowd computation that must be addressed to realize this vision of crowd work where people would be proud if their children pursued it as a career.
![http://flic.kr/p/9dGryK
estimated future volume [Blinder 2006, Horton 2013]
$454,000,000,000 per year
91,000,000,000 hours per year
45,000,000 full-time workers
Friday, March 1, 13
And in fact, Blinder argues that about 20% of current American jobs could be sent down a
wire. These include tasks like programming, accounting, marketing, and even machine
operators. Recent evidence for crowd work in particular suggests that its volume will be
roughly 454 billion dollars per year. That’s 91 billion hours per year, employing about 45
million fulltime workers.
What might this mean? Think of current workers having the ability to become fulltime
contractors, able to control their jobs and their career as they desire. On the other end of the
spectrum, we get far more flexibility in time, like a stay-at-home dad who uses his skills
while the baby is sleeping.](https://image.slidesharecdn.com/futureofcrowdworkmerged06final-130302010557-phpapp02/85/The-Future-of-Crowd-Work-5-320.jpg)
![Interdependencies
Required expertise
Drop out
Low quality work
Propagation of errors
Unpredictability
http://flic.kr/p/7nEE5K
Friday, March 1, 13
Let’s say we tried to write a symphony with the crowd using the simple approach. Obviously
everyone generating their own stanza and putting them together wouldn’t work, though I’m
told that this is how they write Justin Bieber’s music.
Early bad decisions would propagate throughout the work, people would drop out midway
through writing, and it would be impossible to get a global view. [switch]](https://image.slidesharecdn.com/futureofcrowdworkmerged06final-130302010557-phpapp02/85/The-Future-of-Crowd-Work-10-320.jpg)
![Friday, March 1, 13
But if you look at crowd work platforms, those kinds of mechanisms are totally absent. Like
Netflix and eBay, there’s a collapse toward high ratings and people signal their expertise like
on Craigslist. [examples]](https://image.slidesharecdn.com/futureofcrowdworkmerged06final-130302010557-phpapp02/85/The-Future-of-Crowd-Work-17-320.jpg)
![“I have 4.2 stars out of 5
as a composer!”
Friday, March 1, 13
The future of reputation is not going to be saying [quote]](https://image.slidesharecdn.com/futureofcrowdworkmerged06final-130302010557-phpapp02/85/The-Future-of-Crowd-Work-18-320.jpg)
![Workflow
CrowdWeaver [Kittur et al. 2012]
Find-Fix-Verify [Bernstein et al. 2010]
TurKit [Little et al. 2009]
CrowdForge [Kittur et al. 2011]
Friday, March 1, 13
crowd work today looks like this. as researchers we’re trying to make more complex
workflows, but it still only looks like this.](https://image.slidesharecdn.com/futureofcrowdworkmerged06final-130302010557-phpapp02/85/The-Future-of-Crowd-Work-30-320.jpg)
The Future of Crowd Work
- 1. The Future of Crowd Work http://flic.kr/p/4vLS2o Aniket Kittur, CMU Elizabeth Gerber, Northwestern Matthew Lease, UT Austin Jeff Nickerson, Stevens Aaron Shaw, Northwestern John Horton, oDesk Michael Bernstein, Stanford John Zimmerman, CMU Friday, March 1, 13 even though only michael and i are up here, this paper was wri6en by a crowd that includes jeff, etc. and what we're trying to do is to go beyond what we currently think of as crowd-‐SOURCING to envision a bigger future for crowd work. there has been a lot of concern lately about online labor and the risks of it becoming exploitaHve, low pay, and monotonous.
- 2. What would it take for us to be proud of our children growing up to be crowd workers? Friday, March 1, 13 aIer becoming a father a year and a half ago (and many of you with children, nieces or nephews may relate), i started to really worry about what would the world look like for my daughter ashima when she grows up. would i want her to be entering a workforce that looked like that? so this paper is framed around the quesHon: what would it take for us to be PROUD of our children growing up to be crowd workers?
- 3. cognitive psych, management, computer science, design, sociology, economics Friday, March 1, 13 that's a pre6y ambiHous goal, and one "I" couldn't answer on my own, so we brought together a bunch of researchers in fields ranging from psychology to ... as well as geSng input from crowd workers, and when you get that many people together... you end up wriHng the longest paper in cscw history. but what we also did is to try to envision a future for crowd work that we would WANT our children to parHcipate in.
- 4. Crowd work Tasks completed online by a distributed, elastic workforce in exchange for pay. Friday, March 1, 13 By crowd work, we mean any work that happens online, and is completed by a distributed, dynamically sized workforce in exchange for pay. So that means not just Mechanical Turk, but potentially any work that could be sent down a wire.
- 5. http://flic.kr/p/9dGryK estimated future volume [Blinder 2006, Horton 2013] $454,000,000,000 per year 91,000,000,000 hours per year 45,000,000 full-time workers Friday, March 1, 13 And in fact, Blinder argues that about 20% of current American jobs could be sent down a wire. These include tasks like programming, accounting, marketing, and even machine operators. Recent evidence for crowd work in particular suggests that its volume will be roughly 454 billion dollars per year. That’s 91 billion hours per year, employing about 45 million fulltime workers. What might this mean? Think of current workers having the ability to become fulltime contractors, able to control their jobs and their career as they desire. On the other end of the spectrum, we get far more flexibility in time, like a stay-at-home dad who uses his skills while the baby is sleeping.
- 6. Friday, March 1, 13 However, I think many of us most strongly associate the concept with the online labor marketplace Amazon Mechanical Turk. Mechanical Turk has framed most of our discussions about paid crowdsourcing.
- 7. Crowd work: Present Friday, March 1, 13 The problem with that is that Mechanical Turk, and most crowdsourcing discussions, look like this. The work is small, simple, independent tasks and the workers are undifferentiated and pseudonymous. Work is automatically aggregated, typically by an algorithm. Image labeling, part of speech tagging, and so on.
- 8. http://flic.kr/p/dbAu4k Intellectual framing of low-cost results & exploitative labor Requesters view workers as exchangeable and untrustworthy Workers view requesters as distant and capricious Friday, March 1, 13 This intellectual framing of crowd work, both in academia and industry, has largely led to goals around low cost. We run the risk, as Six Silberman, Lilly Irani, and Ben Bederson have noted, of falling into this framing permanently. In that framing, requesters will view workers as undifferentiated, discardable and essentially shirkers. Likewise, the workers will view the requesters as distant and wielding incredible power to deny payment or harm reputations.
- 9. Crowd work: Future • Much real work is complex, creative, and interdependent • Writing a news story • Programming software • Composing a symphony Friday, March 1, 13 This is an incredibly dangerous framing for a number of reasons, but one major one is that work simply doesn’t look like that. The things we really want to help people coordinate to accomplish are complex, they’re creative, and they’re interdependent. Think about writing, engineering software, or consider: could a crowd compose a symphony?
- 10. Interdependencies Required expertise Drop out Low quality work Propagation of errors Unpredictability http://flic.kr/p/7nEE5K Friday, March 1, 13 Let’s say we tried to write a symphony with the crowd using the simple approach. Obviously everyone generating their own stanza and putting them together wouldn’t work, though I’m told that this is how they write Justin Bieber’s music. Early bad decisions would propagate throughout the work, people would drop out midway through writing, and it would be impossible to get a global view. [switch]
- 11. distributed distributed organizations computing http://flic.kr/p/bLcQBX http://flic.kr/p/6SZom Friday, March 1, 13 So how can we get there? Well, we need some kind of roadmap or framework for helping us think about where we need to go and what are the challenges we need to overcome to get there. To develop this framework we draw on ideas from distributed organizaHons on the one hand and distributed compuHng on the other. The idea is that both are facing the same fundamental challenges, First, of decomposing tasks...
- 12. WORK WORKERS Decompose task Assemble teams WORKFLOW Execute workflow OUTPUT Friday, March 1, 13 the first challenge is to decompose the task: if you were planning a conference, you might split up finding a venue from reviewing papers if you were google, you might split up different parts of the web for different machines to process need to assemble the right teams of people: if you were conference chair you need to find respectable academics or coercable friends to be on the committee if you were google: assign different machines play different roles, like a master node coordinating a mapreduce process finally, you have to execute workflows, which may have multiple stages and decision processes for a conference we have multistage review processes in computing, we have algorithms: for example, the output of one mapreduce process may get passed to another
- 13. Future Model of Crowd Work WORK WORKERS platform motivation task decomposition job design hierarchy reputation task assignment WORKFLOW collaboration real-time work workflow quality assurance crowds AI OUTPUT Friday, March 1, 13 Synthesizing these we propose a model of the future crowd work in which we might use crowd work to accomplish complex and creaHve tasks, and idenHfy 12 key research challenges we need to solve in order to get to a posiHve future for crowd work. Now I don't expect you to read all of these; what we'll do in the rest of the prez is to give you a brief flavor…
- 14. Outline • Brief flavor of three core areas: • Crowd Workers • Crowd Work • Crowd Computation Friday, March 1, 13 We’ll discuss research challenges associated with three core areas and provide some high level goals as grand challenges for the community. for more details, including a review of the current state of the field in each of these areas and calls for action, see the paper.
- 15. CROWD WORKERS: credentials and education Friday, March 1, 13 We’re going to start with opportunities for crowd workers. I’m going to focus on questions of education and credentials.
- 16. “I graduated from Stanford.” “I work for Google.” http://flic.kr/p/6vB32z Friday, March 1, 13 Today there are important status markers that we use to establish our reputation. I can say things like...
- 17. Friday, March 1, 13 But if you look at crowd work platforms, those kinds of mechanisms are totally absent. Like Netflix and eBay, there’s a collapse toward high ratings and people signal their expertise like on Craigslist. [examples]
- 18. “I have 4.2 stars out of 5 as a composer!” Friday, March 1, 13 The future of reputation is not going to be saying [quote]
- 19. Reputation Goal: mechanisms as straightforward to deploy and interpret as today’s affiliations and degrees Friday, March 1, 13 We need credentialing, because reputation is critical to setting up meaningful paths to success. Our goal here should be to develop a reputation mechanism that’s as easy to deploy and interpret as affiliations and degrees.
- 20. Education “hello world!” http://flic.kr/p/7FWaZK Friday, March 1, 13 To get to credentialing, you need to think about education. The internet has been a real boon to self-driven learning, with people engaging in massive online courses and learning-by- doing. Crowd work has the opportunity to drastically empower this process. We want to think about how to help someone like this, who’s hacking together their first Arduino prototype, to go from their “Hello World” program
- 21. Education “hello world!” expert http://flic.kr/p/7FWaZK Friday, March 1, 13 ...to becoming an expert maker and hacker. Can we facilitate this process and deliver work suited to the person’s expertise, all the way along that process?
- 22. Education Goal: crowd work = education http://flic.kr/p/7FWaZK Friday, March 1, 13 crowd work drives the demand for the next wave of education, and education relies on crowd work for learning opportunities and credentialing. The job she wants to do involves C programming on an arduino. So she turns to MOOCs to learn C, and the class uses simpler work opportunities as project assignments. It keeps pushing her with work at the edge of her abilities until she can tackle the expert requirements. Then, she can use her credentials from that class to take on new, higher-paying and more interesting jobs.
- 23. CROWD WORK Friday, March 1, 13 When you think of work in tradiHonal organizaHons one of the first things you think of is hierarchy and management.
- 24. CROWD WORK hierarchy Friday, March 1, 13 But in crowd work we currently see very li6le of this. And so we propose the grand challenge of what if crowd work could be as good as the best managers.
- 25. Hierarchy goal: crowd work as good as the best managers Friday, March 1, 13 what if crowd work could be as good as the best managers. and when we think of good managers, you might think of this guy
- 26. Tim Cook CEO, Apple $378,000,000 Friday, March 1, 13 Here’s tim cook, ceo of apple, who many think is one of the best managers in the world. Tim got paid $378M last year, which is probably more than most crowd workers. And he does a lot of things for Apple, ranging from hiring and firing to streamlining Apple’s supply chain and improving apple’s workflow.
- 27. John Doe CEO, Crowd $378,000,000 Friday, March 1, 13 The question is, could we have a crowd-CEO who would be worth as much as Tim Cook? And what would it take to make that possible?
- 28. {John, Jane, Joe, ...} Doe CEOs, Crowd $378,000,000 Friday, March 1, 13 If that doesn't seem that plausible, just think about how leadership can be distributed. Different people can bring different leadership skills to the table, with some providing feedback, some direcHon, some rewards and sancHons. Can we draw on what we know from shared leadership in online communiHes like wikipedia or open source soIware to develop dynamic and distributed leadership teams that together would be as valuable as a Hm cook.
- 29. CROWD WORK: hierarchy workflow Friday, March 1, 13
- 30. Workflow CrowdWeaver [Kittur et al. 2012] Find-Fix-Verify [Bernstein et al. 2010] TurKit [Little et al. 2009] CrowdForge [Kittur et al. 2011] Friday, March 1, 13 crowd work today looks like this. as researchers we’re trying to make more complex workflows, but it still only looks like this.
- 31. goal: as complex and high quality but faster and more dynamic than traditional workflows http://www.vsn-tv.com/blog/view/51-risks-in-ui-based-workflows-systems Friday, March 1, 13 This is a real workflow from how a tv broadcasting station generates their programming. how can we get to something like this? to do that we need crowdflows to be more robust, modular, and reusable
- 32. http://www.vsn-tv.com/blog/view/51-risks-in-ui-based-workflows-systems Friday, March 1, 13 ultimately, we might want to move towards shared libraries or design patterns so we can build up more and more complex workflows instead of every person creating workflows from scratch
- 33. CROWD COMPUTATION: platforms Friday, March 1, 13
- 34. Goal: crowd platforms replace firms Friday, March 1, 13 An audacious goal here would be for crowd platforms to replace firms as the default method for organizing work and workers. So could you create a flash startup that only exists for an afternoon, or a team of expert crowd workers who consult for thousands of jobs over a period of years.
- 35. Platforms determine work Friday, March 1, 13 How the platform is designed has a huge influence on the kind of work that eventually gets done. Given how Mechanical Turk was designed, it was almost inevitable that it would feel isolating, focus on small tasks, and lean toward low pay. But alternative platforms will create alternate outcomes.
- 36. Platforms • Create platforms that enable new kinds of work and new kinds of careers • There are problems that only a platform can solve: e.g., reliability of long-term employment, setting and achieving wage goals Friday, March 1, 13 Graduate students can do it: think about Google
- 37. CROWD WORKERS CROWD WORK CROWD COMPUTATION Friday, March 1, 13 In order to make this happen, we need to make advances in...
- 38. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Friday, March 1, 13 and we need to do it quickly. this picture is of my daughter when we started this paper. it’s now been a year ... and I don’t have too many of those left before she might join the workforce. There’s a LOT to be done between now and then, certainly more than I can do, certainly more than we can do.
- 39. WORKERS education & credentials motivation & rewards job design WORK hierarchy workflow task assignment realtime, synchronous work quality control COMPUTATION platform crowds guiding AIs AIs guiding crowds Friday, March 1, 13 and in fact in this talk we only covered a quarter of the challenges we raise in the paper, and there are many more ranging from motivation to quality control and the interaction between crowds and artificial intelligence. But we want to turn this over to you and ask: First, what’s missing here? What should we be aiming for, but we haven’t recognized yet? What would it take to get there? Second, what can YOU do to help? For any expertise that you bring to the table, there are lots of problems here that you can help with. We challenge you to pick one and have that impact by solving it. If you know qualitative methods well, you can help us think about motivation. If you’re technical, you can help us build this future. To close: crowd work does not have to be simple low-cost tasks. It can be the way that work is done in the future. But we need all of us to make that future one that our children will be proud to grow up in. Thanks.