![community Slack channel
we love the CRAs!!
Community is essential
community manager
runs worker support
builds platform documentation
community pulse surveys
promote the needs of the community
“Clara Remote Assistant” (CRA)
beta launches
CRAs love trying new tools before they roll out
we get amazing feedback
ask/answer questions, pictures of pets,
turtlebot, etc.
[realwaystoearnmoneyonline.com]
[workfromhomehappines.com]](https://image.slidesharecdn.com/jasonlaskaoreillyaipublic-170726005402/85/Strategies-for-integrating-people-and-machine-learning-in-online-systems-37-320.jpg)
Strategies for integrating people and machine learning in online systems
- 1. Strategies for integrating people and machine learning in online systems Jason Laska Ph.D. Machine Learning @ Clara Labs June 2017 presenting the work of Michael Akilian, Briana Burgess, Joey Carmello, Matthew Ebeweber, David Gouldin, Evan Hadfield, Olga Narvskaia, Maran Nelson, Jodi Nicolli, Emily Pitts, Gavin Schulz, Oliver Song @claralabswww.claralabs.com @chappaquack
- 3. Scheduling assistant Customer PreferencesEmail Coordination
- 4. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch
- 5. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Apply constraints: Lunch Lunch
- 6. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch Apply constraints: Location: Greenwich, CT
- 7. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch Apply constraints: Location: Greenwich, CT Coffee: 8am — Noon (preference)
- 8. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch Apply constraints: Location: Greenwich, CT Max Daily Meetings: 3 Coffee: 8am — Noon (preference) (preference)
- 9. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch ?Apply constraints: OOO: out of the office, what does that mean in this context? Apply NLP on calendar: Location: Greenwich, CT Max Daily Meetings: 3 Coffee: 8am — Noon (preference) (preference)
- 10. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch ? ?Apply constraints: OOO: out of the office, what does that mean in this context? Lunch: can we schedule over this or is it important? Apply NLP on calendar: Location: Greenwich, CT Max Daily Meetings: 3 Coffee: 8am — Noon (preference) (preference)
- 11. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” Location: Greenwich, CT Max Daily Meetings: 3 next week: OOO: out of the office, what does that mean in this context? Lunch: can we schedule over this or is it important? Apply NLP on calendar: Lunch Lunch ? ?Apply constraints: Relax: can relax constraints if there’s enough travel time? ? Coffee: 8am — Noon (preference) (preference)
- 12. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” Location: Greenwich, CT Max Daily Meetings: 3 next week: OOO: out of the office, what does that mean in this context? Lunch: can we schedule over this or is it important? Apply NLP on calendar: Lunch Lunch ? ?Apply constraints: Relax: can relax constraints if there’s enough travel time? ? Coffee: 8am — Noon (preference) (preference) ?
- 13. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NY Lunch Lunch ? ? ? graceful and intuitive edge-case handling customers really want
- 14. How Clara handles this example
- 15. How Clara handles this example
- 16. How Clara handles this example preference constraints participant availabilities/unavailabilities any accessible party calendars integrated with calendar
- 17. Breaking work into tasks “Let’s meet in Greenwich for coffee next week.” location: Greenwich channel: coffee time-pref: next week intent: schedule TASK TYPE: Predict & Annotate
- 18. TASK TYPE: Predict & Annotate fix incorrect predictions augment with missing parameters feedback loop to machine learning simple high precision rules (before feedback) after feedback detector only single parameter example Breaking work into tasks
- 19. TASK TYPE: Compute & Review location: Greenwich channel: coffee time-pref: next week intent: schedule state: new + action: suggest times Breaking work into tasks
- 20. TASK TYPE: Compute & Review check output makes sense check customer needs met feedback loop to product and engineering Breaking work into tasks
- 21. manual scheduling highly credentialed worker TASK TYPE: Manual Override Breaking work into tasks
- 22. Automation is a spectrum tightly integrate worker-operations, machine learning, and UX-design leverage task differences avg throughput by hour feb mar apr ~1.4xall sessions sessions with worker ~1.8x 1x tasks fully to partially automated match task difficulty with processing skill (person or machine) cost and speed gains without full automation requires
- 23. Distributing a knowledge-workforce constraints (requirements) challenges bounded processing time bounded processing cost people are naturally slow at data entry tasks people hours are more expensive than cpu cycles/hr workforce size learning the platform people are naturally noisy at data entry high accuracy bar “staffing on a dime”workforce elasticity
- 24. Worker lifecycle self-directed program build commitment to platform through promotion as increase credentials qualification sandbox beginner expert candidate worker sourcing exit platform • english comprehension • other testing • hours practice • exceeds threshold worker support customer support exit platform if accuracy drops below threshold for too long production
- 25. task aggregate & compare replicate state production sandbox — “scheduling multiverse” candidate 1 score candidate 2 score candidate n score worker candidates submit Sandbox environment
- 26. Task assignment hardeasiesttasks (most automated) (less automated)automatable
- 27. escalate escalate expert Task assignment beginner $$$$ hardeasiesttasks (most automated) (less automated)automatable cost worker pool queue
- 28. escalate escalate expert Task assignment beginner $$$$ hardeasiesttasks (most automated) (less automated)automatable cost worker pool queue
- 29. escalate escalate expert Task assignment beginner $$$$ hardeasiesttasks (most automated) (less automated)automatable cost worker pool queue
- 30. 2nd escalate easiest 1st escalate Work recycling recycle the first queue n times beginners may escalate when unsure (equivalent to “skip” in this case) # of recycles = proxy to task difficulty works well with incentive to avoid mistakes
- 31. 2nd escalate easiest 1st escalate other good strategies Work recycling “Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing” by N. Shah, D. Zhou recycle the first queue n times beginners may escalate when unsure (equivalent to “skip” in this case) # of recycles = proxy to task difficulty works well with incentive to avoid mistakes allow worker to “skip” tasks reward based on known-examples: no pay for skipping high penalty for incorrect labels high reward for correct labels incentivizes skipping if worker confidence is low
- 32. workforce size Distributing a knowledge-workforce constraints (requirements) challenges bounded processing time bounded processing cost workforce elasticity people are naturally slow at data entry tasks “staffing on a dime” people hours are more expensive than cpu cycles/hr learning the platform people are naturally noisy at data entry high accuracy bar
- 33. Incentives competing incentives drive both throughput and accuracy worker throughput worker accuracy per-task payment time preference (by ranked accuracy*) *workers must also maintain a minimum acuracy to remain on the platform
- 34. via turtlebot in community Slack channel Staffing: Supply and demand (workers) (customer requests) (all potential weekly supply) workers find out about “pick up” work requested hours predicted demand/hour worker reliability worker throughput worker accuracy when supply > demand: when supply << demand: hour assignments Jodi’s 3-pass assignment algorithm
- 35. Measuring accuracy: Mistake tracking worker peer review Worker Dashboard workers have access to previous annotations workers have task context function of time/cost constraints
- 36. Measuring accuracy: Mistake tracking other good strategies worker peer review Worker Dashboard 1) assign multiple workers to same task 2) estimate underlying annotation values and worker accuracy and biases a) jointly e.g., “Quality Management on Amazon Mechanical Turk” by Ipeirotis, Provost, & Wang b) via Cohen/Fleiss kappa: … many other ways := 1 1 pobserved agreement 1 pchance agreement workers have access to previous annotations workers have task context function of time/cost constraints
- 37. community Slack channel we love the CRAs!! Community is essential community manager runs worker support builds platform documentation community pulse surveys promote the needs of the community “Clara Remote Assistant” (CRA) beta launches CRAs love trying new tools before they roll out we get amazing feedback ask/answer questions, pictures of pets, turtlebot, etc. [realwaystoearnmoneyonline.com] [workfromhomehappines.com]
- 38. kinds of people we’re looking for Come work with us! @claralabswww.claralabs.com modeling email requests combining ML & UX for data-entry systems natural language understanding fullstack & frontend engineers machine learning engineers come talk to me! we work in python/flask, react/redux, aws services, sklearn, keras… jason@claralabs.com @chappaquack software problems design problems ML problems continuous model train and integration worker task management defining new Clara capabilities