The human face of AI: how collective and augmented intelligence can help solve societal problems
- 1. THE HUMAN FACE OF AI: HOW COLLECTIVE AND AUGMENTED INTELLIGENCE CAN HELP SOLVE SOCIETAL PROBLEMS Elena Simperl ACM-W UK, June 2020 @esimperl
- 2. AUGMENTED INTELLIGENCE Human-centred design paradigm for systems that utilise artificial intelligence (AI) People and AI work together to enhance cognitive performance, support decision making and create new experiences
- 3. AI DEPENDS ON PEOPLE Applications require more or better data e.g. from mobile or IoT devices Machine learning algorithms learn from human labellers Knowledge-based AI approaches acquire domain knowledge from people
- 4. AI BENEFITS FROM COLLECTIVE INTELLIGENCE Collective intelligence (CI) emerges when groups or communities come together, implicitly or explicitly, to achieve a common goal CI techniques help AI applications design and manage interactions with people In human computation a machine performs a function by outsourcing some steps to people
- 5. How do we design systems that bring together human, collective & computational intelligence
- 6. IN THIS TALK Design patterns for socio- technical systems Socio-technical challenges when defining and applying the patterns Directions for future research
- 7. EXAMPLE: SUPPORTING DISASTER RELIEF Human computation has provided huge advances to disaster relief efforts 40,000 independent reports mapped through Ushahidi after Haiti earthquake Crisis teams sift through large volumes of crowdsourced reports from social media and other sources Volunteer efforts predominantly limited to initial phase of recovery Human interest and effort often fails before later stages of process
- 9. TASK ALLOCATION EXPERIMENT Increase learning and engagement by ordering tasks by difficulty or similar content Public dataset of tweet URLs about hurricanes Harvey, Irma and Maria, curated manually to 2000 tweets, 1000 text-only, 1000 with images People were asked to classify tweets to help recovery teams process social media reports Recruitment via Amazon’s Mechanical Turk Labels train machine learning classifiers
- 10. TASK DESIGN Presented participants with disaster relief tweets (text or text + image) Participants asked to: Classify text based on content Rate task according to difficulty Three conditions: Random baseline Difficult tweets Easy tweets Monitored accuracy of responses
- 11. FINDINGS Accuracy influenced by difficulty Text: Weak association when comparing easy and difficult clusters Images: Strong association when comparing difficult and random clusters No significant association between difficulty and volume of completed tasks Only 30% of workers completed more than one task
- 12. FEEDBACK EXPERIMENT Two forms of feedback Expert feedback (using gold standard) Crowd feedback (randomly selected) Workflow: Participant gives answer Prompted with pre-existing answer and offered chance to edit Asked to explain decision Monitored decisions and justifications
- 13. FINDINGS Participants generally poor at taking feedback into account: 57% of workers felt expert feedback matched their responses (only 7% did) 36% of workers felt crowd feedback matched (only 4% did) Participants presented with crowd feedback more likely to change their answer in response (41% vs 26%) Also more likely to deem presented feedback from crowd as incorrect than from experts (22% vs 16%)
- 14. FUTURE WORK Difficulty impacts accuracy, but not engagement Participants struggled with more complex tasks Significant support required for maximum accuracy Generic feedback not sufficient – more personalised support required, resource-intensive
- 15. EXAMPLE: URBAN AUDITING ON DEMAND Urban datasets are often out-of- date Survey methodologies: expensive, error-prone, no validation VGI (e.g. OpenStreetMap): no control over data updates, coverage etc. Online tool using paid microtask crowdsourcing Uses digital street view imagery Task performed remotely Participants recruited from online marketplaces
- 16. VIRTUAL CITY EXPLORER QROWD-POI.HEROKUAPP.COM/ Urban planner defines an area and the instructions for the participants Participants explore an area virtually and identify points of interest Urban planner monitors task execution, quality and rewards
- 17. CHALLENGE: CROWDSOURCING DESIGN TASK DESIGN DATA QUALITY INCENTIVES FAIRNESS
- 18. EXPERIMENT: CYCLING TRENTO & NANTES 150 participants per city, random starting positions 5 PoIs (bike racks) per participant for $0.15 Total cost per city: $45 (7 days) Mixed methods approach, including metrics and manual inspection RQ1: Feasibility and precision as task progresses RQ2: Completeness (overlap with benchmark datasets) RQ3: Coverage (percentage of visited nodes on explorable path) RQ4: Crowd experience (interface errors triggered, number of escapes) Trento Nantes Area 0.347km2 0.336km2 Nodes 906 1177 Explorable distance 9127m 12104m StreetView coverage 93% 92%
- 19. RQ1: TASK FEASIBILITY AND PRECISION AS TASK PROGRESSES UX supports discovery of PoIs Photoshoot paradigm and triangulation method help identify low-quality answers Precision drops as all PoIs are submitted
- 20. RQ2: DATA COMPLETENESS Approach complements existing data sources and is able to find new PoIs Highly customisable (area of interest, budget, questions, timing) 52 54
- 21. RQ3: COVERAGE OF THE DESIGNATED AREA Approach achieves high coverage of the area of interest Some parts of the map are visited more often than others (resources) Black dots are points on StreetView that are difficult to explore
- 22. RQ4: CROWD EXPERIENCE Most participants were able to complete their tasks without any incidents. Some did not manage to triangulate or stepped outside of the designated area Positive feedback, fair payment, despite taboo mechanism. Small percentage submit some data and dropped out. Most participants who dropped out did not seem to try to complete the task
- 23. FINDINGS VCE adds value to urban auditing methods Accuracy comparable to OpenStreetMap, easier to manage than VGI Additional resources upon demand (at a cost) Free exploration achieves good coverage Taboo mechanism helps reduce costs and avoid duplicated work
- 24. FUTURE WORK Allocating starting positions: randomly, centre, to confirm item, to cover new area etc. Coordinating among participants: map showing progress of other participants Understanding the impact of urban topology on feasibility, accuracy, coverage Direct comparisons with other approaches Hybrid workflows with crowds on the ground and online
- 25. EXAMPLE: UNDERSTANDING MOBILITY PATTERNS City planners lack detailed mobility information about their residents Human-AI workflow Bespoke app for data collection Combination of symbolic and numerical ML classifiers to match trip segments to modes of transport Active learning approach to ask travellers to validate trips the machine is unsure about
- 26. CHALLENGE: USER EXPERIENCE Iterative UX development via citizen lab to improve journey data and ML predictions Lab and field studies with 250+ participants
- 27. CHALLENGE: ASSESSING THE QUALITY OF THE DATA Naïve model assumes people will notice and correct errors in journeys detected by the algorithm Is this true? If not, can we detect errors and estimate residual error rate? Are people employing specific ‘strategies’ to check and correct journeys?
- 28. EXPERIMENT DESIGN No independent ground truth! Inject artificial errors and measure if they are corrected Assume artificial errors are not accidental corrections Use ratio of discovered natural errors to discovered artificial errors to estimate initial and residual natural error rate Assume natural errors are comparable to artificial ones and people are not adding new errors (‘mis-corrections’)
- 29. EXPERIMENT DESIGN (2) 10 participants, ~5 journeys per participant, from Google Timelines (KML) Pre-process to add artificial errors in four classes: Under- or over-segmentation Bad mode Bad point (100/400m GPS point move) Score. Manual process, tool supported
- 30. PRELIMINARY FINDINGS: MORE RESEARCH NEEDED INTO DATA COLLECTION METHODOLOGIES FOR ML Errors can be corrected Errors can mislead Errors can persist A range of complex cases
- 31. How do we design systems that bring together human, collective & computational intelligence
- 32. Mix of CI approaches Iterative UX design Methods to assess data quality and improve human-AI interactions Aligned motivation and incentives
- 33. THANKS TO LUIS-DANIEL IBÁÑEZ, EDDY MADDALENA, RICHARD GOMER, NEAL REEVES, THE QROWD PROJECT, NESTA AND THE EUROPEAN COMMISSION @esimperl Maddalena, E., Ibáñez, L.D. and Simperl, E., 2020. On the mapping of Points of Interest through StreetView imagery and paid crowdsourcing. To appear in ACM TIST. qrowd-poi.herokuapp.com Nesta, June 2020. Combining Crowds and Machines: Experiments in collective intelligence design 1.0. nesta.org.uk/report/combining-crowds- and-machines/ Nesta, June 2020. Collective intelligence grants 1.0. nesta.org.uk/feature/collective-intelligence-grants/