Exploiting disagreement through open ended tasks for capturing interpretation spaces
1 like5,225 views
The document discusses using open-ended crowdsourcing tasks to capture interpretation spaces for multimedia data. It outlines challenges with existing approaches that assume a single ground truth and stimulate agreement. The methodology proposes using open-ended tasks without predefined answers to gather a range of interpretations. Preliminary results show capturing diverse sound interpretations through open-ended tasks to tag sounds. The conclusion is that open-ended tasks do not force agreement but can capture the full interpretation space for more complete multimedia data.
Exploiting disagreement through open ended tasks for capturing interpretation spaces
- 1. Exploiting disagreement through open- ended tasks for capturing interpretation spaces Doctoral Consortium By /Benjamin Timmermans @8w
- 2. Outline Introduction State of the Art Problem Statement Methodology Preliminary Results Conclusions
- 3. Introduction
- 5. How many dogs were in the picture?
- 6. There is no universal "truth"
- 7. For the training, testing and evaluation of machines we rely on a... ground "truth"
- 8. State of the Art
- 9. Crowdsourcing Approach 1-3 annotators Evaluate workers Inner-annotator agreement Use test questions Predefined answer choices
- 11. The CrowdTruth Approach 10-15 annotators Evaluate the input, annotations and workers Disagreement-based analytics
- 13. Problems with multimedia annotations Are sparse Are homogeneous Do not represent everything that can be heard or seen
- 14. Problems with crowdsourcing tasks Are designed to stimulate agreement Assumes answers are right or wrong
- 15. Closed task How many beams do you see? 1 2 3 4 5 1 1 2 3 4 5
- 16. Open- ended tasks How many beams do you see?
- 18. Gathering the interpretation space of multimedia through open-ended crowdsourcing tasks Goal More efficient crowdsourcing Higher quality ground truth data Improved search and discovery of multimedia
- 19. Are open-ended crowdsourcing tasks a feasible method for capturing the interpretation space of multimedia? Research Question
- 20. Methodology
- 21. 1. Improving quality evaluation Comparing Closed and open-ended tasks Measure worker confidence
- 22. 2. Improving open- ended task design Combine constrains with open-ended designs Showing known annotations Detecting the distribution of answers
- 23. 3. Applying the ground "truth" Compare different contexts Improve indexing of multimedia
- 25. Gathering training data for IBM Watson
- 26. Range of tasks Passage Justification Passage Alignment Distributional disambiguation
- 27. Sound Interpretations 2.133 short sounds Top 5000 search terms = 11 mil. searches
- 29. Conclusions There is no ultimate "truth" Do not stimulate agreement Capture the interpretation space Use open-ended crowdsourcing tasks Evaluation more difficult
- 30. Who we are Lora Aroyo Robert-Jan Sips Chris Welty Oana Inel Anca Dumitrache Benjamin Timmermans
- 31. Acknowledgements Supervisor: Dr. Lora Aroyo Mentor: Dr. Matteo Palmonari