A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing
2 likes855 views
The document discusses a capability requirements approach for predicting worker performance in crowdsourcing, emphasizing the importance of effectively assigning tasks to workers based on their capabilities. It introduces a capabilities taxonomy and capability tracing model to evaluate and predict worker performance on various micro-tasks. Future work aims to expand evaluations across more task types and develop standard tests for measuring worker capabilities.
A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing
- 1. Digital Enterprise Research Institute www.deri.ie A CAPABILITY REQUIREMENTS APPROACH FOR PREDICTING WORKER PERFORMANCE IN CROWDSOURCING Umair ul Hassan, Edward Curry Digital Enterprise Research Institute National University of Ireland, Galway 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing Austin, Texas, United States October 20–23, 2013 Copyright 2010 Digital Enterprise Research Institute. All rights reserved.
- 2. Agenda Digital Enterprise Research Institute Motivation Background Task Modelling Capability Requirements www.deri.ie Capabilities Taxonomy Capability Tracing Experiment Summary 2
- 3. Motivation: Heterogeneity Digital Enterprise Research Institute www.deri.ie 3
- 4. Motivation: Task Routing Digital Enterprise Research Institute www.deri.ie Assigning heterogeneous tasks to heterogeneous workers TASK MODELLING Models Models Models TASK ROUTING WORKER PROFILING Matching Profiles Profiles Profiles Task↔Worker 4
- 5. Proposal: Performance Prediction Digital Enterprise Research Institute www.deri.ie Predict performance of workers on new tasks based on the capabilities required for tasks and assign tasks accordingly TASK MODELLING Models Models Models Capability Requirements Approach TASK ROUTING WORKER PROFILING Matching Profiles Profiles Profiles Task↔Worker Performance Prediction Capability Tracing Model 5
- 6. Background: Micro tasks Digital Enterprise Research Institute www.deri.ie When micro tasks are crowd sourced Single person cannot do the task Computers cannot do the task Work can be split into smaller tasks Some online microtask platforms 6
- 7. Background: Micro tasks Digital Enterprise Research Institute www.deri.ie Most common tasks in Amazon Mechanical Turk (AMT) and CrowdFlower (CFL) 7
- 8. Background: Micro tasks Digital Enterprise Research Institute www.deri.ie Example of information extraction task in AMT 8
- 9. Background: Micro tasks Digital Enterprise Research Institute www.deri.ie Example of video transcription task in AMT 9
- 10. Task Modelling Digital Enterprise Research Institute www.deri.ie Appropriate models are needed to compare and contrast micro tasks. Capability Requirements approach Capability is defined as the ability of humans to do things in terms of both the capacity and the opportunity. Four types of capabilities – – – – Knowledge, Skill, Ability, Other characteristics (e.g. motivation, price, etc) 10
- 11. Capability Requirements Digital Enterprise Research Institute www.deri.ie Taxonomies have be used to study human task performance, e.g. Bloom’s taxonomy of classification of learning objectives Fleishman’s taxonomy of human abilities O*NET-SOC taxonomy of occupational classification We are interested in taxonomy that Describes tasks in terms of human capabilities Helps in comparing tasks in terms of differences and similarities of capabilities 11
- 12. Capabilities Taxonomy Digital Enterprise Research Institute www.deri.ie Based on Fleishman’s abilities taxonomy Selected abilities relevant to micro tasks Comprehension (C): The ability to understand the meaning or importance of something Bilingualism (B): The ability to speak and understand two languages Writing (W): The ability or capacity to write text in a given language Comparison (M): The ability or capacity to compare things based on some criteria Judgment (J): The act or process of judging; the formation of an opinion after consideration Perception (P): The ability or capacity to perceive items visually or phonetically Identification (I): The process of recognizing something Reasoning (R): The ability to draw conclusions from facts, evidence, relationships, etc. 12
- 13. Requirements of Micro Tasks Digital Enterprise Research Institute www.deri.ie 13
- 14. Capability Tracing Digital Enterprise Research Institute www.deri.ie How to model worker’s capabilities? Capability tracing Inspired by Knowledge Tracing* Estimates probability of a worker knowing a capability given worker’s responses to test tasks Worker Profile constrains Set of binary variables representing capabilities Probability estimates of each variable being in a state * A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,” User Modeling and User-Adapted Interaction, vol. 4, no. 4, pp. 253–278, 1994. 14
- 15. Capability Tracing Digital Enterprise Research Institute www.deri.ie Probabilistic network of a capability and four parameters of capability tracing model States of Capability Variable Not Learned p(T): Probability of transition between states p(T) Learned p(L) p(G) Values of Response Variable p(L): Probability of a worker learning to employ the capability p(S) p(G): Probability of guess Correct Incorrect 15 p(S): Probability of slip
- 16. Experiment Digital Enterprise Research Institute www.deri.ie Objective Solicit capability requirements of tasks from crowds Evaluation of capability tracing for performance prediction Three types of micro tasks with manually created ground truth data Image comparison Fact verification Information Extraction 37 crowd workers including University students Workers from Shorttask.com 16
- 17. Crowdsourcing Digital Enterprise Research Institute www.deri.ie Custom web application for gathering data Example of fact verification task 17
- 18. Capability Requirements of Tasks Digital Enterprise Research Institute www.deri.ie Objective 1: Solicit capability requirements of tasks from crowds (a) fact verification (b) image comparison 18 (c) information extraction
- 19. Crowd Performance Digital Enterprise Research Institute www.deri.ie How the crowd performed on each type of task? Fact Verification task • 37 workers • Best workers perform with both precision and recall above 0.8 • More variation in recall means some workers were could not spot the incorrect facts • Ideally tasks should be assigned to workers that lie in the top-right quadrant of the plot 19
- 20. Crowd Performance Digital Enterprise Research Institute www.deri.ie Image Comparison (20 workers) and Information Extraction (17 workers) 20
- 21. Performance Prediction Digital Enterprise Research Institute www.deri.ie Objective 2: Evaluation of capability tracing for performance prediction Two phases Build model with observation tasks Predict performance on new tasks AC: Consider previous Accuracy as prediction of future performance CT: Capability Tracing 21
- 22. Summary Digital Enterprise Research Institute Capabilities taxonomy is first steps towards modelling of micro tasks based on human factors Capability tracing is effective in predicting future performance www.deri.ie Even across tasks if there are similar capabilities Predicted performance can be used to make right task routing decisions Future Work Evaluate on more types of tasks Evaluate capabilities such as domain knowledge and skills Define standard tests for measuring worker capabilities 22
- 23. Further Reading Digital Enterprise Research Institute www.deri.ie 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing Austin, Texas, United States October 20–23, 2013 U. Ul Hassan and E. Curry, “A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing,” in 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, 2013. http://deri.ie/users/umair-ul-hassan 23
- 24. Capability Tracing Digital Enterprise Research Institute www.deri.ie Conditional probability of worker learning to employ capability p(Ln|On) is calculated When evidence On is positive When evidence On is negative 24
- 25. Capability Tracing Digital Enterprise Research Institute www.deri.ie Probability of worker learning to employ capability Performance of worker on next task 25
Editor's Notes
- #7: AMT is amazonmechincalturk
- #8: AMT is amazonmechincalturk
- #9: AMT is amazonmechincalturk
- #10: AMT is amazon mechanical turk
- #15: Probability estimates are generated while learning the model through capability tracing
- #19: As can be seen, more than majority of workers believed that identification capability is essential for all three types of tasks. Majority of workers also agreed that the judgment and comprehension capabilities are important for the Fact Verification task. In the case of the Image Comparison task most workers specified that comparison and perception are important as well. There is general consensus between workers that comprehension, judgment and reasoning capabilities are also useful for Information Extraction tasks. We selected top-3 capabilities for each type of task for building capability tracing models.
- #20: Interestingly, no worker achieved the highest recall for the information extraction task, which highlights the difference between workers and ground truth in terms of the entities extracted from the Wikipedia articles. Nevertheless, these distributions emphasize that in order to achieve high accuracy tasks should be assigned to workers that lie in the top-right quadrant of the plots.
- #22: Results show that the capability tracing approach is comparable to the baseline approach in general and achieves better accuracy of prediction between similar tasks. The Fact Verification and Information Extraction tasks have similar capabilities requirements, therefore capability tracingcan better predict the performance of workers between them. The drop in prediction quality of capability tracing for Image Comparison task can be attributed to the little variation is the performance of workers on this task.
- #25: Hide these
- #26: Hide these