INSEMTIVES Tutorial ISWC2011 - Session1
0 likes355 views
This document discusses ways to motivate users to contribute semantic content through incentives and gamification. It provides examples of semantic authoring tasks that can be crowdsourced or turned into games. Guidelines are presented for designing incentive structures and game mechanics to encourage ongoing user engagement with semantic tasks. The goal is to help semantic technologies reach critical mass by involving millions of end users in content creation.
INSEMTIVES Tutorial ISWC2011 - Session1
- 1. Ten ways to make your semantic app addicted - REVISITED Elena Simperl Tutorial at the ISWC2011, Bonn, Germany 10/24/2011 www.insemtives.eu 1
- 2. Executive summary • Many aspects of semantic content authoring naturally rely on human contribution. • Motivating users to contribute is essential for semantic technologies to reach critical mass and ensure sustainable growth. • This tutorial is about – Methods and techniques to study incentives and motivators applicable to semantic content authoring scenarios. – How to implement the results of such studies through technology design, usability engineering, and game mechanics. www.insemtives.eu 2
- 3. Incentives and motivators • Motivation is the driving • Incentives can be related force that makes humans to both extrinsic and achieve their goals. intrinsic motivations. • Incentives are ‘rewards’ • Extrinsic motivation if assigned by an external task is considered boring, ‘judge’ to a performer for dangerous, useless, undertaking a specific socially undesirable, task. dislikable by the – Common belief (among performer. economists): incentives • Intrinsic motivation is can be translated into a sum of money for all driven by an interest or practical purposes. enjoyment in the task itself.
- 4. Examples of applications www.insemtives.eu 4
- 5. Extrinsic vs intrinsic motivations • Successful volunteer crowdsourcing is difficult to predict or replicate. – Highly context-specific. – Not applicable to arbitrary tasks. • Reward models often easier to study and control.* – Different models: pay-per-time, pay-per-unit, winner- takes-it-all… – Not always easy to abstract from social aspects (free- riding, social pressure…). – May undermine intrinsic motivation. * in cases when performance can be reliably measured
- 6. Examples (ii) Mason & Watts: Financial incentives and the performance of the crowds, HCOMP 2009.
- 7. Amazon‘s Mechanical Turk • Types of tasks: transcription, classification, and content generation, data collection, image tagging, website feedback, usability tests.* • Increasingly used by academia. • Vertical solutions built on top. • Research on extensions for complex tasks. * http://behind-the-enemy-lines.blogspot.com/2010/10/what-tasks-are-posted-on-mechanical.html
- 8. Tasks amenable to crowdsourcing • Tasks that are decomposable into simpler tasks that are easy to perform. • Performance is measurable. • No specific skills or expertise are required.
- 9. Patterns of tasks* • Solving a task • Example: open-scale tasks – Generate answers in Mturk – Find additional information – Generate, then vote. – Improve, edit, fix – Introduce random noise to • Evaluating the results of a identify potential issues in the second step task – Vote for accept/reject Label Correct Vote answers Generate answer – Vote up/down to rank potentially correct answers image or not? – Vote best/top-n results • Flow control – Split the task – Aggregate partial results * „Managing Crowdsourced Human Computation“@WWW2011, Ipeirotis
- 10. Examples (iii) www.insemtives.eu 10
- 11. What makes game mechanics successfull?* • Accelerated feedback cycles. – Annual performance appraisals vs immediate feedback to maintain engagement. • Clear goals and rules of play. – Players feel empowered to achieve goals vs fuzzy, complex system of rules in real-world. • Compelling narrative. – Gamification builds a narrative that engages players to participate and achieve the goals of the activity. • But in the end it’s about what task users want to get better at. *http://www.gartner.com/it/page.jsp?id=1629214 Images from http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/ and http://www.hideandseek.net/wp- content/uploads/2010/10/gamification_badges.jpg
- 12. Guidelines • Focus on the actual goal and incentivize related actions. – Write posts, create graphics, annotate pictures, reply to customers in a given time… • Build a community around the intended actions. – Reward helping each other in performing the task and interaction. – Reward recruiting new contributors. • Reward repeated actions. – Actions become part of the daily routine. Image from http://t1.gstatic.com/images?q=tbn:ANd9GcSzWEQdtagJy6lxiR2focH2D01Wpz7dzAilDuPsWnL0i4GAHgnm_0hyw3upqw
- 13. What tasks can be gamified?* • Tasks that are decomposable into simpler tasks, nested tasks. • Performance is measurable. • Obvious rewarding scheme. • Skills can be arranged in a smooth learning curve. *http://www.lostgarden.com/2008/06/what-actitivies-that-can-be-turned-into.html Image from http://www.powwownow.co.uk/blog/wp-content/uploads/2011/06/gamification.jpeg
- 14. What is different about semantic systems? • It‘s still about the context of the actual application. • User engagement with semantic tasks in order to – Ensure knowledge is relevant and up-to-date. – People accept the new solution and understand its benefits. – Avoid cold-start problems. – Optimize maintenance costs.
- 15. Tasks in knowledge engineering • Definition of vocabulary • Conceptualization – Based on competency questions – Identifying instances, classes, attributes, relationships • Documentation – Labeling and definitions. – Localization • Evaluation and quality assurance – Matching conceptualization to documentation • Alignment • Validating the results of automatic methods www.insemtives.eu 15
- 17. OntoGame API • API that provides several methods that are shared by the OntoGame games, such as: – Different agreement types (e.g. selection agreement). – Input matching (e.g. , majority). – Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal partner to play). – Resource (i.e., data needed for games) management. – Creating semantic content. • http://insemtives.svn.sourceforge.net/vie wvc/insemtives/generic-gaming-toolkit 10/24/2011 www.insemtives.eu 17
- 18. OntoGame games 10/24/2011 www.insemtives.eu 18
- 19. Case studies • Methods applied – Mechanism design. – Participatory design. – Games with a purpose. – Crowdsourcing via MTurk. • Semantic content authoring scenarios – Extending and populating an ontology. – Aligning two ontologies. – Annotation of text, media and Web APIs.
- 20. Lessons learned • Approach is feasible for mainstream domains, where a (large-enough) knowledge corpus is available. • Advertisement is important. • Game design vs useful content. – Reusing well-kwown game paradigms. – Reusing game outcomes and integration in existing workflows and tools. • But, the approach is per design less applicable because – Knowledge-intensive tasks that are not easily nestable. – Repetitive tasks players‘ retention? • Cost-benefit analysis.
- 21. Using Mechanical Turk for semantic content authoring • Many design decisions similar to GWAPs. – But clear incentives structures. – How to reliably compare games and MTurk results? • Automatic generation of HITs depending on the types of tasks and inputs. • Integration in productive environments. – Protégé plug-in for managing and using crowdsourcing results.
- 22. Outline of the tutorial Time Presentation 14:00 – Human contributions in semantic content authoring 14:45 14:45 – Case study: motivating employees to annotate enterprise 15:30 content semantically at Telefonica 15:30 – Coffee break 16:00 16:00 – Case study: Crowdsourcing the annotation of dynamic Web 16:45 content at seekda 16:45 – Case study: Content tagging at MoonZoo and 17:30 MyTinyPlanets 17:30 – Ten ways to make your semantic app addicted - revisited 18:00 www.insemtives.eu 22
- 23. Realizing the Semantic Web by encouraging millions of end-users to create semantic content. 10/24/2011 www.insemtives.eu 23