Insemtives iswc2011 session1
- 1. Ten ways to make your semantic app addicted - REVISITED Elena Simperl Tutorial at the ISWC2011, Bonn, Germany 10/24/2011 www.insemtives.eu 1
- 2. Executive summary • Many aspects of semantic content authoring naturally rely on human contribution. • Motivating users to contribute is essential for semantic technologies to reach critical mass and ensure sustainable growth. • This tutorial is about – Methods and techniques to study incentives and motivators applicable to semantic content authoring scenarios. – How to implement the results of such studies through technology design, usability engineering, and game mechanics. www.insemtives.eu 2
- 3. Incentives and motivators • Motivation is the driving • Incentives can be related force that makes humans to both extrinsic and achieve their goals. intrinsic motivations. • Incentives are ‘rewards’ • Extrinsic motivation if assigned by an external task is considered boring, ‘judge’ to a performer for dangerous, useless, undertaking a specific socially undesirable, task. dislikable by the – Common belief (among performer. economists): incentives • Intrinsic motivation is can be translated into a sum of money for all driven by an interest or practical purposes. enjoyment in the task itself.
- 4. Examples of applications www.insemtives.eu 4
- 5. Extrinsic vs intrinsic motivations • Successful volunteer crowdsourcing is difficult to predict or replicate. – Highly context-specific. – Not applicable to arbitrary tasks. • Reward models often easier to study and control.* – Different models: pay-per-time, pay-per-unit, winner- takes-it-all… – Not always easy to abstract from social aspects (free- riding, social pressure…). – May undermine intrinsic motivation. * in cases when performance can be reliably measured
- 6. Examples (ii) Mason & Watts: Financial incentives and the performance of the crowds, HCOMP 2009.
- 7. Amazon‘s Mechanical Turk • Types of tasks: transcription, classification, and content generation, data collection, image tagging, website feedback, usability tests.* • Increasingly used by academia. • Vertical solutions built on top. • Research on extensions for complex tasks. * http://behind-the-enemy-lines.blogspot.com/2010/10/what-tasks-are-posted-on-mechanical.html
- 8. Tasks amenable to crowdsourcing • Tasks that are decomposable into simpler tasks that are easy to perform. • Performance is measurable. • No specific skills or expertise are required.
- 9. Patterns of tasks* • Solving a task • Example: open-scale tasks – Generate answers in Mturk – Find additional information – Generate, then vote. – Improve, edit, fix – Introduce random noise to • Evaluating the results of a identify potential issues in the second step task – Vote for accept/reject Label Correct Vote answers Generate answer – Vote up/down to rank potentially correct answers image or not? – Vote best/top-n results • Flow control – Split the task – Aggregate partial results * „Managing Crowdsourced Human Computation“@WWW2011, Ipeirotis
- 10. Examples (iii) www.insemtives.eu 10
- 11. What makes game mechanics successfull?* • Accelerated feedback cycles. – Annual performance appraisals vs immediate feedback to maintain engagement. • Clear goals and rules of play. – Players feel empowered to achieve goals vs fuzzy, complex system of rules in real-world. • Compelling narrative. – Gamification builds a narrative that engages players to participate and achieve the goals of the activity. • But in the end it’s about what task users want to get better at. *http://www.gartner.com/it/page.jsp?id=1629214 Images from http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/ and http://www.hideandseek.net/wp- content/uploads/2010/10/gamification_badges.jpg
- 12. Guidelines • Focus on the actual goal and incentivize related actions. – Write posts, create graphics, annotate pictures, reply to customers in a given time… • Build a community around the intended actions. – Reward helping each other in performing the task and interaction. – Reward recruiting new contributors. • Reward repeated actions. – Actions become part of the daily routine. Image from http://t1.gstatic.com/images?q=tbn:ANd9GcSzWEQdtagJy6lxiR2focH2D01Wpz7dzAilDuPsWnL0i4GAHgnm_0hyw3upqw
- 13. What tasks can be gamified?* • Tasks that are decomposable into simpler tasks, nested tasks. • Performance is measurable. • Obvious rewarding scheme. • Skills can be arranged in a smooth learning curve. *http://www.lostgarden.com/2008/06/what-actitivies-that-can-be-turned-into.html Image from http://www.powwownow.co.uk/blog/wp-content/uploads/2011/06/gamification.jpeg
- 14. What is different about semantic systems? • It‘s still about the context of the actual application. • User engagement with semantic tasks in order to – Ensure knowledge is relevant and up-to-date. – People accept the new solution and understand its benefits. – Avoid cold-start problems. – Optimize maintenance costs.
- 15. Tasks in knowledge engineering • Definition of vocabulary • Conceptualization – Based on competency questions – Identifying instances, classes, attributes, relationships • Documentation – Labeling and definitions. – Localization • Evaluation and quality assurance – Matching conceptualization to documentation • Alignment • Validating the results of automatic methods www.insemtives.eu 15
- 17. OntoGame API • API that provides several methods that are shared by the OntoGame games, such as: – Different agreement types (e.g. selection agreement). – Input matching (e.g. , majority). – Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal partner to play). – Resource (i.e., data needed for games) management. – Creating semantic content. • http://insemtives.svn.sourceforge.net/vie wvc/insemtives/generic-gaming-toolkit 10/24/2011 www.insemtives.eu 17
- 18. OntoGame games 10/24/2011 www.insemtives.eu 18
- 19. Case studies • Methods applied – Mechanism design. – Participatory design. – Games with a purpose. – Crowdsourcing via MTurk. • Semantic content authoring scenarios – Extending and populating an ontology. – Aligning two ontologies. – Annotation of text, media and Web APIs.
- 20. Lessons learned • Approach is feasible for mainstream domains, where a (large-enough) knowledge corpus is available. • Advertisement is important. • Game design vs useful content. – Reusing well-kwown game paradigms. – Reusing game outcomes and integration in existing workflows and tools. • But, the approach is per design less applicable because – Knowledge-intensive tasks that are not easily nestable. – Repetitive tasks players‘ retention? • Cost-benefit analysis.
- 21. Using Mechanical Turk for semantic content authoring • Many design decisions similar to GWAPs. – But clear incentives structures. – How to reliably compare games and MTurk results? • Automatic generation of HITs depending on the types of tasks and inputs. • Integration in productive environments. – Protégé plug-in for managing and using crowdsourcing results.
- 22. Outline of the tutorial Time Presentation 14:00 – Human contributions in semantic content authoring 14:45 14:45 – Case study: motivating employees to annotate enterprise 15:30 content semantically at Telefonica 15:30 – Coffee break 16:00 16:00 – Case study: Crowdsourcing the annotation of dynamic Web 16:45 content at seekda 16:45 – Case study: Content tagging at MoonZoo and 17:30 MyTinyPlanets 17:30 – Ten ways to make your semantic app addicted - revisited 18:00 www.insemtives.eu 22
- 23. Realizing the Semantic Web by encouraging millions of end-users to create semantic content. 10/24/2011 www.insemtives.eu 23