Trust Evaluation through User Reputation and Provenance Analysis
2 likes670 views
This document proposes combining user reputation and provenance analysis to evaluate trust in crowdsourced video annotations. It outlines using a beta distribution to model user reputation based on past annotation matches. Provenance-based trust is estimated using support vector machines on annotation metadata like timestamp. Results found reputation provides a more fine-grained trust estimate than provenance alone, but provenance can estimate trust when reputation data is limited. Future work aims to apply the approach to other domains and develop a generic trust assessment platform based on annotated provenance graphs.
Trust Evaluation through User Reputation and Provenance Analysis
- 1. Trust Evaluation through User Reputation and Provenance Analysis Davide Ceolin, Paul Groth, Willem Robert van Hage, Archana Nottamkandath, Wan Fokkink
- 2. Outline ● Problem Definition ● Trust estimates as probability distributions ● Reputation-based trust ● Provenance-based trust ● Combining reputation- and provenance-trust ● Results ● Future Work
- 3. Our Problem Trusting crowdsourced information Specifically: video annotations We can not tag videos automatically (complex, many layers,...), so we refer to the crowd, but crowdsourced annotations need to be evaluated.
- 4. Our Case Study Waisda, a video tagging platform. http://waisda.nl Users score if they tag simultaneously. Tags are useful (e.g. to index video) but how to trust tags?
- 5. Measuring tags trustworthiness How to measure tags trustworthiness? Let's exploit the game mechanism: #matches ≈ trustworthiness However, if a tag did not get any match it is not necessarily wrong.
- 6. Trust as a Beta Distribution
- 7. Reputation-based trust A classical approach: ● Gather all the evidence about a user ● Estimate the Beta distribution (user reputation) ● Predict the trustworthiness of the (other) tags inserted by the user (using the Beta)
- 8. Our hypothesis We believe we can go beyond the classical reputation-based trust: Who ≤ Who + How
- 9. (How) Provenance-based trust ● “How”-provenance can be used as a stereotype for tags authorship ● May be less precise than reputation but more easily available
- 10. Computing provenance-based trust “How-provenance” is composed by several “features” (timestamp, typingDuration, ...) We use Support Vector Machines to learn the model and make our estimates.
- 11. Combining reputation- and provenance-trust Is reputation "solid" enough? (E.g. based on 20 obs.) Use Reputation! Use how-provenance-based (More fine-grained) estimate!
- 12. Results #{tag: Real_Trust(tag)>Threshold = Pred_Trust(tag) > Threshold} Accuracy = #tags
- 13. Discussion Who =* How Who ≤* Who + How How ≤* Who + How (At least in our case study) =* statistically not different ≤* statistically different (less)
- 14. Future Work ● We plan to apply this approach to other case studies as well ● A generic platform for producing trust assessments based on annotated provenance graphs is under development ● We also plan to merge provenance-based estimates with semantic similarity-based ones in order to provide a tool for a wide range of applications.