Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017
2 likes390 views
This document summarizes Alexey Grigorev's approach to detecting large scale vandalism in knowledge bases like Wikidata for the WSDM Cup 2017 competition on vandalism detection. The competition task was to predict whether a Wikidata revision should be rolled back or not based on features from the revision. Grigorev's best performing model used a linear SVM on a hashed one-hot encoding of combined features from the revision including user, title, and comment features. This approach achieved an AUC of around 0.96 on the competition test set.
![Comment Features
● Structured” part: inside /* */ - split on “:” and “|”:
○ wbsetdescription-add 1 es
○ wbsetsitelink-add 1 idwiki
○ clientsitelink-remove 1 frwiki
● Wiki-Links: [[Property:P31]], [[Q5]]
● Free text: outside of /* */:
○ futbolista irlandes
○ autolist2
○ origyn web browser
/* wbsetdescription-add:1|es */ futbolista irlandes
/* wbcreateclaim-create:1| */ [[Property:P31]]: [[Q5]], #autolist2
/* wbsetsitelink-add:1|idwiki */ Megaloharpya
/* clientsitelink-remove:1||frwiki */ Origyn Web Browser
<subject predicate object>](https://image.slidesharecdn.com/wsdmcup-pydata-170701201506/85/Large-Scale-Vandalism-Detection-in-Knowledge-Bases-PyData-Berlin-2017-13-320.jpg)
Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017
- 1. Large Scale Vandalism Detection in Knowledge Bases by Alexey Grigorev ods.ai
- 2. About Me Software Engineer BI Masters @ TU Berlin Data Scientist
- 3. Knowledge Bases <subject predicate object> <ML partOf CS>
- 5. WSDM Cup 2017: Vandalism Detection ● http://www.wsdm-cup-2017.org/vandalism-detection.html ● Goal of the competition: ● Predict if a Wikidata revision should be rolled back or not ● Not a usual “kaggle-like” competition: ○ Code to be executed within a VM ○ Read data from socket, predict, write back ○ Hardware: 1 core, 4 GB of RAM
- 7. Dataset ● Wikidata XML dump with edits ○ From 2012 to 2016, 72 mln revisions ○ 24.8 Gb compressed, 400 Gb uncompressed ● Very skewed: ○ 127k rollbacks (out of 72m) ○ i.e. 0.0025 - fraction of positives
- 8. Validation Train 2012-10-29 to 2016-02-29 65 mln Validation 2016-03-01 to 2016-04-30 7.2 mln Test from 2016-05-01 10.4 mln Train 2015-01-01 to 2015-31-31 27.9 mln Validation 2016-01-01 to 2016-02-29 9.3 mln Test 2016-03-01 to 2016-04-30 7.2 mln ← Not available ← Available from start ← Available towards end
- 9. <page> <title>Q5704066</title> <ns>0</ns> <id>5468191</id> <revision> <id>185142906</id> <parentid>185051367</parentid> <timestamp>2015-01-01</timestamp> <contributor> <username>username</username> <id>52267</id> </contributor> <comment>/* wbsetdescription-add:1|es */ futbolista irlandes</comment> <model>wikibase-item</model> <format>application/json</format> <text xml:space="preserve">{{PAGE_JSON}}</text> </revision> </page> IP if anonymous & geo info in meta file
- 10. <page> <title>Q5704066</title> <ns>0</ns> <id>5468191</id> <revision> <id>185142906</id> <parentid>185051367</parentid> <timestamp>2015-01-01</timestamp> <contributor> <username>username</username> <id>52267</id> </contributor> <comment>/* wbsetdescription-add:1|es */ futbolista irlandes</comment> <model>wikibase-item</model> <format>application/json</format> <text xml:space="preserve">{{PAGE_JSON}}</text> </revision> </page> Title features User features JSON (not used) Comment features
- 11. User Features ● Username, if logged in, if not “anonymous=True” ● IP: 90.219.230.105 → ○ 90, 90_219, 90_219_230, 90_219_230_105 ● Geo information from the meta file: ○ country_code=GB ○ continent_code=EU ○ time_zone=GMT ○ regio_code=EN ○ city_name=LEEDS ○ county_name=WEST_YORKSHIRE ● Use One Hot Encoding
- 12. User Features ● OHE: put this together as one string ○ anonymous 90 90_219 90_219_230 90_219_230_105 country_code=GB … ○ username ○ Use CountVectorizer from sklearn to get a matrix
- 13. Comment Features ● Structured” part: inside /* */ - split on “:” and “|”: ○ wbsetdescription-add 1 es ○ wbsetsitelink-add 1 idwiki ○ clientsitelink-remove 1 frwiki ● Wiki-Links: [[Property:P31]], [[Q5]] ● Free text: outside of /* */: ○ futbolista irlandes ○ autolist2 ○ origyn web browser /* wbsetdescription-add:1|es */ futbolista irlandes /* wbcreateclaim-create:1| */ [[Property:P31]]: [[Q5]], #autolist2 /* wbsetsitelink-add:1|idwiki */ Megaloharpya /* clientsitelink-remove:1||frwiki */ Origyn Web Browser <subject predicate object>
- 14. Models ● High dimensionality? Sparse? Linear SVM! ● LinearSVC, dual=False, penalty=L1
- 15. Ensembling ● Usual “kaggle” way to improve: ● Ensembling and stacking ● Not successful - user model still better on test
- 16. Imbalance ● The dataset is highly skewed: 0.0025 positive ● Ways to deal with it: ○ Oversampling and undersampling ● Not helpful
- 17. Final Model ● Put all features together as one string ● title=Q123 username=user wbsetdescription-add 1 es P31 Q5 … ● OHE? ○ CountVectorizer - dictionary too large ○ HashingVectorizer with 10m columns
- 18. Hashing for One Hot Encoding CountVectorizer HashingVectorizer
- 19. Final Model ● OHE with HashingVectorizer (no memory) ● SVM on OHE matrix (300 mb) ● ~ 0.96 AUC on my test
- 20. Final Results That’s me :-)
- 21. Conclusions Lessons Learned: ● Trust your CV ● Sometimes ensembling does not work ● Prefer simpler models to avoid overfitting New tools I tried: ● Feather - very fast!
- 23. Links and Further Info ● Competition website: http://www.wsdm-cup-2017.org/ ● Competition platform: http://tira.io/ ● My solution: ○ https://github.com/alexeygrigorev/wsdmcup17-vandalism-de tection
- 24. What’s next? ● Wikimedia plans to integrate the model ● Not as is, but they found many ideas useful