Speeding Up Minwise Hashing for Weighted Sets
- 3. 3
- 4. 4 • Useful if objects can be represented as sets of features • and Jaccard similarity is an appropriate similarity measure coronavirus hate the “I hate the coronavirus!” I “I hate lockdowns!” 25 21 18 41 98 12 15 41 25 32 18 11 98 56 33 72 Set representation lockdowns hateI Object Signature Similarity estimation Minwise hashing Minwise hashing used for deduplication of similar web pages
- 5. 5 I 25 63 98 hate 67 41 18 the 79 34 35 coronavirus 36 21 52 25 21 18 input set signature minimum hash value defines signature component independent hash functions
- 6. 6
- 7. 7
- 9. 9
- 10. 10
- 11. 11
- 12. 12
- 13. 13
- 14. 14 Step 1 Step 2 Step 1 Step 2
- 15. 15 claims that Ioffe’s algorithm is wrong!
- 16. 16
- 17. 17
- 18. 18
- 19. 19
- 20. 20
- 21. 21
- 22. 22
- 23. 23
- 24. 24
- 25. 25
- 26. 26
- 27. 27
- 28. 28
- 29. 29
- 30. 30
- 31. 31 “Bagminhash - Minwise hashing algorithm for weighted sets” (Ertl, KDD 2018)
- 32. 32 “DartMinHash: Fast Sketching for Weighted Sets” (Christiani, 2020)
- 33. 33 “DartMinHash: Fast Sketching for Weighted Sets” (Christiani, 2020)
- 40. 40 DartMinHash performs best if weights are normalized Performance of DartMinHash depends on total weight https://github.com/oertl/treeminhash
- 42. 42
- 43. 43 “Maximally consistent sampling and the Jaccard index of probability distributions” (Moulton & Jiang, ICDMW 2018)
- 44. 44 “ProbMinHash–A Class of Locality-Sensitive Hash Algorithms for the (Probability) Jaccard Similarity” (Ertl, TKDE 2020)
- 45. 45
- 46. 46
- 47. 47
- 48. 48 ProbMinHash4ProbMinHash3 ProbMinHash2ProbMinHash1 with replacement w/o replacement Label sampling uncorrelatedcorrelated Pointsampling
- 49. 49
- 50. 50 Correlated point generation of ProbMinHash3/4 may reduce estimation error for small sets!
- 51. 51
- 52. 52