![the problem to address
compute cardinality of a multiset
compute distinct elements in a data set with duplicated elements
e.g. there are 3 distinct elements in [a, b, a, c]](https://image.slidesharecdn.com/hyperloglog-170804033148/85/Hyper-loglog-3-320.jpg)
Hyper loglog
- 2. the problem to address compute cardinality of a multiset
- 3. the problem to address compute cardinality of a multiset compute distinct elements in a data set with duplicated elements e.g. there are 3 distinct elements in [a, b, a, c]
- 4. solution 1 len(hash_set(a_multi_set)) cons: out of memory for big data set
- 5. solution 2 sorted_data_on_disk = external_sort(a_multi_set) count(sorted_data_on_disk) cons: slowness
- 6. the problem to address compute estimated cardinality of a very big multiset
- 7. Demo
- 9. Most HyperLogLog explanations on the web
- 11. Some observations for evenly distributed numbers
- 13. Observation for evenly distributed numbers estimate cardinality using the min value cardinality ≈ max / min
- 15. Observation for evenly distributed numbers estimate using the number of leading zeros in a number ever see cardinality ≈ 2k (where k is the biggest number of leading zeros found in a number)
- 16. cardinality ≈ 2k (where k is the biggest number of leading zeros found in a number) e.g. 232 => log(232) = max 32 leading zeros => log(32) = 5-bit counter LogLog(232)
- 17. But what if our data set isn't evenly distributed integers? hash functions
- 18. high error rate? divided subsets stochastic averaging
- 19. a couple more technicalities correcting your estimate if it is below a certain amount, or if it is very large use harmonic mean instead of the geometric mean
- 20. References • https://github.com/sergeio/hyperloglog/blob/master/README.md • http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality- Estimation • http://antirez.com/news/75 • https://www.periscopedata.com/blog/hyperloglog-in-pure-sql.html • https://stackoverflow.com/questions/12327004/how-does-the- hyperloglog-algorithm-work • http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf • http://opensourceconnections.com/blog/2015/02/04/its-log-its-log- its-big-its-hyper-its-good/
Editor's Notes
- #19: count the-maximum-amount-of-trailing-zeroes-on-the-hash-of-each-value -- *gaasp* -- for each subset, and average them together, we can get much closer. "stochastic averaging"
- #20: count the-maximum-amount-of-trailing-zeroes-on-the-hash-of-each-value -- *gaasp* -- for each subset, and average them together, we can get much closer. "stochastic averaging"