Probabilistic data structures in real life
Download as PPTX, PDF1 like596 views
The document discusses probabilistic data structures, focusing on techniques like Bloom filters and HyperLogLog to efficiently estimate large datasets with minimal memory usage. It provides a business case for segmenting a large audience using these methods, highlighting challenges related to intersections and subtractions in set operations. Key concepts like Jaccard distance and the inclusion-exclusion principle are also mentioned, along with error rates associated with the architecture.
![HYPERLOGLOG INTUITION
00101010101010001111010101101 => a[2] = 0
10010101010100101010101001011 => a[9] = 1
00000101010100101010101110101 => a[0] = 1
01010101010100100101010101010 => a[5] = 1
01010000000000000000000000010 => a[5] = 23](https://image.slidesharecdn.com/probabilisticdatastructuresinreallife-151026090545-lva1-app6891/85/Probabilistic-data-structures-in-real-life-7-320.jpg)
Probabilistic data structures in real life
- 1. PROBABILISTIC DATA STRUCTURES IN REAL LIFE Valentin Bazarevsky
- 2. WHO THEY ARE? Bloom Filter LogLog Family MinHash
- 3. BUSINESS CASE: ESTIMATE YOUR AUDIENCE
- 4. SEGMENT BUILDER 15 Tb of transactional data 4h SLA
- 5. POSSIBLE SOLUTIONS Brute force (15 TB of transactional data) Sampling (1 % of users => 1.2 mb / b.o.) Magic tool (?!) Estimator HyperLogLog allows to estimate > 1 000 000 000 sets of unique elements with 1% error, and requires only 4kb memory 50 000 000 basic operations
- 6. OOPS… Supports only Unions But we need Intersections, Subtractions, Not operators
- 7. HYPERLOGLOG INTUITION 00101010101010001111010101101 => a[2] = 0 10010101010100101010101001011 => a[9] = 1 00000101010100101010101110101 => a[0] = 1 01010101010100100101010101010 => a[5] = 1 01010000000000000000000000010 => a[5] = 23
- 9. MINHASH Store only x (8192) smallest hashes in set Jaccard Distance
- 10. UNION OF INTERSECTIONS A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ B) A - B - C = A - (B ∪ C)
- 12. I WANT EVERYONE EXCEPT… A and not B Not A and Not B
- 13. CORNER CASES |(A ∪ not(B)) ∩ C| => |A ∩ C| |A ∪ not(B)| = |Everything| - |B| + |A ∩ B| |A ∩ not(B)| => |A| - |A ∩ B|
- 14. ARCHITECTURE
- 15. ERROR RATE Median = 5% Percentile 75 = 8%