Count-min sketch to Infinity.pdf

Count-min sketch to Infinity:
Using Probabilistic Data Structures to Solve Presence, Counting, and Distinct
Count Problems in .NET
presented by: Steve Lorello - Developer Advocate @Redis

Agenda
● What are Probabilistic Data Structures?
● Set Membership problems
● Bloom Filters
● Counting problems
● Count-Min-Sketch
● Distinct Count problems
● HyperLogLog
● Using Probabilistic Data Structures with Redis

What are Probabilistic Data Structures?
● Class of specialized data structures
● Tackle specific problems
● Use probability approximate

Probabilistic Data Structures Examples
Name Problem Solved Optimization
Bloom Filter Presence Space, Insertion Time, Lookup Time
Quotient Filter Presence Space, Insertion Time, Lookup Time
Skip List Ordering and Searching Insertion Time, Search time
HyperLogLog Set Cardinality Space, Insertion Time, Lookup Time
Count-min-sketch Counting occurrences on large sets Space, Insertion Time, Lookup Time
Cuckoo Filter Presence Space, Insertion Time, Lookup Time
Top-K Keep track of top records Space, Insertion Time, Lookup Time

Set Membership Problems
● Has a given element been inserted?
● e.g. Unique username for registration

Presence Problem Naive Approach 1
● Store User Info in table ‘users’ and Query
Check username Query username

SELECT 1
FROM users
WHERE username = ‘selected_username’

Summary
Access Type Disk
Lookup Time O(n)
Extra Space
(beyond storing
user info)
O(1)

● Store User Info in table ‘users’
● Index username

SELECT 1
FROM users
WHERE username = ‘selected_username’

Summary
Access Type Disk
Lookup Time O(log(n))
Extra Space
(beyond storing
user info)
O(n)

● Store usernames in Redis cache
Check username SISMEMBER

● Store usernames in Redis cache
SADD usernames selected_username
SISMEMBER usernames selected_username
Check username SISMEMBER

Summary
Access Type Memory
Lookup Time O(1)
Extra Space
(beyond storing
user info)
O(n)

Bloom Filter
● Specialized ‘Probabilistic’ Data Structure for presence checks
● Can say if element has deﬁnitely not been added
● Can say if element has probably been added
● Uses constant K-hashes scheme
● Represented as a 1D array of bits
● All operations O(1) complexity
● Space complexity O(n) - bits

Insert:
For i = 0->K:
FILTER[H[ i ](key)] = 1

Query:
For i = 0 -> K:
If FILTER[H[ i ](key)] == 0:
Return False
Return true

Complexities
Type Worst Case
Space O(n) - BITS
Insert O(1)
Lookup O(1)
Delete Not Available

Example Initial State
Bloom Filter k = 3
bit 0 1 2 3 4 5 6 7 8 9
state 0 0 0 0 0 0 0 0 0 0

Example Insert username ‘razzle’
Bloom Filter k = 3
bit 0 1 2 3 4 5 6 7 8 9
state 0 0 0 0 0 0 0 0 0 0

Bloom Filter k = 3
bit 0 1 2 3 4 5 6 7 8 9
state 0 0 1 0 0 0 0 0 0 0
● H1(razzle) = 2

Bloom Filter k = 3
bit 0 1 2 3 4 5 6 7 8 9
state 0 0 1 0 0 1 0 0 0 0
● H1(razzle) = 2
● H2(razzle) = 5

Bloom Filter k = 3
bit 0 1 2 3 4 5 6 7 8 9
state 0 0 1 0 0 1 0 0 1 0
● H1(razzle) = 2
● H2(razzle) = 5
● H3(razzle) = 8

Example Query username ‘fizzle’
Bloom Filter k = 3
bit 0 1 2 3 4 5 6 7 8 9
state 0 0 1 0 0 1 0 0 1 0
H1(ﬁzzle) = 8 - bit 8 is set—maybe?

Bloom Filter k = 3
bit 0 1 2 3 4 5 6 7 8 9
state 0 0 1 0 0 1 0 0 1 0

Bloom Filter k = 3
bit 0 1 2 3 4 5 6 7 8 9
state 0 0 1 0 0 1 0 0 1 0
H2(ﬁzzle) = 4 - bit 4 is not set—deﬁnitely not.

False Positives and Optimal K
● This algorithm will never give you false negatives, but it is
possible to report false positives
● You can optimize false positives by optimizing K
● Let c = hash-table-size/num-records
● Optimal K = c * ln(2)
● This will result in a false positivity rate of .6185^c, this will be
quite small

What’s a Counting Problem?
● How many times does an individual occur in a stream
● Easy to do on small-mid size streams of data
● e.g. Counting Views on YouTube
● Nearly impossible to scale to enormous data sets

Naive Approach: Hash Table
● Hash Table of Counters
● Lookup name in Hash table, instead of storing record,
store an integer
● On insert, increment the integer
● On query, check the integer

Pros
● Straight Forward
● Guaranteed accuracy (if
storing whole object)
● O(n) Space Complexity in the
best case
● O(n) worst case time
complexity
● Scales poorly (think billions
of unique records)
● If relying on only a single
hash - very vulnerable to
collisions and overcounts
Cons

Naive Approach Relational DB
● Issue a Query to a traditional Relational Database searching for a count of
record where some condition occurs
SELECT COUNT( * ) FROM views
WHERE name=”Gangnam Style”
Linear Time Complexity O(n)
Linear Space Complexity O(n)

What’s the problem with a Billion Unique Records?
● Each unique record needs its own space in a Hash Table or row in a RDBMS
(perhaps several rows across multiple tables)
● Taxing on memory for Hash Table
○ 8 bit integer? 1GB
○ 16 bit? 2GB
○ 32 bit? 4GB
○ 64 bit? 8GB
● Maintaining such large data structures in a typical program’s memory isn’t
feasible
● In a relational database, it’s stored on disk

Count-Min Sketch
● Specialized data structure for keeping count on very large streams of data
● Similar to Bloom ﬁlter in Concept - multi-hashed record
● 2D array of counters
● Sublinear Space Complexity
● Constant Time complexity
● Never undercounts, sometimes over counts

Increment:
For i = 0 -> k:
Table[ H(i) ][ i ] += 1

Query:
minimum = infinity
For i = 0 -> k:
minimum = min(minimum,Table[H(i)][i])
return minimum

Video Views Sketch 10 x 3
Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 0 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 0 0 0 0 0 0
H3 0 0 0 0 0 0 0 0 0 0

Increment Gangnam Style
Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 0 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 0 0 0 0 0 0
H3 0 0 0 0 0 0 0 0 0 0

● H1(Gangnam Style) = 0
Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 1 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 0 0 0 0 0 0
H3 0 0 0 0 0 0 0 0 0 0

Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 1 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 0 0 0 0 0
H3 0 0 0 0 0 0 0 0 0 0

Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 1 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 0 0 0 0 0
H3 0 0 0 0 0 0 1 0 0 0

Increment Baby Shark
● H1(Baby Shark) = 0
Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 1 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 0 0 0 0 0
H3 0 0 0 0 0 0 1 0 0 0

Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 2 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 0 0 0 0 0
H3 0 0 0 0 0 0 1 0 0 0

Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 2 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 1 0 0 0 0
H3 0 0 0 0 0 0 1 0 0 0

Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 2 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 1 0 0 0 0
H3 0 0 0 0 0 0 2 0 0 0

Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 2 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 1 0 0 0 0
H3 0 0 0 0 0 0 2 0 0 0
Query Gangnam Style
● MIN (2)

Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 2 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 1 0 0 0 0
H3 0 0 0 0 0 0 2 0 0 0
Query Gangnam Style
● MIN (2, 1)

Count Min Sketch
position 0 1 2 3 4 5 6 7 8 9
H1 2 0 0 0 0 0 0 0 0 0
H2 0 0 0 0 1 1 0 0 0 0
H3 0 0 0 0 0 0 2 0 0 0
Query Gangnam Style
● MIN (2, 1, 2) = 1

Type Worst Case
Space Sublinear
Increment O(1)
Query O(1)
Delete Not Available
Complexities

CMS Pros
● Extremely Fast - O(1)
● Super compact - sublinear
● Impossible to undercount
● incidence of overcounting - all
results are approximations
CMS Cons

When to Use a Count Min Sketch?
● Counting many unique instances
● When Approximation is fine
● When counts are likely to be skewed (think YouTube video views)

Set Cardinality
● Counting distinct elements inserted into set
● Easier on smaller data sets
● For exact counts - must preserve all unique elements
● Scales very poorly

HyperLogLog
● Probabilistic Data Structure to Count Distinct Elements
● Space Complexity is about O(1)
● Time Complexity O(1)
● Can handle billions of elements with less than 2 kB of memory
● Can scale up as needed - HLL is effectively constant even
though you may want to increase its size for enormous
cardinalities.

HyperLogLog Walkthrough
● Initialize an array of registers of size 2^P (where P is some constant, usually
around 16-18)
● When an Item is inserted
○ Hash the Item
○ Determine the register to update: i - from the left P bits of the item’s hash
○ Set registers[i] to the index of the rightmost 1 in the binary representation of the hash
● When Querying
○ Compute harmonic mean of the registers that have been set
○ Multiply by a constant determined by size of P

Example: Insert Username ‘bar’ P = 16
H(bar) = 3103595182
● 1011 1000 1111 1101 0001 1010 1010 1110
● Take ﬁrst 16 bits -> 1011 1000 1111 1101 -> 47357 = register index
● Index of rightmost 1 = 1
● registers[47357] = 1

Get Cardinality
● Calculate harmonic mean of only set registers.
Only 1 set register: 47357 -> 1
ceiling(.673 * 1 * 1/(2^1)) = 1
Cardinality = 1

Probabilistic Data
Structures with
Redis-Stack

What Is Redis Stack?
● Grouping of modules for Redis, including Redis Bloom
● Adds additional functionality, e.g. Probabilistic Data structures to Redis

Steve Lorello
Developer Advocate
@Redis
@slorello
github.com/slorello89
slorello.com

Resources
Redis
https://redis.io
RedisBloom
https://redisbloom.io
Source Code For Demo:
https://github.com/slorello89/probablistic-data-structures-blazor
C# Implementation Bloom Filter, HyperLogLog, and Count-Min Sketch:
https://github.com/TheAlgorithms/C-Sharp/tree/master/DataStructures/Probabilistic

Come Check Us Out!
Redis University:
https://university.redis.com
Discord:
https://discord.com/invite/redis

Count-min sketch to Infinity.pdf

Baby Name Freq hash table
0 1 2 3 4 5 6 7 8 9
0 0 0 0 0 0 0 0 0 0

H(Liam) = 4
0 1 2 3 4 5 6 7 8 9
0 0 0 0 0 0 0 0 0 0

H(Liam) = 4
0 1 2 3 4 5 6 7 8 9
0 0 0 0 1 0 0 0 0 0

H(Sophia) = 8
0 1 2 3 4 5 6 7 8 9
0 0 0 0 1 0 0 0 0 0

H(Sophia) = 8
0 1 2 3 4 5 6 7 8 9
0 0 0 0 1 0 0 0 1 0

H(Liam) = 4
0 1 2 3 4 5 6 7 8 9
0 0 0 0 2 0 0 0 1 0

Baby Name existence table k = 3
0 1 2 3 4 5 6 7 8 9
0 0 0 0 0 0 0 0 0 0

H1(Liam)=0 H2(Liam) = 4 H3(Liam) = 6
0 1 2 3 4 5 6 7 8 9
0 0 0 0 0 0 0 0 0 0

H1(Liam)=0 H2(Liam) = 4 H3(Liam) = 6
0 1 2 3 4 5 6 7 8 9
1 0 0 0 1 0 1 0 0 0

H1(Susan)=0 H2(Susan) = 5 H3(Susan) = 6
0 1 2 3 4 5 6 7 8 9
1 0 0 0 1 0 1 0 0 0

H1(Susan)=0 H2(Susan) = 5 H3(Susan) = 6
0 1 2 3 4 5 6 7 8 9
1 0 0 0 1 1 1 0 0 0

Does Tom Exist? H1(Tom)=1 H2(Tom)=4 H3(Tom) = 5
0 1 2 3 4 5 6 7 8 9
1 0 0 0 1 1 1 0 0 0

Does Tom Exist? H1(Tom)=1 H2(Tom)=4 H3(Tom) = 5
0 1 2 3 4 5 6 7 8 9
1 0 0 0 1 1 1 0 0 0
A hash of Tom = 0, so no Tom does not exist!

Does Liam Exist? H1(Liam)=0 H2(Liam) = 4 H3(Liam) = 6
0 1 2 3 4 5 6 7 8 9
1 0 0 0 1 1 1 0 0 0
All Hashes of Liam = 1, so we repot YES

2-Choice Hashing
● Use two Hash Functions instead of one
● Store @ index with Lowest Load (smallest linked list)
● Time Complexity goes from log(n) in traditional chain hash table -> log(log(n))
with high probability, so nearly constant
● Benefit stops at 2 hashes, additional hashes don’t help
● Still O(n) space complexity

Hash Table
● Ubiquitous data structure for
storing associated data. E.g.
Map, Dictionary, Dict
● Set of Keys associated with
array of values
● Run hash function on key to find
position in array to store value
Source: wikipedia

Hash Collisions
● Hash Functions can
produce the same
output for different
keys - creates
collision
● Collision Resolution
either sequentially of
with linked-list

Hash Table Complexity - with chain hashing
Type Amortized Worst Case
Space O(n) O(n)
Insert O(1) O(n)
Lookup O(1) O(n)
Delete O(1) O(n)

Count-min sketch to Infinity.pdf

More Related Content

What's hot (20)

Similar to Count-min sketch to Infinity.pdf (20)

More from Stephen Lorello (7)

Recently uploaded (20)

Count-min sketch to Infinity.pdf