Introduction to Bloom Filters
0 likes72 views
This document provides an overview of Bloom filters, including what they are, why you would want to use one, practical examples of their use, and how they work. Bloom filters are a data structure that can be used to test if an element is present in a set. They have a tiny memory footprint compared to storing the full set, use constant memory, and allow extremely fast lookups. However, they can produce false positives. The document discusses examples of Bloom filters being used by companies like Akamai, Medium, and Cassandra to reduce storage usage and improve performance. It also explains how Bloom filters work under the hood using hash functions and bit arrays.
Introduction to Bloom Filters
- 1. Hayden Marchant, Software Engineer haydenm@wix.com Bloom Filters
- 2. Agenda A Bloom what? Why would I want one? Practical examples How it works Next step
- 3. A Bloom-what? A Bloom what? Why would I want one? Practical examples How it works Next step
- 4. Just a Data Structure ‘Is an element in a set or not’
- 6. Definitely not in set Probably in set OR
- 7. ILLUSTRATION
- 8. Why would I want one? A Bloom what? Why would I want one? Practical examples How it works Next step
- 9. Why not just store all used keys?
- 10. Element Key size: # elements: Memory required: 100 bytes 10 million 100 * 10M = 1GB SIMPLE MATHS
- 11. Bloom Filters: Tiny footprint ~ 10 bits per element
- 12. >98% reduction in space Element Key size: # elements: Memory required: 10 bits 10 million 100M bits ~= 12MB SIMPLE MATHS (PART 2)
- 13. Footprints: Traditional = 1GB Bloom Filter <=12MB
- 14. Practical Examples A Bloom what? Why would I want one? Practical examples How it works Next step
- 15. Akamai Prevent caching of One-Hit-Wonders 75% of web requests are only ever called once - Called One-Hit-Wonders Use Bloom Filters to prevent one-hit-wonders to be stored in Disk Cache 1. Saves disk-access for these 75% of requests 2. reducing workload
- 16. Medium Avoid recommending articles a user has read - Add article recommended to bloom filter - $userid-$articleid - Before recommending, check in Bloom Filter if (user,article) exists. - If not, then recommend and then add to bloom filter
- 17. Cassandra Reduce disk lookup for non-existent rows in file storage - Shard responsible over range of data - Each shard stored in immutable shard-files - Create Bloom Filter in server for each shard-file - Do not access shard-files if Bloom Filter returns false for key
- 18. Chrome Identifying Malicious URLs Store malicious URLs in Bloom Filter In-depth check on positive response
- 19. How it works A Bloom what? Why would I want one? Practical examples How it works Next step
- 20. Basics of Bloom Filter ● Array of m bits ● initially set to 0 ● Hash functions ● k hash functions defined ● maps or hashes some set element to one of the m array positions
- 22. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 { }m = 20 k = 3
- 23. 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 { cat}m = 20 k = 3
- 24. 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 0 1 { cat , dog }m = 20 k = 3
- 25. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1 {cat , dog , mouse }m = 20 k = 3
- 26. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1 {cat , dog , mouse } owl m = 20 k = 3 owl is definitely NOT in set
- 27. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1 rat m = 20 k = 3 rat might be in set {cat , dog , mouse }
- 30. Low Error Rate
- 32. 1 2 3 4 5 PROBABILITIES 101 Probability of slot not selected for a single hash function Probability of slot not selected for all k hashes Probability of slot not selected for all k hashes of n elements Probability of slot selected for all k hashes of n elements Probability of k slots selected for n elements === False positive
- 34. If you want to read more on the maths, take a look at https://en.wikipedia.org/ wiki/Bloom_filter
- 35. A Simple Calculator Click here: https://hur.st/bloomfilter
- 36. Next step A Bloom what? Why would I want one? Practical examples How it works Next step
- 37. Bloom Filters in Code ● Libraries exist in most languages ○ https://github.com/alexan drnikitin/bloom-filter-scala ○ https://www.npmjs.com/p ackage/bloom-filters ● Distributed Bloom Filters in Redis ○ Perfect for multiple instance of a service using single bloom filter
- 38. Now, go think how using Bloom Filters in your services could reduce payload and speed things up
- 39. Other probabilistic structures ● Counting Bloom Filters ○ Allow deletes ● TopK ○ Keep track of Top K counts of elements ● Count-Min Sketch ○ Frequency table of elements
- 40. Q&A