SlideShare a Scribd company logo

Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook

The Hive, profile picture
The Hive

The document discusses RocksDB, an embedded key-value store optimized for fast storage, emphasizing its architecture, performance improvements over previous databases like LevelDB, and features such as pluggable components and efficient compaction strategies. It highlights RocksDB's ability to substantially reduce write amplification, improve read efficiency with bloom filters, and support various storage backends. Additionally, it outlines practical use cases and optimizations that cater to specific application requirements, underscoring its suitability for server workloads.

TechnologyEducation
1 of 101
Downloaded 617 times
1
2
3
4
5
6
Most read
7
8
9
10
Most read
11
12
13
14
15
16
17
18
Most read
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
The Story of RocksDB Embedded Key-Value Store for Flash and RAM Dhruba Borthakur & Haobo Xu Database Engineering@Facebook Monday, December 9, 13
Monday, December 9, 13
Monday, December 9, 13
A Client-Server Architecture with disks Application Server Network roundtrip = 50 micro sec Database Server Disk access = 10 milli seconds Locally attached Disks Monday, December 9, 13
Client-Server Architecture with fast storage Application Server Network roundtrip = 50 micro sec Database Server 100 microsecs SSD Latency dominated by network Monday, December 9, 13 100 nanosecs RAM
Architecture of an Embedded Database Application Server Monday, December 9, 13 Network roundtrip = 50 micro sec Database Server
Architecture of an Embedded Database Application Server Monday, December 9, 13 Network roundtrip = 50 micro sec Database Server
Architecture of an Embedded Database Network roundtrip = 50 micro sec Application Server 100 microsecs SSD Monday, December 9, 13 100 nanosecs RAM Database Server
Architecture of an Embedded Database Network roundtrip = 50 micro sec Application Server 100 microsecs SSD 100 nanosecs RAM Storage attached directly to application servers Monday, December 9, 13 Database Server
Any pre-existing embedded databases? Open Source Monday, December 9, 13 FB Proprietary
Any pre-existing embedded databases? Key-value stores 1. 2. 3. 4. Berkeley DB SQLite Kyoto TreeDB LevelDB Open Source Monday, December 9, 13 FB Proprietary
Any pre-existing embedded databases? Key-value stores Open Source 1. High Performant 2. No transaction log 3. Fixed size keys FB Proprietary Monday, December 9, 13
Comparison of open source databases Monday, December 9, 13
Comparison of open source databases Random Reads Monday, December 9, 13
Comparison of open source databases Random Reads Random Writes Monday, December 9, 13
Comparison of open source databases Random Reads LevelDB Kyoto TreeDB SQLite3 Random Writes Monday, December 9, 13 129,000 ops/sec 151,000 ops/sec 134,000 ops/sec
Comparison of open source databases Random Reads LevelDB Kyoto TreeDB SQLite3 129,000 ops/sec 151,000 ops/sec 134,000 ops/sec Random Writes LevelDB Kyoto TreeDB SQLite3 Monday, December 9, 13 164,000 ops/sec 88,500 ops/sec 9,860 ops/sec
HBase and HDFS (in April 2012) Details of this experiment: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html Monday, December 9, 13
HBase and HDFS (in April 2012) Random Reads Details of this experiment: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html Monday, December 9, 13
HBase and HDFS (in April 2012) Random Reads HDFS (1 node) HBase (1 node) 93,000 ops/sec 35,000 ops/sec Details of this experiment: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html Monday, December 9, 13
Log Structured Merge Architecture Read Write data in RAM Monday, December 9, 13
Log Structured Merge Architecture Write Request from Application Read Write data in RAM Monday, December 9, 13
Log Structured Merge Architecture Write Request from Application Read Write data in RAM Monday, December 9, 13
Log Structured Merge Architecture Write Request from Application Read Write data in RAM Monday, December 9, 13
Log Structured Merge Architecture Write Request from Application Read Write data in RAM Transaction log Monday, December 9, 13
Log Structured Merge Architecture Write Request from Application Read Write data in RAM Transaction log Monday, December 9, 13
Log Structured Merge Architecture Write Request from Application Read Write data in RAM Transaction log Monday, December 9, 13
Log Structured Merge Architecture Write Request from Application Read Write data in RAM Read Only data in RAM on disk Monday, December 9, 13 Transaction log
Log Structured Merge Architecture Write Request from Application Periodic Compaction Read Write data in RAM Read Only data in RAM on disk Monday, December 9, 13 Transaction log
Log Structured Merge Architecture Scan Request from Application Periodic Compaction Read Write data in RAM Read Only data in RAM on disk Monday, December 9, 13 Write Request from Application Transaction log
Leveldb has low write rates Facebook Application 1: • Write rate 2 MB/sec only per machine • Only one cpu was used Monday, December 9, 13
Leveldb has low write rates Facebook Application 1: • Write rate 2 MB/sec only per machine • Only one cpu was used We developed multithreaded compaction 10x improvement on write rate Monday, December 9, 13 + 100% of cpus are in use
Leveldb has stalls Facebook Feed: • P99 latencies were tens of seconds • Single-threaded compaction Monday, December 9, 13
Leveldb has stalls Facebook Feed: • P99 latencies were tens of seconds • Single-threaded compaction We implemented thread aware compaction Dedicated thread(s) to flush memtable Monday, December 9, 13 Pipelined memtables P99 reduced to less than a second
Leveldb has high write amplification • Facebook Application 2: • Level Style Compaction • Write amplification of 70 very high Monday, December 9, 13
Leveldb has high write amplification • Facebook Application 2: • Level Style Compaction • Write amplification of 70 very high Level-0 5 bytes Level-1 6 bytes 11 bytes Level-2 10 bytes 10 bytes 10 bytes Stage 1 Stage 2 Stage 3 Two compactions by LevelDB Style Compaction Monday, December 9, 13
Our solution: lower write amplification • Facebook Application 2: • We implemented Universal Style Compaction • Start from newest file, include next file in candidate set if • Candidate set size >= size of next file Monday, December 9, 13
Our solution: lower write amplification • Facebook Application 2: • We implemented Universal Style Compaction • Start from newest file, include next file in candidate set if • Candidate set size >= size of next file Level-0 5bytes Level-1 6 bytes Level-2 10 bytes 10 bytes Stage 1 Stage 2 Single compaction by Universal Style Compaction Write amplification reduced to <10 Monday, December 9, 13
Leveldb has high read amplification Monday, December 9, 13
Leveldb has high read amplification • Secondary Index Service: • Leveldb does not use blooms for scans Monday, December 9, 13
Leveldb has high read amplification • Secondary Index Service: • Leveldb does not use blooms for scans • We implemented prefix scans • Range scans within same key prefix • Blooms created for prefix • Reduces read amplification Monday, December 9, 13
Leveldb: read modify write = 2X IOs Monday, December 9, 13
Leveldb: read modify write = 2X IOs • Counter increments • Get value, value++, Put value • Leveldb uses 2X IOPS Monday, December 9, 13
Leveldb: read modify write = 2X IOs • Counter increments • Get value, value++, Put value • Leveldb uses 2X IOPS • We implemented MergeRecord • Put “++” operation in MergeRecord • Background compaction merges all MergeRecords • Uses only 1X IOPS Monday, December 9, 13
Leveldb has a Rigid Design Monday, December 9, 13
Leveldb has a Rigid Design • LevelDB Design • Cannot tune system, fixed file sizes Monday, December 9, 13
Leveldb has a Rigid Design • LevelDB Design • Cannot tune system, fixed file sizes • We wanted a pluggable architecture • Pluggable compaction filter, e.g. TimeToLive • Pluggable memtable/sstable for RAM/Flash • Pluggable Compaction Algorithm Monday, December 9, 13
The Changes we did to LevelDB Monday, December 9, 13
The Changes we did to LevelDB Inherited from LevelDB • Log Structured Merge DB • Gets/Puts/Scans of keys • Forward and Reverse Iteration Monday, December 9, 13
The Changes we did to LevelDB Inherited from LevelDB • Log Structured Merge DB • Gets/Puts/Scans of keys • Forward and Reverse Iteration Monday, December 9, 13 RocksDB • 10X higher write rate • Fewer stalls • 7x lower write amplification • Blooms for range scans • Ability to avoid read-modify-write • Optimizations for flash or RAM • And many more…
RocksDB is born! • Key-Value persistent store • Embedded • Optimized for fast storage • Server workloads Monday, December 9, 13
RocksDB is born! • Key-Value persistent store • Embedded • Optimized for fast storage • Server workloads Monday, December 9, 13
What is it not? • Not distributed • No failover • Not highly-available, if machine dies you lose your data Monday, December 9, 13
What is it not? • Not distributed • No failover • Not highly-available, if machine dies you lose your data Monday, December 9, 13
RocksDB API ▪ Keys and values are arbitrary byte arrays. ▪ Data is stored sorted by key. ▪ The basic operations are Put(key,value), Get(key), Delete(key) and Merge(key, delta) ▪ Forward and backward iteration is supported over the data. Monday, December 9, 13
RocksDB Architecture Active MemTable log Switch Switch ReadOnly MemTable log log LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
RocksDB Architecture Write Request Active MemTable log Switch Switch ReadOnly MemTable log log LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
RocksDB Architecture Write Request Active MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
RocksDB Architecture Memory Write Request Active MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
RocksDB Architecture Memory Write Request Persistent Storage Active MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
Log Structured Merge Tree -- Writes ▪ Log Structured Merge Tree ▪ New Puts are written to memory and optionally to transaction log ▪ Also can specify log write sync option for each individual write ▪ We say RocksDB is optimized for writes, what does this mean? Monday, December 9, 13
RocksDB Write Path Active MemTable log Switch Switch ReadOnly MemTable log log LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
RocksDB Write Path Write Request Active MemTable log Switch Switch ReadOnly MemTable log log LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
Log Structured Merge Tree -- Reads ▪ Data could be in memory or on disk ▪ Consult multiple files to find the latest instance of the key ▪ Use bloom filters to reduce IO Monday, December 9, 13
RocksDB Read Path Active MemTable log ReadOnly MemTable log log LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
RocksDB Read Path Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
RocksDB Read Path Memory Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
RocksDB Read Path Memory Persistent Storage Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
RocksDB: Open & Pluggable Customizable WAL Blooms Pluggable Memtable format in RAM Monday, December 9, 13
RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Monday, December 9, 13
RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Monday, December 9, 13
RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Monday, December 9, 13
RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Transaction log Monday, December 9, 13
RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Transaction log Monday, December 9, 13
RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Transaction log Monday, December 9, 13
RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Pluggable sst data format on storage Monday, December 9, 13 Transaction log
RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Compaction Pluggable Memtable format in RAM Pluggable sst data format on storage Monday, December 9, 13 Transaction log
RocksDB: Open & Pluggable Get or Scan Request from Application Write Request from Application Customizable WAL Blooms Pluggable Compaction Pluggable Memtable format in RAM Pluggable sst data format on storage Monday, December 9, 13 Transaction log
Example: Customizable WALogging • In-house Replication solution wants to be able to embed arbitrary blob in the rocksdb WAL stream for log annotation • Use Case: Indicate where a log record came from in multi-master replication • Solution: Monday, December 9, 13 A Put that only speaks to the log
Example: Customizable WALogging Active MemTable k1 Replication Layer In one write batch: PutLogData(“I came from Mars”) Put(k1,v1) Monday, December 9, 13 v1 log “I came from Mars” k1/v1
Example: Customizable WALogging Active MemTable Write Request Replication Layer In one write batch: PutLogData(“I came from Mars”) Put(k1,v1) Monday, December 9, 13 k1 v1 log “I came from Mars” k1/v1
Example: Pluggable SST format • One Facebook use case needs extreme fast response but could tolerate some loss of durability • Quick hack: mount sst in tmpfs • Still not performant: • existing sst format is block based • Solution: A much simpler format that just stores sorted key/value pairs sequentially • no blocks, no caching, mmap the whole file • build efficient lookup index on load Monday, December 9, 13
Example: Blooms for MemTable • Same use case, after we optimized sst access, memtable lookup becomes a major cost in query • Problem: Get needs to go through the memtable lookups that eventually return no data • Solution: Monday, December 9, 13 Just add a bloom filter to memtable!
RocksDB Read Path Blooms Blooms Active MemTable log ReadOnly MemTable log log LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
RocksDB Read Path Blooms Blooms Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
RocksDB Read Path Memory Blooms Blooms Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
RocksDB Read Path Memory Blooms Blooms Persistent Storage Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
Example: Pluggable memtable format • Another Facebook use case has a distinct load phase where no query is issued. • Problem: write throughput is limited by single writer thread • Solution: A new memtable representation that does not keep keys sorted Monday, December 9, 13
Example: Pluggable memtable format Unsorted MemTable log Switch Switch ReadOnly MemTable log log LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
Example: Pluggable memtable format Write Request Unsorted MemTable log Switch Switch ReadOnly MemTable log log LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
Example: Pluggable memtable format Write Request Unsorted MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
Example: Pluggable memtable format Memory Write Request Unsorted MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
Example: Pluggable memtable format Memory Write Request Persistent Storage Unsorted MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
Possible workloads for RocksDB? ▪ Serving data to users via a website ▪ A spam detection backend that needs fast access to data ▪ A graph search query that needs to scan a dataset in realtime ▪ Distributed Configuration Management Systems ▪ Fast serve of Hive Data ▪ A Queue that needs a high rate of inserts and deletes Monday, December 9, 13
Futures Monday, December 9, 13
Futures • Scale linearly with number of cpus • 32, 64 or higher core machines • ARM processors Monday, December 9, 13
Futures • Scale linearly with number of cpus • 32, 64 or higher core machines • ARM processors • Scale linearly with storage iops • Striped flash cards • RAM & NVRAM storage Monday, December 9, 13
Come Hack with us Monday, December 9, 13
Come Hack with us • RocksDB is Open Sourced • http://rocksdb.org • Developers group https://www.facebook.com/groups/rocksdb.dev/ Monday, December 9, 13
Come Hack with us • RocksDB is Open Sourced • http://rocksdb.org • Developers group https://www.facebook.com/groups/rocksdb.dev/ • Help us HACK RocksDB Monday, December 9, 13
Monday, December 9, 13

More Related Content

PDF
MyRocks Deep Dive
Yoshinori Matsunobu
 
PPTX
RedisConf17- Using Redis at scale @ Twitter
Redis Labs
 
PDF
RedisConf17 - Lyft - Geospatial at Scale - Daniel Hochman
Redis Labs
 
PDF
Google Cloud Dataflow
Alex Van Boxel
 
PPTX
RocksDB detail
MIJIN AN
 
PDF
RocksDB Performance and Reliability Practices
Yoshinori Matsunobu
 
PDF
Landscape of AI/ML in 2023
HyunJoon Jung
 
PDF
MyRocks introduction and production deployment
Yoshinori Matsunobu
 
MyRocks Deep Dive
Yoshinori Matsunobu
 
RedisConf17- Using Redis at scale @ Twitter
Redis Labs
 
RedisConf17 - Lyft - Geospatial at Scale - Daniel Hochman
Redis Labs
 
Google Cloud Dataflow
Alex Van Boxel
 
RocksDB detail
MIJIN AN
 
RocksDB Performance and Reliability Practices
Yoshinori Matsunobu
 
Landscape of AI/ML in 2023
HyunJoon Jung
 
MyRocks introduction and production deployment
Yoshinori Matsunobu
 

What's hot (20)

PDF
Log Structured Merge Tree
University of California, Santa Cruz
 
PPTX
RocksDB compaction
MIJIN AN
 
PDF
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
SrirakshaSrinivasan2
 
PDF
Cassandra Introduction & Features
DataStax Academy
 
PDF
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
InfluxData
 
KEY
Redis overview for Software Architecture Forum
Christopher Spring
 
PDF
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
 
PPTX
Apache Spark Architecture
Alexey Grishchenko
 
PDF
MongoDB Database Replication
Mehdi Valikhani
 
PPTX
Apache Flink and what it is used for
Aljoscha Krettek
 
PDF
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
mumrah
 
PDF
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
 
PPTX
Introduction to Storm
Chandler Huang
 
PPTX
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward
 
PDF
Etsy Activity Feeds Architecture
Dan McKinley
 
PDF
Redis cluster
iammutex
 
PDF
Blazing Performance with Flame Graphs
Brendan Gregg
 
PDF
Parquet performance tuning: the missing guide
Ryan Blue
 
PPTX
The Missing Manual for Leveled Compaction Strategy (Wei Deng & Ryan Svihla, D...
DataStax
 
PDF
InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxD...
InfluxData
 
Log Structured Merge Tree
University of California, Santa Cruz
 
RocksDB compaction
MIJIN AN
 
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
SrirakshaSrinivasan2
 
Cassandra Introduction & Features
DataStax Academy
 
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
InfluxData
 
Redis overview for Software Architecture Forum
Christopher Spring
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
 
Apache Spark Architecture
Alexey Grishchenko
 
MongoDB Database Replication
Mehdi Valikhani
 
Apache Flink and what it is used for
Aljoscha Krettek
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
mumrah
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
 
Introduction to Storm
Chandler Huang
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward
 
Etsy Activity Feeds Architecture
Dan McKinley
 
Redis cluster
iammutex
 
Blazing Performance with Flame Graphs
Brendan Gregg
 
Parquet performance tuning: the missing guide
Ryan Blue
 
The Missing Manual for Leveled Compaction Strategy (Wei Deng & Ryan Svihla, D...
DataStax
 
InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxD...
InfluxData
 
Ad

Similar to Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook (20)

PDF
Data Replication Options in AWS
Irawan Soetomo
 
PDF
Mysql features for the enterprise
Giuseppe Maxia
 
PDF
Cassandra at scale
Patrick McFadin
 
PDF
No C-QL (Or how I learned to stop worrying, and love eventual consistency) (N...
Brian Brazil
 
PDF
Ruby meetup 7_years_in_testing
Digital Natives
 
PDF
Choosing the Right Database Service (김상필, 유타카 호시노) - AWS DB Day
Amazon Web Services Korea
 
PDF
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
PHP Conference Argentina
 
PDF
Escalando una PHP App con DB sharding - PHP Conference
Matias Paterlini
 
PDF
How Agoda Scaled 50x Throughput with ScyllaDB by Worakarn Isaratham
ScyllaDB
 
ODP
The care and feeding of a MySQL database
Dave Stokes
 
PDF
Tuning Linux Windows and Firebird for Heavy Workload
Marius Adrian Popa
 
PDF
2013 CPM Conference, Nov 6th, NoSQL Capacity Planning
asya999
 
PPTX
Capacity Planning
MongoDB
 
PDF
High performance Infrastructure Oct 2013
Server Density
 
PDF
Qcon talk
bcoverston
 
PDF
Say Hello to MyRocks
Sergey Petrunya
 
PPTX
Evaluating Storage for VDI Projects
Tegile Systems
 
PDF
Proud to be polyglot!
NLJUG
 
PPTX
Scalable Text File Service with MongoDB (Intuit)
MongoDB
 
PPTX
Arc305 how netflix leverages multiple regions to increase availability an i...
Ruslan Meshenberg
 
Data Replication Options in AWS
Irawan Soetomo
 
Mysql features for the enterprise
Giuseppe Maxia
 
Cassandra at scale
Patrick McFadin
 
No C-QL (Or how I learned to stop worrying, and love eventual consistency) (N...
Brian Brazil
 
Ruby meetup 7_years_in_testing
Digital Natives
 
Choosing the Right Database Service (김상필, 유타카 호시노) - AWS DB Day
Amazon Web Services Korea
 
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
PHP Conference Argentina
 
Escalando una PHP App con DB sharding - PHP Conference
Matias Paterlini
 
How Agoda Scaled 50x Throughput with ScyllaDB by Worakarn Isaratham
ScyllaDB
 
The care and feeding of a MySQL database
Dave Stokes
 
Tuning Linux Windows and Firebird for Heavy Workload
Marius Adrian Popa
 
2013 CPM Conference, Nov 6th, NoSQL Capacity Planning
asya999
 
Capacity Planning
MongoDB
 
High performance Infrastructure Oct 2013
Server Density
 
Qcon talk
bcoverston
 
Say Hello to MyRocks
Sergey Petrunya
 
Evaluating Storage for VDI Projects
Tegile Systems
 
Proud to be polyglot!
NLJUG
 
Scalable Text File Service with MongoDB (Intuit)
MongoDB
 
Arc305 how netflix leverages multiple regions to increase availability an i...
Ruslan Meshenberg
 
Ad

More from The Hive (20)

PDF
"Responsible AI", by Charlie Muirhead
The Hive
 
PPTX
Translating a Trillion Points of Data into Therapies, Diagnostics, and New In...
The Hive
 
PDF
Digital Transformation; Digital Twins for Delivering Business Value in IIoT
The Hive
 
PDF
Quantum Computing (IBM Q) - Hive Think Tank Event w/ Dr. Bob Sutor - 02.22.18
The Hive
 
PPTX
The Hive Think Tank: Rendezvous Architecture Makes Machine Learning Logistics...
The Hive
 
PDF
Data Science in the Enterprise
The Hive
 
PDF
AI in Software for Augmenting Intelligence Across the Enterprise
The Hive
 
PPTX
“ High Precision Analytics for Healthcare: Promises and Challenges” by Sriram...
The Hive
 
PPTX
"The Future of Manufacturing" by Sujeet Chand, SVP&CTO, Rockwell Automation
The Hive
 
PPTX
Social Impact & Ethics of AI by Steve Omohundro
The Hive
 
PDF
The Hive Think Tank: AI in The Enterprise by Venkat Srinivasan
The Hive
 
PDF
The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...
The Hive
 
PDF
The Hive Think Tank: The Future Of Customer Support - AI Driven Automation
The Hive
 
PPTX
The Hive Think Tank: Talk by Mohandas Pai - India at 2030, How Tech Entrepren...
The Hive
 
PDF
The Hive Think Tank: The Content Trap - Strategist's Guide to Digital Change
The Hive
 
PPTX
Deep Visual Understanding from Deep Learning by Prof. Jitendra Malik
The Hive
 
PDF
The Hive Think Tank: Heron at Twitter
The Hive
 
PPTX
The Hive Think Tank: Unpacking AI for Healthcare
The Hive
 
PPTX
The Hive Think Tank: Translating IoT into Innovation at Every Level by Prith ...
The Hive
 
PDF
The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...
The Hive
 
"Responsible AI", by Charlie Muirhead
The Hive
 
Translating a Trillion Points of Data into Therapies, Diagnostics, and New In...
The Hive
 
Digital Transformation; Digital Twins for Delivering Business Value in IIoT
The Hive
 
Quantum Computing (IBM Q) - Hive Think Tank Event w/ Dr. Bob Sutor - 02.22.18
The Hive
 
The Hive Think Tank: Rendezvous Architecture Makes Machine Learning Logistics...
The Hive
 
Data Science in the Enterprise
The Hive
 
AI in Software for Augmenting Intelligence Across the Enterprise
The Hive
 
“ High Precision Analytics for Healthcare: Promises and Challenges” by Sriram...
The Hive
 
"The Future of Manufacturing" by Sujeet Chand, SVP&CTO, Rockwell Automation
The Hive
 
Social Impact & Ethics of AI by Steve Omohundro
The Hive
 
The Hive Think Tank: AI in The Enterprise by Venkat Srinivasan
The Hive
 
The Hive Think Tank: Machine Learning Applications in Genomics by Prof. Jian ...
The Hive
 
The Hive Think Tank: The Future Of Customer Support - AI Driven Automation
The Hive
 
The Hive Think Tank: Talk by Mohandas Pai - India at 2030, How Tech Entrepren...
The Hive
 
The Hive Think Tank: The Content Trap - Strategist's Guide to Digital Change
The Hive
 
Deep Visual Understanding from Deep Learning by Prof. Jitendra Malik
The Hive
 
The Hive Think Tank: Heron at Twitter
The Hive
 
The Hive Think Tank: Unpacking AI for Healthcare
The Hive
 
The Hive Think Tank: Translating IoT into Innovation at Every Level by Prith ...
The Hive
 
The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...
The Hive
 

Recently uploaded (20)

PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
PDF
Modernizing your data center with Dell and AMD
Principled Technologies
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Approach and Philosophy of On baking technology
30anthuong17
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
Modernizing your data center with Dell and AMD
Principled Technologies
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 

Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook

  • 1. The Story of RocksDB Embedded Key-Value Store for Flash and RAM Dhruba Borthakur & Haobo Xu Database Engineering@Facebook Monday, December 9, 13
  • 4. A Client-Server Architecture with disks Application Server Network roundtrip = 50 micro sec Database Server Disk access = 10 milli seconds Locally attached Disks Monday, December 9, 13
  • 5. Client-Server Architecture with fast storage Application Server Network roundtrip = 50 micro sec Database Server 100 microsecs SSD Latency dominated by network Monday, December 9, 13 100 nanosecs RAM
  • 6. Architecture of an Embedded Database Application Server Monday, December 9, 13 Network roundtrip = 50 micro sec Database Server
  • 7. Architecture of an Embedded Database Application Server Monday, December 9, 13 Network roundtrip = 50 micro sec Database Server
  • 8. Architecture of an Embedded Database Network roundtrip = 50 micro sec Application Server 100 microsecs SSD Monday, December 9, 13 100 nanosecs RAM Database Server
  • 9. Architecture of an Embedded Database Network roundtrip = 50 micro sec Application Server 100 microsecs SSD 100 nanosecs RAM Storage attached directly to application servers Monday, December 9, 13 Database Server
  • 10. Any pre-existing embedded databases? Open Source Monday, December 9, 13 FB Proprietary
  • 11. Any pre-existing embedded databases? Key-value stores 1. 2. 3. 4. Berkeley DB SQLite Kyoto TreeDB LevelDB Open Source Monday, December 9, 13 FB Proprietary
  • 12. Any pre-existing embedded databases? Key-value stores Open Source 1. High Performant 2. No transaction log 3. Fixed size keys FB Proprietary Monday, December 9, 13
  • 13. Comparison of open source databases Monday, December 9, 13
  • 14. Comparison of open source databases Random Reads Monday, December 9, 13
  • 15. Comparison of open source databases Random Reads Random Writes Monday, December 9, 13
  • 16. Comparison of open source databases Random Reads LevelDB Kyoto TreeDB SQLite3 Random Writes Monday, December 9, 13 129,000 ops/sec 151,000 ops/sec 134,000 ops/sec
  • 17. Comparison of open source databases Random Reads LevelDB Kyoto TreeDB SQLite3 129,000 ops/sec 151,000 ops/sec 134,000 ops/sec Random Writes LevelDB Kyoto TreeDB SQLite3 Monday, December 9, 13 164,000 ops/sec 88,500 ops/sec 9,860 ops/sec
  • 18. HBase and HDFS (in April 2012) Details of this experiment: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html Monday, December 9, 13
  • 19. HBase and HDFS (in April 2012) Random Reads Details of this experiment: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html Monday, December 9, 13
  • 20. HBase and HDFS (in April 2012) Random Reads HDFS (1 node) HBase (1 node) 93,000 ops/sec 35,000 ops/sec Details of this experiment: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html Monday, December 9, 13
  • 21. Log Structured Merge Architecture Read Write data in RAM Monday, December 9, 13
  • 22. Log Structured Merge Architecture Write Request from Application Read Write data in RAM Monday, December 9, 13
  • 23. Log Structured Merge Architecture Write Request from Application Read Write data in RAM Monday, December 9, 13
  • 24. Log Structured Merge Architecture Write Request from Application Read Write data in RAM Monday, December 9, 13
  • 25. Log Structured Merge Architecture Write Request from Application Read Write data in RAM Transaction log Monday, December 9, 13
  • 26. Log Structured Merge Architecture Write Request from Application Read Write data in RAM Transaction log Monday, December 9, 13
  • 27. Log Structured Merge Architecture Write Request from Application Read Write data in RAM Transaction log Monday, December 9, 13
  • 28. Log Structured Merge Architecture Write Request from Application Read Write data in RAM Read Only data in RAM on disk Monday, December 9, 13 Transaction log
  • 29. Log Structured Merge Architecture Write Request from Application Periodic Compaction Read Write data in RAM Read Only data in RAM on disk Monday, December 9, 13 Transaction log
  • 30. Log Structured Merge Architecture Scan Request from Application Periodic Compaction Read Write data in RAM Read Only data in RAM on disk Monday, December 9, 13 Write Request from Application Transaction log
  • 31. Leveldb has low write rates Facebook Application 1: • Write rate 2 MB/sec only per machine • Only one cpu was used Monday, December 9, 13
  • 32. Leveldb has low write rates Facebook Application 1: • Write rate 2 MB/sec only per machine • Only one cpu was used We developed multithreaded compaction 10x improvement on write rate Monday, December 9, 13 + 100% of cpus are in use
  • 33. Leveldb has stalls Facebook Feed: • P99 latencies were tens of seconds • Single-threaded compaction Monday, December 9, 13
  • 34. Leveldb has stalls Facebook Feed: • P99 latencies were tens of seconds • Single-threaded compaction We implemented thread aware compaction Dedicated thread(s) to flush memtable Monday, December 9, 13 Pipelined memtables P99 reduced to less than a second
  • 35. Leveldb has high write amplification • Facebook Application 2: • Level Style Compaction • Write amplification of 70 very high Monday, December 9, 13
  • 36. Leveldb has high write amplification • Facebook Application 2: • Level Style Compaction • Write amplification of 70 very high Level-0 5 bytes Level-1 6 bytes 11 bytes Level-2 10 bytes 10 bytes 10 bytes Stage 1 Stage 2 Stage 3 Two compactions by LevelDB Style Compaction Monday, December 9, 13
  • 37. Our solution: lower write amplification • Facebook Application 2: • We implemented Universal Style Compaction • Start from newest file, include next file in candidate set if • Candidate set size >= size of next file Monday, December 9, 13
  • 38. Our solution: lower write amplification • Facebook Application 2: • We implemented Universal Style Compaction • Start from newest file, include next file in candidate set if • Candidate set size >= size of next file Level-0 5bytes Level-1 6 bytes Level-2 10 bytes 10 bytes Stage 1 Stage 2 Single compaction by Universal Style Compaction Write amplification reduced to <10 Monday, December 9, 13
  • 39. Leveldb has high read amplification Monday, December 9, 13
  • 40. Leveldb has high read amplification • Secondary Index Service: • Leveldb does not use blooms for scans Monday, December 9, 13
  • 41. Leveldb has high read amplification • Secondary Index Service: • Leveldb does not use blooms for scans • We implemented prefix scans • Range scans within same key prefix • Blooms created for prefix • Reduces read amplification Monday, December 9, 13
  • 42. Leveldb: read modify write = 2X IOs Monday, December 9, 13
  • 43. Leveldb: read modify write = 2X IOs • Counter increments • Get value, value++, Put value • Leveldb uses 2X IOPS Monday, December 9, 13
  • 44. Leveldb: read modify write = 2X IOs • Counter increments • Get value, value++, Put value • Leveldb uses 2X IOPS • We implemented MergeRecord • Put “++” operation in MergeRecord • Background compaction merges all MergeRecords • Uses only 1X IOPS Monday, December 9, 13
  • 45. Leveldb has a Rigid Design Monday, December 9, 13
  • 46. Leveldb has a Rigid Design • LevelDB Design • Cannot tune system, fixed file sizes Monday, December 9, 13
  • 47. Leveldb has a Rigid Design • LevelDB Design • Cannot tune system, fixed file sizes • We wanted a pluggable architecture • Pluggable compaction filter, e.g. TimeToLive • Pluggable memtable/sstable for RAM/Flash • Pluggable Compaction Algorithm Monday, December 9, 13
  • 48. The Changes we did to LevelDB Monday, December 9, 13
  • 49. The Changes we did to LevelDB Inherited from LevelDB • Log Structured Merge DB • Gets/Puts/Scans of keys • Forward and Reverse Iteration Monday, December 9, 13
  • 50. The Changes we did to LevelDB Inherited from LevelDB • Log Structured Merge DB • Gets/Puts/Scans of keys • Forward and Reverse Iteration Monday, December 9, 13 RocksDB • 10X higher write rate • Fewer stalls • 7x lower write amplification • Blooms for range scans • Ability to avoid read-modify-write • Optimizations for flash or RAM • And many more…
  • 51. RocksDB is born! • Key-Value persistent store • Embedded • Optimized for fast storage • Server workloads Monday, December 9, 13
  • 52. RocksDB is born! • Key-Value persistent store • Embedded • Optimized for fast storage • Server workloads Monday, December 9, 13
  • 53. What is it not? • Not distributed • No failover • Not highly-available, if machine dies you lose your data Monday, December 9, 13
  • 54. What is it not? • Not distributed • No failover • Not highly-available, if machine dies you lose your data Monday, December 9, 13
  • 55. RocksDB API ▪ Keys and values are arbitrary byte arrays. ▪ Data is stored sorted by key. ▪ The basic operations are Put(key,value), Get(key), Delete(key) and Merge(key, delta) ▪ Forward and backward iteration is supported over the data. Monday, December 9, 13
  • 58. RocksDB Architecture Write Request Active MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 59. RocksDB Architecture Memory Write Request Active MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 60. RocksDB Architecture Memory Write Request Persistent Storage Active MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 61. Log Structured Merge Tree -- Writes ▪ Log Structured Merge Tree ▪ New Puts are written to memory and optionally to transaction log ▪ Also can specify log write sync option for each individual write ▪ We say RocksDB is optimized for writes, what does this mean? Monday, December 9, 13
  • 63. RocksDB Write Path Write Request Active MemTable log Switch Switch ReadOnly MemTable log log LSM Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 64. Log Structured Merge Tree -- Reads ▪ Data could be in memory or on disk ▪ Consult multiple files to find the latest instance of the key ▪ Use bloom filters to reduce IO Monday, December 9, 13
  • 66. RocksDB Read Path Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
  • 67. RocksDB Read Path Memory Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
  • 68. RocksDB Read Path Memory Persistent Storage Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
  • 69. RocksDB: Open & Pluggable Customizable WAL Blooms Pluggable Memtable format in RAM Monday, December 9, 13
  • 70. RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Monday, December 9, 13
  • 71. RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Monday, December 9, 13
  • 72. RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Monday, December 9, 13
  • 73. RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Transaction log Monday, December 9, 13
  • 74. RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Transaction log Monday, December 9, 13
  • 75. RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Transaction log Monday, December 9, 13
  • 76. RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Memtable format in RAM Pluggable sst data format on storage Monday, December 9, 13 Transaction log
  • 77. RocksDB: Open & Pluggable Write Request from Application Customizable WAL Blooms Pluggable Compaction Pluggable Memtable format in RAM Pluggable sst data format on storage Monday, December 9, 13 Transaction log
  • 78. RocksDB: Open & Pluggable Get or Scan Request from Application Write Request from Application Customizable WAL Blooms Pluggable Compaction Pluggable Memtable format in RAM Pluggable sst data format on storage Monday, December 9, 13 Transaction log
  • 79. Example: Customizable WALogging • In-house Replication solution wants to be able to embed arbitrary blob in the rocksdb WAL stream for log annotation • Use Case: Indicate where a log record came from in multi-master replication • Solution: Monday, December 9, 13 A Put that only speaks to the log
  • 80. Example: Customizable WALogging Active MemTable k1 Replication Layer In one write batch: PutLogData(“I came from Mars”) Put(k1,v1) Monday, December 9, 13 v1 log “I came from Mars” k1/v1
  • 81. Example: Customizable WALogging Active MemTable Write Request Replication Layer In one write batch: PutLogData(“I came from Mars”) Put(k1,v1) Monday, December 9, 13 k1 v1 log “I came from Mars” k1/v1
  • 82. Example: Pluggable SST format • One Facebook use case needs extreme fast response but could tolerate some loss of durability • Quick hack: mount sst in tmpfs • Still not performant: • existing sst format is block based • Solution: A much simpler format that just stores sorted key/value pairs sequentially • no blocks, no caching, mmap the whole file • build efficient lookup index on load Monday, December 9, 13
  • 83. Example: Blooms for MemTable • Same use case, after we optimized sst access, memtable lookup becomes a major cost in query • Problem: Get needs to go through the memtable lookups that eventually return no data • Solution: Monday, December 9, 13 Just add a bloom filter to memtable!
  • 85. RocksDB Read Path Blooms Blooms Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
  • 86. RocksDB Read Path Memory Blooms Blooms Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
  • 87. RocksDB Read Path Memory Blooms Blooms Persistent Storage Active MemTable log ReadOnly MemTable log log Read Request LSM Flush sst sst sst sst sst sst d Blooms Monday, December 9, 13 Compaction
  • 88. Example: Pluggable memtable format • Another Facebook use case has a distinct load phase where no query is issued. • Problem: write throughput is limited by single writer thread • Solution: A new memtable representation that does not keep keys sorted Monday, December 9, 13
  • 89. Example: Pluggable memtable format Unsorted MemTable log Switch Switch ReadOnly MemTable log log LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 90. Example: Pluggable memtable format Write Request Unsorted MemTable log Switch Switch ReadOnly MemTable log log LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 91. Example: Pluggable memtable format Write Request Unsorted MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 92. Example: Pluggable memtable format Memory Write Request Unsorted MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 93. Example: Pluggable memtable format Memory Write Request Persistent Storage Unsorted MemTable log Switch Switch ReadOnly MemTable log log Read Request LSM Sort, Flush sst sst sst sst sst sst d Monday, December 9, 13 Compaction
  • 94. Possible workloads for RocksDB? ▪ Serving data to users via a website ▪ A spam detection backend that needs fast access to data ▪ A graph search query that needs to scan a dataset in realtime ▪ Distributed Configuration Management Systems ▪ Fast serve of Hive Data ▪ A Queue that needs a high rate of inserts and deletes Monday, December 9, 13
  • 96. Futures • Scale linearly with number of cpus • 32, 64 or higher core machines • ARM processors Monday, December 9, 13
  • 97. Futures • Scale linearly with number of cpus • 32, 64 or higher core machines • ARM processors • Scale linearly with storage iops • Striped flash cards • RAM & NVRAM storage Monday, December 9, 13
  • 98. Come Hack with us Monday, December 9, 13
  • 99. Come Hack with us • RocksDB is Open Sourced • http://rocksdb.org • Developers group https://www.facebook.com/groups/rocksdb.dev/ Monday, December 9, 13
  • 100. Come Hack with us • RocksDB is Open Sourced • http://rocksdb.org • Developers group https://www.facebook.com/groups/rocksdb.dev/ • Help us HACK RocksDB Monday, December 9, 13