HBase internals
3 likes1,419 views
This document summarizes a presentation about HBase storage internals and future developments. It discusses how HBase provides random read/write access on HDFS using tables, regions, and region servers. It describes the write path involving the client, master, and region servers as well as the read path. It also covers topics like snapshots, compactions, and future plans to improve encryption, security, write-ahead logs, and compaction policies.
![What is HBase?
• Open source Storage Manager that provides random
read/write on top of HDFS
• Provides Tables with a “Key:Column/Value” interface
• Dynamic columns (qualifiers), no schema needed
• “Fixed” column groups (families)
• table[row:family:column] = value
2](https://image.slidesharecdn.com/hbase-96-internals-130324060009-phpapp02/85/HBase-internals-2-320.jpg)
![The Read Path – Append Only to Random R/W
Each flush a new file is created
Key0 – value 0.1
•
Key0 – value 0.0
Key2 – value 2.0 Key5 – value 5.0
Key3 – value 3.0 Key1 – value 1.0
Key5 – value 5.0 Key5 – [deleted]
Each file have KeyValues sorted by key
Key6 – value 6.0
•
Key8 – value 8.0
Key9 – value 9.0 Key7– value 7.0
• Two or more files can contains the same key
(updates/deletes)
• To find a Key you need to scan all the files
• …with some optimizations
• Filter Files Start/End Key
• Having a bloom filter on each file
11](https://image.slidesharecdn.com/hbase-96-internals-130324060009-phpapp02/85/HBase-internals-11-320.jpg)
![HFile Format
• Only Sequential Writes, just append(key, value) Blocks
Header
• Large Sequential Reads are better
Record 0
• Why grouping records in blocks? Record 1
…
• Easy to split Record N
• Easy to read Key/Value Header
(record) Record 0
• Easy to cache Key Length : int Record 1
Value Length : int …
• Easy to index (if records are sorted) Key : byte[]
Record N
• Block Compression (snappy, lz4, gz, …) Index 0
…
Value : byte[]
Index N
Trailer
13](https://image.slidesharecdn.com/hbase-96-internals-130324060009-phpapp02/85/HBase-internals-13-320.jpg)
![Data Block Encoding
• “Be aware of the data”
• Block Encoding allows to compress the Key based on what we know
• Keys are sorted… prefix may be similar in most cases
• One file contains keys from one Family only
“on-disk”
• Timestamps are “similar”, we can store the diff KeyValue
• Type is “put” most of the time… Row Length : short
Row : byte[]
Family Length : byte
Family : byte[]
Qualifier : byte[]
Timestamp : long
Type : byte
14](https://image.slidesharecdn.com/hbase-96-internals-130324060009-phpapp02/85/HBase-internals-14-320.jpg)
![Compactions
Reduce the number of files to look into during a scan
Key0 – value 0.1
•
Key0 – value 0.0
Key2 – value 2.0 Key1 – value 1.0
Key3 – value 3.0 Key4– value 4.0
Key5 – value 5.0 Key5 – [deleted]
• Removing duplicated keys (updated values)
Key8 – value 8.0 Key6 – value 6.0
Key9 – value 9.0 Key7– value 7.0
• Removing deleted keys
• Creates a new file by merging the content of two or more files Key0 – value 0.1
Key1 – value 1.0
Key2 – value 2.0
Key4– value 4.0
• Remove the old files Key6 – value 6.0
Key7– value 7.0
Key8– value 8.0
Key9– value 9.0
16](https://image.slidesharecdn.com/hbase-96-internals-130324060009-phpapp02/85/HBase-internals-16-320.jpg)
![Pluggable Compactions
Try different algorithm
Key0 – value 0.1
•
Key0 – value 0.0
Key2 – value 2.0 Key1 – value 1.0
Key3 – value 3.0 Key4– value 4.0
Key5 – value 5.0 Key5 – [deleted]
Be aware of the data
Key6 – value 6.0
•
Key8 – value 8.0
Key9 – value 9.0 Key7– value 7.0
• Time Series? I guess no updates from the 80s
• Be aware of the requests Key0 – value 0.1
Key1 – value 1.0
Key2 – value 2.0
Key4– value 4.0
• Compact based on statistics Key6 – value 6.0
Key7– value 7.0
Key8– value 8.0
Key9– value 9.0
• which files are hot and which are not
• which keys are hot and which are not
17](https://image.slidesharecdn.com/hbase-96-internals-130324060009-phpapp02/85/HBase-internals-17-320.jpg)
HBase internals
- 1. HBase Storage Internals, present and future! Matteo Bertozzi | @Cloudera Speaker Name or Subhead Goes Here March 2013 - Hadoop Summit Europe 1
- 2. What is HBase? • Open source Storage Manager that provides random read/write on top of HDFS • Provides Tables with a “Key:Column/Value” interface • Dynamic columns (qualifiers), no schema needed • “Fixed” column groups (families) • table[row:family:column] = value 2
- 3. HBase ecosystem • Apache Hadoop HDFS for data durability and reliability (Write-Ahead Log) • Apache ZooKeeper for distributed coordination App MR • Apache Hadoop MapReduce built-in support for running MapReduce jobs ZK HDFS 3
- 4. How HBase Works “View from 10000ft” 4
- 5. Master, Region Servers and Regions Client • Region Server • Server that contains a set of Regions • Responsible to handle reads and writes ZooKeeper • Region Master • The basic unit of scalability in HBase • Subset of the table’s data • Contiguous, sorted range of rows stored together. Region Server Region Server Region Server • Master Region Region Region • Coordinates the HBase Cluster Region Region Region • Assignment/Balancing of the Regions Region Region Region • Handles admin operations • create/delete/modify table, … HDFS 5
- 6. Autosharding and .META. table • A Region is a Subset of the table’s data • When there is too much data in a Region… • a split is triggered, creating 2 regions • The association “Region -> Server” is stored in a System Table • The Location of .META. Is stored in ZooKeeper Table Start Key Region ID Region Server machine01 Region 1 - testTable testTable Key-00 1 machine01.host Region 4 - testTable testTable Key-31 2 machine03.host machine02 testTable Key-65 3 machine02.host Region 3 - testTable testTable Key-83 4 machine01.host Region 1 - users … … … … machine03 users Key-AB 1 machine03.host Region 2 - testTable users Key-KG 2 machine02.host Region 2 - users 6
- 7. The Write Path – Create a New Table • The client asks to the master to create a new Table Client • hbase> create ‘myTable’, ‘cf’ createTable() • The Master Master • Store the Table information (“schema”) Store Table “Metadata” • Create Regions based on the key-splits provided Assign the Regions • no splits provided, one single region by default “enable” • Assign the Regions to the Region Servers Region Region Region • The assignment Region -> Server Server Server Server Region is written to a system table called “.META.” Region Region Region Region Region 7
- 8. The Write Path – “Inserting” data • table.put(row-key:family:column, value) Client Where is • The client asks ZooKeeper the location of .META. .META.? Scan .META. • The client scans .META. searching for the Region Server ZooKeeper Region Server Region responsible to handle the Key Insert KeyValue Region • The client asks the Region Server to insert/update/delete Region Server the specified key/value. Region • The Region Server process the request and dispatch it to Region Region the Region responsible to handle the Key • The operation is written to a Write-Ahead Log (WAL) • …and the KeyValues added to the Store: “MemStore” 8
- 9. The Write Path – Append Only to Random R/W • Files in HDFS are RS Region Region Region WAL • Append-Only • Immutable once closed MemStore + Store Files (HFiles) • HBase provides Random Writes? • …not really from a storage point of view • KeyValues are stored in memory and written to disk on pressure • Don’t worry your data is safe in the WAL! Key0 – value 0 • (The Region Server can recover data from the WAL is case of crash) Key1 – value 1 Key2 – value 2 Key3 – value 3 • But this allow to sort data by Key before writing on disk Key4 – value 4 Key5 – value 5 • Deletes are like Inserts but with a “remove me flag” Store Files 9
- 10. The Read Path – “reading” data • The client asks ZooKeeper the location of .META. Client Where is • The client scans .META. searching for the Region Server .META.? Scan .META. responsible to handle the Key ZooKeeper Region Server Region • The client asks the Region Server to get the specified key/value. Get Key Region • The Region Server process the request and dispatch it to the Region Server Region responsible to handle the Key Region • MemStore and Store Files are scanned to find the key Region Region 10
- 11. The Read Path – Append Only to Random R/W Each flush a new file is created Key0 – value 0.1 • Key0 – value 0.0 Key2 – value 2.0 Key5 – value 5.0 Key3 – value 3.0 Key1 – value 1.0 Key5 – value 5.0 Key5 – [deleted] Each file have KeyValues sorted by key Key6 – value 6.0 • Key8 – value 8.0 Key9 – value 9.0 Key7– value 7.0 • Two or more files can contains the same key (updates/deletes) • To find a Key you need to scan all the files • …with some optimizations • Filter Files Start/End Key • Having a bloom filter on each file 11
- 12. HFile HBase Store File Format 12
- 13. HFile Format • Only Sequential Writes, just append(key, value) Blocks Header • Large Sequential Reads are better Record 0 • Why grouping records in blocks? Record 1 … • Easy to split Record N • Easy to read Key/Value Header (record) Record 0 • Easy to cache Key Length : int Record 1 Value Length : int … • Easy to index (if records are sorted) Key : byte[] Record N • Block Compression (snappy, lz4, gz, …) Index 0 … Value : byte[] Index N Trailer 13
- 14. Data Block Encoding • “Be aware of the data” • Block Encoding allows to compress the Key based on what we know • Keys are sorted… prefix may be similar in most cases • One file contains keys from one Family only “on-disk” • Timestamps are “similar”, we can store the diff KeyValue • Type is “put” most of the time… Row Length : short Row : byte[] Family Length : byte Family : byte[] Qualifier : byte[] Timestamp : long Type : byte 14
- 15. Compactions Optimize the read-path 15
- 16. Compactions Reduce the number of files to look into during a scan Key0 – value 0.1 • Key0 – value 0.0 Key2 – value 2.0 Key1 – value 1.0 Key3 – value 3.0 Key4– value 4.0 Key5 – value 5.0 Key5 – [deleted] • Removing duplicated keys (updated values) Key8 – value 8.0 Key6 – value 6.0 Key9 – value 9.0 Key7– value 7.0 • Removing deleted keys • Creates a new file by merging the content of two or more files Key0 – value 0.1 Key1 – value 1.0 Key2 – value 2.0 Key4– value 4.0 • Remove the old files Key6 – value 6.0 Key7– value 7.0 Key8– value 8.0 Key9– value 9.0 16
- 17. Pluggable Compactions Try different algorithm Key0 – value 0.1 • Key0 – value 0.0 Key2 – value 2.0 Key1 – value 1.0 Key3 – value 3.0 Key4– value 4.0 Key5 – value 5.0 Key5 – [deleted] Be aware of the data Key6 – value 6.0 • Key8 – value 8.0 Key9 – value 9.0 Key7– value 7.0 • Time Series? I guess no updates from the 80s • Be aware of the requests Key0 – value 0.1 Key1 – value 1.0 Key2 – value 2.0 Key4– value 4.0 • Compact based on statistics Key6 – value 6.0 Key7– value 7.0 Key8– value 8.0 Key9– value 9.0 • which files are hot and which are not • which keys are hot and which are not 17
- 18. Snapshots Zero-Copy Snapshots and Table Clones 18
- 19. What Is a Snapshot? • “a Snapshot is not a copy of the table” • a Snapshot is a set of metadata information • The table “schema” (column families and attributes) • The Regions information (start key, end key, …) • The list of Store Files ZK ZK • The list of WALs active Master ZK RS RS Region Region Region Region Region Region WAL WAL Store Files (HFiles) Store Files (HFiles) 19
- 20. How Taking a Snapshot Works? • The master orchestrate the RSs • the communication is done via ZooKeeper • using a “2-phase commit like” transaction (prepare/commit) • Each RS is responsible to take its “piece” of snapshot • For each Region store the metadata information needed • (list of Store Files, WALs, region start/end keys, …) ZK ZK Master ZK RS RS Region Region Region Region Region Region WAL WAL Store Files (HFiles) Store Files (HFiles) 20
- 21. Cloning a Table from a Snapshots • hbase> clone_snapshot ‘snapshotName’, ‘tableName’ … • Creates a new table with the data “contained” in the snapshot • No data copies involved • HFiles are immutable, and shared between tables and snapshots • You can insert/update/remove data from the new table • No repercussions on the snapshot, original tables or other cloned tables 21
- 22. Compactions & Archiving • HFiles are immutable, and shared between tables and snapshots • On compaction or table deletion, files are removed from disk • If one of these files are referenced by a snapshot or a cloned table • The file is moved to an “archive” directory • And deleted later, when there’re no references to it 22
- 23. Future What can be improved? 23
- 24. 0.96 is coming up • Moving RPC to Protobuf • Allows rolling upgrades with no surprises • HBase Snapshots • Pluggable Compactions • Remove -ROOT- • Table Locks 24
- 25. 0.98 and Beyond • Transparent Table/Column-Family Encryption • Cell-level security • Multiple WALs per Region Server (MTTR) • Data Placement Awareness (MTTR) • Data Type Awareness • Compaction policies, based on the data needs • Managing blocks directly (instead of files) 25
- 26. DO NOT USE PUBLICLY PRIOR TO 10/23/12 Questions? Headline Goes Here Matteo Name or @Cloudera SpeakerBertozzi | Subhead Goes Here 26
- 27. 27