Getting Started with HBase

© 2014 MapR Techno©lo 2g0ie1s4 MapR Technologies 1
Getting Started with HBase Application development
http://answers.mapr.com/

© 2014 MapR Technologies 2
Objectives of this session
• Why do we need NoSQL / HBase?
• Quick overview of HBase & HBase data model
• Design considerations when migrating from RDBMS to HBase
• How to work around transactions that span multiple rows

Why do we need NoSQL / HBase?
Relational database model
Relational Data is typed and structured before stored:
– Entities map to tables, normalized
– Structured Query Language
• Joins tables to bring back data

Why do we need NoSQL / HBase?
Relational Model
• Pros
– Standard persistence model
• standard language for data manipulation
– Transactions handle concurrency ,
consistency
– efficient and robust structure for storing
data

What changed to bring on NoSQL?
Lots of data & the need to scale
With the internet, increase in traffic and data, came the need to scale
• Vertical Scaling is expensive and has limits
• Not all use cases require ACID transactions
Vertical scale = big box
$$

What changed to bring on NoSQL?
Lots of data, the need to scale horizontally
• Horizontal scaling
– Cheaper than vertical
– parallel execution
– high reliability (doesn’t depend on one box)
• Relational databases were not designed to do this automatically
Horizonal scale:
Split table by rows into
partitions across a cluster
Key colB colC
val val val
xxx val val
id 1-1000
id 1000-2000
id 2000=3000
Key colB colC
val val val
xxx val val
Key colB colC
val val val
xxx val val

Facebook 2010
•9000 memcache instances
•4000 Shards mysql
Lots of people tried to spread
databases across a cluster.
http://gigaom.com/2011/07/07/facebook-trapped-in-mysql-fate-worse-than-death/

What changed to bring on NoSQL? Big data
• Cons of the Relational Model:
– Does not scale horizontally:
• Sharding is difficult to manage
• Distributed join, transactions do not scale across shards
Horizonal scale : partition or shard tables across cluster
Distributed Joins, Transactions are Expensive
bottleneck

Google File System MapReduce
• Distributed Storage System
• Designed to scale
• Paper published in 2006.
Big Table
Distributed File Model Distributed Compute Model

Google
Big Table
Crawlers
MapReduce
Indexing Queries
Indexes
CF
colA colB colC
val val
val
Metadata
MapReduce
metadata

Distributed
File System
(HDFS)
Map
Reduce

HBase is a Distributed Database
Data is automatically
distributed across the cluster.
• Row is indexed by a row
key
• Key range is used for
horizontal partitioning
• Table splits happen
automatically as the data
grows
Key
Range
xxxx
xxxx
CF1
colA colB colC
val val
val
CF2
colA colB colC
val val
val
Key
Range
xxxx
xxxx
CF1
colA colB colC
val val
val
CF2
colA colB colC
val val
val
Key
Range
xxxx
xxxx
CF1
colA colB colC
val val
val
CF2
colA colB colC
val val
val
Put, Get by Key

HBase is a ColumnFamily oriented Database
Data is accessed and stored together:
• Row is indexed by a row key
• Similar Data is grouped & stored in Column Families
– share common properties:
• Number of versions
• Time to Live (TTL)
• Compression [lz4, lzf, Zlib]
• In memory option …
CF1
colA colB colC
Val val
val
CF2
colA colB colC
val val
val
RowKey
axxx
gxxx
Customer id Customer Address data Customer order data

HBase designed for Distribution
• distributed data stored and accessed together:
– Key range is used for horizontal partitioning
• Pros
– scalable handles data volume and velocity
– Fast Writes and Reads by Key
• Cons
– No joins
– No dynamic queries
– Need to know how data will be queried in advance to
achieve best schema design
Key
Range
axxx
kxxx
Key
Range
axxx
kxxx
Key
Range
axxx
kxxx

© 2014 © 201 M4 aMpaRp RTe Tcehcnhonloogloiegsies 15
Introduction to Hbase, data model and
Architecture
– Understand how the data flows for writes and reads

HBase Data Model
Row Keys: identify the rows in an HBase table
Columns are grouped into column families
– can contain an arbitrary number of columns.
Row
Key
CF1 CF2 …
colA colB colC colA colB colC colD
R1
axxx val val val val
…
gxxx val val val val
R2
hxxx val val val val val val val
…
jxxx val
R3
kxxx val val val val
…
rxxx val val val val val val

HBase Data Storage - Cells
• Data is stored in Key value format
• Value for each cell is specified by complete coordinates:
• (Row key, ColumnFamily, Column Qualifier, timestamp ) => Value
– RowKey:CF:Col:Version:Value
– smithj:data:city:1391813876369:nashville
Cell Coordinates= Key
Row key Column Family Column Qualifier Timestamp Value
Smithj data city 1391813876369 nashville
Column Key
Value

Logical Data Model vs Physical Data Storage
• Data is stored in Key Value format
• Key Value is stored for each Cell
• Column families data are stored in
separate files
RowK
ey
CF1 CF2
colA colB colA colC
ra 1 2
rxxxx
rxxx
Logical Model
Row
Key
CF1:Col version value
ra cf1:cola 1 1
row
Key
CF2:Col version value
ra cf2:cola 1 2
Physical Storage
Key Value Key Value
Physical Storage

Sparse Data with Cell Versions
CF1:colA CF1:colB CF1:colC
Row1
Row10
Row11
Row2
@time1:
value1
@time5:
value2
@time7:
value3
@time2:
value1
@time3:
value1
@time4:
value1
@time2:
value1
@time4:
value1
@time6:
value2

Versioned Data
• Version
– each put, delete adds new cell, new version
– A long
• by default the current time in milliseconds if no version specified
– Last 3 versions are stored by default
• Configurable by column family
– You can delete specific cell versions
– When a cell exceeds the maximum number of versions, the extra
records are removed
Key CF1:Col version value
ra cf1:cola 3 3
ra cf1:cola 2 2
ra cf1:cola 1 1

Table Physical View
Physically data is stored per Column family as a sorted map
• Ordered by row key, column qualifier in ascending order
• Ordered by timestamp in descending order
Row
key
Column
qualifier
Cell
value
Timestamp
(long)
Row1 CF1:colA value3 time7
Row 10 CF1:colB value1 time4
Sorted by
Row key
and Column
Sorted in
descending
order

Logical Data Model vs Physical Data Storage
ra cf1:ca 1 1
rb cf1:cb 2 4
rb cf1:cb 1 3
rc cf1:ca 1 5
Row
Key
CF1 CF2
ca cb ca cd
ra 1 2
rb 3,4
rc 5 6,7 8
ra cf2:ca 1 2
rc cf2:ca 2 7
rc cf2:ca 1 6
rc cf2:cd 1 8
Physical Storage
Logical
Model
Column families are stored
separately
Row keys, Qualifiers are sorted
lexicographically
Key Value Key Value

HBase Table is a Sorted map of maps
SortedMap<Key, Value>
Table
Map of Rows
Map of CF
Map of columns
Map of cells
ra cf1:ca v1 1
rb cf1:cb v2 4
rb cf1:cb v1 3
rc cf1:ca v1 5
ra cf2:ca v1 2
rc cf2:ca v2 7
rc cf2:ca v1 6
rc cf2:cd v1 8
SortedMap<RowKey,
SortedMap< ColumnFamily,
SortedMap< ColumnName,
SortedMap < version, Value> >>>

HBase Table SortedMap<Key, Value>
<ra,<cf1, <ca, <v1, 1>>
<cf2, <ca, <v1, 2>>>
<rb,<cf1, <cb, <v2, 4>
<v1, 3>>>
<rc,<cf1, <ca, <v1, 5>>
<cf2, <ca, <v2, 7>>
<ca, <v1, 6>>
<cd, <v1, 8>>>
ra cf1:ca v1 1
rb cf1:cb v2 4
rb cf1:cb v1 3
rc cf1:ca v1 5
ra cf2:ca v1 2
rc cf2:ca v2 7
rc cf2:ca v1 6
rc cf2:cd v1 8

Basic Table Operations
• Create Table, define Column Families before data is
imported
– but not the rows keys or number/names of columns
• Low level API, technically more demanding
• Basic data access operations (CRUD):
put Inserts data into rows (both add and update)
get Accesses data from one row
scan Accesses data from a range of rows
delete Delete a row or a range of rows or columns

Hbase Architecture

What is a Region?
• Tables are partitioned into key ranges (regions)
• Region servers serve data for reads and writes
– For the range of keys it is responsible for
Region Server
Client
Region Region
HMaster
zookeepe
zozoorkoekeepeeprer
Region Server
Region Region
Get
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val

Region Server Components
• WAL: write ahead log on disk (commit log), Used for recovery
• BlockCache: Read Cache, LRU, Least Recently Used evicted
• MemStore: Write Cache, sorted map of keyValue updates.
– 1 memstore per column family per region
• Hfile=sorted KeyValues on disk
Region
Server
HDFS Data Node
BlockCache
memstore
HFile
memstore
HFile
Region Region
HFile HFile
memstore memstore
WAL

HBase Write Steps
Put
each incoming record written
to WAL for durability:
• log on disk
• updates appended sequentially
HDFS Data Node
memstore memstore
Region Server
Region
WAL

HBase Write Steps
Put
Next updates are written to the
Memstore:
• write cache
• in-memory
• sorted list of KeyValue updates HDFS Data Node
memstore memstore
Region Server
Region
WAL
Ack
Updates quickly sorted in memory are available to queries after put returns

HBase Memstore
• in-memory
• sorted list of Key → Value
• One per column family
• Updates quickly sorted in memory
memstore memstore
Region
ra cf1:ca v1 1
rb cf1:cb v2 4
rb cf1:cb v1 3
rc cf1:ca v1 5
ra cf2:ca v1 2
rc cf2:ca v2 7
rc cf2:ca v1 6
rc cf2:cd v1 8
Key Value Key Value

HBase Region Flush
When 1 Memstore is full:
• all memstores in region flushed
to new Hfiles on disk
• Hfile: sorted list of key → values
On disk
HDFS Data Node
memstore memstore
Region Server
Region
WAL
HFile HFile
FLUSH
HHFFileile

HBase HFile
• On disk sorted list of key → values
• One per column family
• Flushed quickly to file
• Sequential write HDFS Data Node
HFile HFile
Sequential write
ra cf1:ca v1 1
rb cf1:cb v2 4
rb cf1:cb v1 3
rc cf1:ca v1 5
ra cf2:ca v1 2
rc cf2:ca v2 7
rc cf2:ca v1 6
rc cf2:cd v1 8
Key Value Key Value

HBase HFile Structure
• Memstore flushes to an Hfile
64Kbyte blocks
are compressed
Key-value
pairs are
stored in
increasing
order
Index points to
row keys
location
B+-tree: leaf
index , root index
Key A Value
…
…
…
Key P Value
…..
….
…
Key T Value
…..
….
…
Key z Value
…..
….
…
Root Index
Bloom
Leaf Index
Leaf Index
Leaf Index
Leaf Index
Leaf Index
Leaf Index
Root Index
Interm Index
Bloom
Trailer
Bloom
Bloom
Intermediate Index
Bloom
Bloom
Bloom
Bloom

HBase Read Merge from Memory and Files
• MemStore creates multiple small store files over time when flushing.
• Read Amplification: When a get/scan comes in, multiple files have to be
examined
HDFS Data Node
BlockCache
memstore
Region Server Region
WAL HFile
HFile
HFile
scanner
read Get or Scan searches for
Row Cell KeyValues:
1. Block Cache ((Memory)
2. Memstore (Memory)
3. Load HFiles from Disk
into Block Cache based on
indexes and bloomfilters
1
2
3

HDFS Data Node
HBase Compaction • minor compaction:
• merges files into fewer larger ones.
• Major compaction:
• merge all Hfiles into one per column
family.
• remove cells marked for
deletion
Region Server
Region Region
memstore memstore
WAL minor
compaction
HFile
HFile
HFile
HFile
HFile
HFile
HFile
HFile
HFile
HFile
HFile
HFile
HFile
updates
HFile
major
compaction
Flush to disk

HBase Background: Log-Structured Merge Trees
• Traditional Databases use B+ trees:
– expensive to update
• HBase: Log Structured Merge Trees
– transforms random writes into
sequential writes
• Writes go to memory
– And WAL
• memstore flushes to disk
– Reads
• From memory, index, sorted disk
predictable disk seeks

Data Model for Fast Writes, Reads
• Predictable disk lay out
• Minimize disk seek
• Get, Put by row key: primary index, fast random access
• Scan by row key range: stored sorted, efficient
sequential access for key range
Region1
Key Range
ra
rx
Region
Region
Region
Server
ra cf1:ca v1 1
rb cf1:cb v2 4
rb cf1:cb v1 3
rc cf1:ca v1 5
Get key
Scan start key,
Stop key
Minimize disk seek

Region = contiguous keys
• Regions fundamental partitioning/sharding object.
• By default, on table creation 1 region is created that holds the
entire key range.
• When region becomes too large, splits into two child regions.
• Typical region size is a few GB, sometimes even 10G or 20G
Region
CF1
colA colB colC
val val
val
CF2
colA colB colC
val val
val
Key
Range
axxx
gxxx
Region

Region Split
• The RegionServer splits a
region
• daughter regions
– represent 1/2 of the
original region
– each with half of the
original regions keys.
– opened in parallel on
same server
• reports the split to the Master
Region 1 Region 2
Region Server 1
Key colB colC
val val
val
Key colB colC
val val
val
Region Server 1
Region 1
Key
Range
axxx
kxxx
Key
Range
Lxxx
zxxx
Key colB colC
val val
val

HBase Use Cases

3 Main Use Case Categories
• Capturing Incremental data – Hi Volume, Velocity Writes
– Time Series Data, Stuff with a Time Stamp
• Sensor, System Metrics, Events,
• Stock Ticker, User clicks,
HBase
Put
App
Server
App
read Server
Put
Put
Put
Event time stamped
data
sensor
OpenTSDB
Data for real-time monitoring.

HBase
Messages
read
Put
App
Server
read
App
Server
read
App
Server Put
Put
Put
App
Server
• Information Exchange Hi Volume, Velocity Write/Read
• Messaging on Facebook is backed by Hbase
– communication coming from email, SMS, Facebook Chat,
and the Inbox
– https://www.facebook.com/UsingHbase

• Content Serving, Web Application Backend– Hi Volume, Velocity
Reads
– Online Catalog: Intuit Merchant, Gap, World Library Catalog.
– CRM: Salesforce CMS: Lily
– Search Index: ebay, photobucket
– Online Pre-Computed View: Groupon, Pinterest
Hbase
Processed
data
read
App
Server
read
App
Server
read
App
Bulk Import Server

Agenda
• Introduction to HBase data model and Architecture
• Using HBase shell to create tables and insert data
– Demo / Lab using Hbase shell to create tables
• Java API fundamentals to perform CRUD operations
– Demo / Lab using Eclipse, HBase Java API & MapR Sandbox
• Understand how the data flows for writes and reads
• Schema design concepts including rowkey design
• Advanced Java APIs to perform scans and do transactions
– Demo / Lab

Lab Exercise
See Lab_Hbase_Shell.pdf
Start MapR Sandbox and log into the cluster
[user: mapr, passwd: mapr]
Use the HBase shell
>Hbase shell
hbase> help
hbase> create ’/user/mapr/mytable’, {NAME =>’cf1’}
hbase> put ’/user/mapr/mytable’, ’row1’, ’cf1:col1’, ‘datacf1c1v1’
hbase> get ’/user/mapr/mytable’, ’row1’
hbase> scan ’/user/mapr/mytable’
hbase> describe ’/user/mapr/mytable’

© MapR Technologies, confidential
HBase
Java API fundamentals to
perform CRUD operations

Shoppingcart Application Requirements
• Need to create Tables: Shoppingcart & Inventory
• Perform CRUD operations on these tables
– Create, Read, Update, and Delete items from these tables

Inventory & Shoppingcart Tables
Perform checkout operation for Mike
Inventory Table Shoppingcart Table
quantity
Pens 24
Notepads 54
Erasers 15
Pencils 90
pens notepads erasers
Mike 5 5
John 10 15 4
Mary 9 10
Adam 18 7 10
CF “stock " CF “items"

Java API Fundamentals
• CRUD operations
– Get, Put, Delete, Scan, checkAndPut, checkAndDelete, Increment
– KeyValue, Result, Scan – ResultScanner,
– Batch Operations

CRUD Operations Follow A Pattern (mostly)
• common pattern
– Instantiate object for an operation: Put put = new Put(key)
– Add attributes to specify what to insert: put.add(…)
– invoke operation with HTable: myTable.put(put)
// Insert value1 into rowKey in columnFamily:columnName1
Put put = new Put(rowKey);
put.add(columnFamily, columnName1, value1);
myTable.put(put);

erasers notepads pens
Mike 2 5 5
CF “items"
Shoppingcart Table
Shopping Cart Table
Key CF:COL ts value
Mike items:erasers 1391813876369 2
Mike items:notepads 1391813876369 5
Mike items:pens 1391813876369 5
Physical Storage

Put Operation
adding multiple column values to a row
byte [] tableName = Bytes.toBytes("/path/Shopping");
byte [] itemsCF = Bytes.toBytes(“items");
byte [] penCol = Bytes.toBytes (“pens”);
byte [] noteCol = Bytes.toBytes (“notes”);
byte [] eraserCol = Bytes.toBytes (“erasers”);
HTableInterface table = new HTable(hbaseConfig, tableName);
Put put = new Put(“Mike”);
put.add(itemsCF, penCol, Bytes.toBytes(5l));
put.add(itemsCF, noteCol, Bytes.toBytes(5l));
put.add(itemsCF, eraserCol, Bytes.toBytes(2l));
table.put(put);
Key CF:COL ts value

Get Example
byte [] tableName = Bytes.toBytes("/path/Shopping");
byte [] itemsCF = Bytes.toBytes(“items");
byte [] penCol = Bytes.toBytes (“pens”);
HTableInterface table = new HTable(hbaseConfig, tableName);
Get get = new Get(“Mike”);
get.addColumn(itemsCF, penCol);
Result result = myTable.get(get);
byte[] val = result.getValue(itemsCF, penCol);
System.out.println("Value: " + Bytes.toLong(val)); //prints 5
Key CF:COL ts value

Result Class
• A Result instance wraps data from a row returned from a get or a
scan operation. Result wraps KeyValues
• Result toString() looks like this :
keyvalues={Adam/items:erasers/1391813876369/Put/vlen=8/ts=0,
Adam/items:notepads/1391813876369/Put/vlen=8/ts=0,
Adam/items:pens/1391813876369/Put/vlen=8/ts=0}
• The Result object provides methods to return values
byte[] b = result.getValue(columnFamilyName,columnName1);
Items:erasers Items:notepads Items:pens
Adam 10 7 18
http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Result.html

KeyValue – The Fundamental HBase Type
• A KeyValue instance is a cell instance
– Contains Key (cell coordinates) and the Value (data)
• Cell coordinates: Row key, Column family, Column qualifier,
Timestamp
• KeyValue toString() looks like this :
Adam/items:erasers/1391813876369/Put/vlen=8/
Key =Cell Coordinates
Row key Column Family Column Qualifier Timestamp Value
Value
Adam items erasers 1391813876369 10

Bytes class
http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/util/Bytes.html
• org.apache.hadoop.hbase.util.Bytes
• Provides methods to convert Java types to and from byte[] arrays
• Support for
– String, boolean, short, int, long, double, and float
byte[] bytesTable = Bytes.toBytes("Shopping");
String table = Bytes.toString(bytesTable);
byte[] amountBytes = Bytes.toBytes(1000l);
long amount = Bytes.toLong(amount);

Scan Operation – Example
byte[] startRow=Bytes.toBytes(“Adam”);
byte[] stopRow=Bytes.toBytes(“N”);
Scan s = new Scan(startRow, stopRow);
scan.addFamily(columnFamily);
ResultScanner rs = myTable.getScanner(s);

ResultScanner - Example
Resultscanner provides iterator-like functionality
Scan scan = new Scan();
scan.addFamily(columnFamily);
ResultScanner scanner = myTable.getScanner(scan);
try {
for (Result res : scanner) {
System.out.println(res);
}
} catch (Exception e) {
System.out.println(e);
} finally {
scanner.close();
}
Calls scanner.next()
Always put in finally block

Lab Exercise Program Structure
• ShoppingCartApp– main class
• InventoryDAO – A DAO for the Inventory CRUD
functionality
• ShoppingCartDAO – A DAO for the Inventory
CRUD functionality
• Inventory – A Java object that holds data for a
single Inventory row
• ShoppingCart – A Java object that holds data for
a single Inventory row
• MockHtable – in memory test hbase table, allows
to run code, debug without hbase running on a
cluster or vm.

Lab Exercise
• See Lab_3_Java_API.pdf
– Import the project “lab-exercises-shopping” into Eclipse
– Setup creates Inventory and Shoppingcart Tables and inserts data
– Use Get, Put, Scan, and Delete operations

Lab: Import, build
• Download the code
• Import Maven project lab-exercises-shopping
into Eclipse
• Build : Run As -> Maven Install

Lab: run TestInventorySetup JUnit
• Select Test Class, Then Run As -> JUnit Test
• Uses MockHTable https://gist.github.com/agaoglu/613217

© 2014 MapR Techno©lo 2g0ie1s4 MapR Technologies 66
Schema Design Guidelines
• HBase tables ≠ Relational tables!
• Be careful about assumptions you bring from past experience.
• Examples:
– Data normalization not required
– Different atomicity rules

Use Case Example: Record Stock Trade Information
in a Table
• Trade data:
Trade
• timestamp
• stock symbol
• price per share
• volume of trade
• Example
– 1381396363000
(epoch timestamp with
millisecond granularity)
– AMZN
– $304.66
– 1333 shares

Intelligent keys
• Only the row keys are indexed
• Compose the key with attributes used for searching
– Composite key : 2 or more identifying attributes
– Like multi-column index design in RDB
– Use fixed length, or separators
Cell Coordinates (Key)
Granularity
Row key Column Family Column Name Timestamp Value
Restrict disk I/O Restrict network traffic

Composite Keys
Use composite rowkey to bound scan ranges and provide sub-indexing
to cell data.
• Include multiple elements in the rowkey
– Use a separator or fixed length
• Example rowkey format:
– Ex: GOOG_20131012
• Get operations require complete row key.
• Scans can use partial keys.
– Ex: “GOOG” or "GOOG_2014"
SYMBOL + DATE (YYYYMMDD)

Consider Access Patterns for Application
• By date? By hour? By companyId? By PersonId?
– Rowkey design
• What if the Date/Timestamp is leftmost ?
How will data be retrieved?
Key
1391813876369_AMZN
1391813876370_AMZN
1391813876371_GOOG

Hot-Spotting and Region Splits
• If rowkeys are written in sequential order then
writes go to only one server
– Split when full
1900
1950 …
1999
Region Server 1
Key colB col
C
1900 val
val
1999
Region 1
Key
Range
1900
1999
Sequential key, like a timestamp
File
Server 1

• Regions split as the table grows.
– RegionServer Creates two new regions, each with half of the
original regions keys.
• Sequential writes will go to new region
2040
2050
2000
Region Server 1
File
Server 1 Key colB col
C
1900 val
val
1999
Region 1
Key
Range
1900
1950
Key colB col
C
1900 val
val
1999
Region 2
Key
Range
1950
2050

3040
3000
Sequential writes will go
to new region
Region Server 1
File
C
1900 val
val
1999
Region 1
Key
Range
1900
1950
Key colB col
C
1900 val
val
1999
Region 2
Key
Range
1950
3050

3045
Regions split as the table
grows.
Sequential writes will go
to new region
Region Server 1
File
C
1900 val
val
1999
Region 1
Key
Range
1900
1950
Key colB col
C
1900 val
val
1999
Region 2
Key
Range
1950
2050
Key colB col
C
1900 val
val
1999
Region 3
Key
Range
2051
3050
3041
3050

Random keys
Key
Range
a23148
3d1a5f
e0e9b4
Key
Range
Key
Range
Key
Range
Key
Range
Key
Range
MD5
Hash
rowkey
Random writes will go to different regions
If table was pre-split or big enough to have split
d = MessageDigest.getInstance("MD5");
byte[] prefix = d.digest(Bytes.toBytes(s));

Sequential vs. Random keys
Random is better for writing , but sequential is better for scanning
row keys
Sequential
Keys
Performance
Salted
Keys
Promoted
Field Keys
Random
Keys

Prefix, Promote a field key
Key
Range
Key
Range
Key
Range
Key
Range
Key
Range
Key
Range
amzn_1999
amzn_2003
amzn_2005
cisc_1998
cisc_2002
cisc_2010
goog_1990
goog_2020
goog_2030

Prefix with a Hashed field key
Key
Range
a23148_2003
1d1a5f_1999
e0e9b4_2000
Key
Range
Key
Range
Key
Range
Key
Range
Key
Range
MD5
Hash
prefix
rowkey
• prefix the rowkey with a (shortened) hash:
byte[] hash = d.digest(Bytes.toBytes(fieldkey));
Bytes.putBytes(rowkey, 0, hash, 0, length);
g0e8b4_2004
b33148_2006
3d1a5f_2007

• Which trade data needs fastest access (or most frequent)?
– Rowkey ordering
• What if you want to retrieve the stocks by symbol and date?
• Scan by row key prefix Increasing time: PREFIX_TIMESTAMP
• What if you usually want to retrieve the most recent?
Key
AMZN_1391813876369
AMZN_1391813876370
GOOG_1391813876371
SYMBOL + timestamp

Last In First Out Access: Use Reverse-Timestamp
• Row keys are sorted in increasing order
• For fast access to most-recent writes:
– design composite rowkey with reverse-timestamp that decreases
over time.
– Scan by row key prefix Decreasing: [MAXTIME–TIMESTAMP]
• Ex: Long.MAX_VALUE-date.getTime()
SYMBOL + Reverse timestamp
Key
AMZN_98618600666
AMZN_98618600777
GOOG_98618608888

• What are the needs for atomicity of transactions?
– Column design
– More Values in a single row
• Works well to get or update multiple values

Rowkey design influences shape of Tables: Tall or Flat
Tall Narrow Flat Wide

Tall Table for Stock Trades
Rowkey format:
Ex: AMZN_98618600888
SYMBOL + Reverse timestamp
rowkey
CF: CF1
CF1:price CF1:vol
… … …
AMZN_98618600666 12.34 2000
AMZN_98618600777 12.41 50
AMZN_98618600888 12.37 10000
… … …
CSCO_98618600777 23.01 1000
…

• Are Price and Volume data typically accessed together, or are
they unrelated?
– Column family structure
• Column Families
– group data based on access pattern & with similar attributes together: #
Min/Max versions, compression, in-memory, Time-To-Live
• Columns
– Column names are dynamic, not pre-defined
– every row does not need to have same columns

Wide Table for Stock Trades
rowkey
CF price CF vol
p:10 p:1000 … p:2000 v:10 v:1000 … v:2000
AMZN_986186006 12.37 13 12.34 10000 2000
…
CSCO_986186070 23.01 1000
Rowkey format:
Ex: AMZN_20131020
SYMBOL + Reverse timestamp rounded to the hour
• Separate price & volume data into column families
• Segregate time into buckets:
– Time rounded to the hour in the rowkey
– Time in column name represents seconds since the timestamp in the key
– Column names can be dynamic, every row does not need to have same columns
– One row stores a bucket of measurements for the hour

Lesson: Schemas can be very
flexible and can even
change on the fly
Column names can be dynamic, every row does not
need to have same columns

• Do all trades need to be saved forever?
– TTL Time to Live , CF can be set to expire cells
• How many Versions?
– Max Versions
– You can have many versions of data in a cell, default is 3

Wide Table for Stock Trades
rowkey
CF price CF vol CF stats
price:00 … price:23 vol:00 … vol:23 Day Hi Day Lo
AMZN_20131020 12.37 12.34 10000 2000
…
CSCO_20130817 23.01 1000
Rowkey format:
Ex: AMZN_20131020
SYMBOL + date YYYYMMDD
• Separate price & volume data into column families
• Segregate time into buckets:
– Date in the rowkey
– Hour in the column name
– Set Column Family to store Max Versions, timestamp in the version

Flat-Wide Vs. Tall-Narrow Tables
• Tall-Narrow provides better query granularity
– Finer grained Row Key
– Works well with scan
• Flat-Wide supports built-in row atomicity
– More Values in a single row
• Works well to update multiple values (row atomicity)
• Works well to get multiple associated values

Lesson: Have to know the
queries to design in
performance

Comparing Relational schema to HBase
• HBase is lower-level than relational tables
– Design is different
• Relational design
– Data centric, focus on entities and relations
– Query joins
• New views of data from different tables easily created
– Does not scale across cluster
• HBase is designed for clustering:
– Distributed data is stored and accessed together
– Query centric, focus on how the data is read
– Design for the questions
Key
Range
axxx
kxxx
Key
Range
xxx
xxx
Key
Range
xxx
zxxx

 Goal Normalization:
– eliminate redundant data
– Put repeating information in its own table
Normalized database :
– Causes joins
• data has to be retrieved from more tables.
• queries can take more time
Normalization

HBase Nested Entity
• A one-to-many relationship can be modeled as a single row
– Embedded, Nested Entity
• Order one-to-many with Line Items
– Row key: parent id
• OrderId
– column name : child id stored
• line Item id
OrderId Data:date item:id1 item:id2 item:id3
123 20131010 $10 $20 $9.45

De-Normalization
Blog and Comments in same table
De-normalization:
– store data about an entity and related entities in the same
table.
• Reads are faster across a cluster
– retrieve data about entity and related entities in one read
Key Data:ts Comment:id1 Comment:id2 Comment:id3
Blog post Id Blog text interesting boring exciting

Many to Many Relationship RDBMS
user
id (primary key)
name
alias
email
book
id (primary key)
title
description
user_book_rating
id (primary key)
userId (foreign key)
bookId (foreign key)
rating
1 ∞ ∞ 1
Online book store
• Querys
• Get name for user x
• Get tiltle for book x
• Get books and corresponding ratings for userId x
• Get all userids and corresponding ratings for book y

Many to Many Relationship HBase
User table Column family for book ratings by userid for bookids
Key data:fname … rating:bookid1 rating:bookid2
userid1 5 4
Key data:title … rating:userid1 rating:userid2
bookid1 5 4
Book table Column family for ratings for bookid by userid
• Querys
• Get books and corresponding ratings for userId x
• Get all userids and corresponding ratings for book y

Generic data, Event Data, Entity-Attribute-Value
• Generic table: Entity, Attributes, Values
– Event Id, Event attributes, Values
– object-property-value , name-value pairs , schema-less
patientXYZ-ts1, Temperature , "102"
patientXYZ-ts2, Coughing, "True"
patientXYZ-ts3, Heart Rate, "98"
• This is the advantage of HBase
– Define columns on the fly,
• put attribute name in column qualifier
– group data by column families
Key event:heartrate event:coughing event:temperature
Patientxyz-ts1 98 true 102
Event id=row key Event type name=qualifier Event measurement=value

Self Join Relationship HBase
Twitter
user x follows user y
user b followed by user a
• Querys
• Get all users who Carol follows
• Get all users following Carol
Key data:timestamp
Carol:follows:SteveJobs
Carol:followedby:BillyBob

Hierarchical Data
• tree like structure
• Use a flat-wide
– Parents, children in columns
usa
TN FL
Nashville Miami
Key P:USA P:TN p:FL c:TN C:FL C:Nashvl C:Miami
USA state state
TN country city
FL country city
Nashville state
miami state

Inheritance mapping
• Online Store Example Product table
– put sub class type abbreviation in in key prefix for searching
– Columns do not all have to be the same for different types
Key price title details model
Bok+id1 10 Hbase
Dvd+Id2 15 stones
Kin+Id3 100 fire

Agenda
• Issues with RDMS and how Hbase helps
• Quick overview of HBase Data Model & API
• Design considerations when moving from RDBMS to Hbase
• How to work around transactions that span multiple rows

Summary
• Why NoSQL and why HBase
• Design considerations when migrating from RDBMS to Hbase
• De-normalize data
• Column Families & Columns
• Versions
• RowKey design
• Transactions using checkAndPut and Increment for simple use
cases

References
• http://hbase.apache.org/
• http://hbase.apache.org/0.94/apidocs/
• http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/pack
age-summary.html
• http://hbase.apache.org/book/book.html
• http://doc.mapr.com/display/MapR/MapR+Overview
• http://doc.mapr.com/display/MapR/M7+-+Native+Storage+for+MapR+Tables
• http://doc.mapr.com/display/MapR/MapR+Sandbox+for+Hadoop
• http://doc.mapr.com/display/MapR/Migrating+Between+Apache+HBase+Ta
bles+and+MapR+Tables

Q & A
@mapr maprtech
yourname@mapr.com
Engage with us!
MapR
maprtech
mapr-technologies

Getting Started with HBase

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Getting Started with HBase (20)

More from Carol McDonald (20)

Recently uploaded (20)

Getting Started with HBase