HBase coprocessors, Uses, Abuses, Solutions

© 2018 Bloomberg Finance L.P. All rights reserved.
Coprocessors:
Uses, Abuses & Solutions
DataWorks Summit
June 20, 2018
Esther Kundin & Amit Anand
Senior Software Developers

About the Speakers
• Esther Kundin
• Senior Software Developer
• Lead architect and engineer
• Machine Learning and Text Analysis
• Open Source contributor
• Amit Anand
• Senior Software Developer
• Hadoop Services / Infrastructure team
• Deployments/Tooling
• Open Source contributor

Outline
• HBase Architecture Review
• Introduction to Coprocessors
• Coprocessor Uses
• Development Abuses and Solutions
• Deployment Abuses and Solutions
• Takeaways
• Q&A

HBase Architecture Review

HBase Concepts
• HBase is the Hadoop database, a distributed, scalable, big data store
• Keys (RowKeys)
• Regions
• Region Servers
Region 1
Keys 1, 2
Region 2
Keys 3, 4
Region 3
Keys 5,6
Region 4
Keys 7, 8
Region Server 1 Region Server 2

HBase processes
• HMaster – table DDL operations
• Region Servers – data read/write operations
• Region Server with .META. table – maintains region with map of regions to
Region Server
• Outside Systems:
— Zookeeper – highly reliable distributed coordination

HBase Read Overview
Client
HDFS
Region Server Region Server Region Server
Region Server
serving .META.
region
Client requests data
from the Region
Server, which gets
data from HDFS
Client determines
which Region Server
to contact and caches
that data
In-Memory Miss
Zookeeper
Client contacts ZK for address
of Region Server serving
.META. region

HBase Write Overview
Region Server
WAL (on
HDFS)
MemStore
HFile
HFile
HFile
Region Server
persists write at the
end of the WAL
Regions Server
saves write in a
sorted map in
memory in the
MemStore
When MemStore reaches
a configurable size, it is
flushed to an HFile

Introduction to Coprocessors

What is a coprocessor?
• Custom code written by users, run as part of the HBase system
• Bundled Jar
• Loaded in HBase daemon JVM
• HMaster
• Region Server
Region Server – HBase code
Coprocessor Jar –
application code

Endpoint Coprocessor
• Similar to Stored Procedure – triggered by client API call
• Computation at data location
• Average
• Summation
• Use Protobuf to specify the input/output structure of your service
• Starting with HBase
• Must be invoked
• Table.CoprocessorService()
• Implementation of service in Java
• Run the coprocessor by calling that service to do the computation on the collocated data
Coprocessor Jar –
application code
Client

Observer Coprocessor
• Similar to trigger – triggered action happening in the daemon
• Hooks available for all HBase event types
• Canonical example - Can implement a security check on preGetOp/prePut
• Referential integrity – constraint on synthetic foreign keys
• Secondary indices – that’s how Phoenix does it
Coprocessor Jar –
application code
Client HDFS
Data Flow

Filters
• Similar to Observer Coprocessors for get/scan
• Can filter on any part of a row
• Many built-in filters available
• Can deploy custom filters
• Support for push-down predicates
• Different API
• Came earlier and much more limited

Filter Example
Snippet courtesy of https://github.com/larsgeorge/hbase-book/blob/master/ch04/src/main/java/filters/CustomFilterExample.java

Observer Coprocessor – Possible Triggers
• Region Server Observer
• preStopRegionServer
• preExecuteProcedures
• postExecuteProcedures
• preClearCompactionQueues
• postClearCompactionQueues
• Region Observer
• preGetOp
• postGetOp
• prePut
• postPut
WAL Observer
• preWALRoll
• preWALWrite
• postWALRoll
• postWALWrite

Observer Coprocessor – Possible Triggers
• Master Observer
• Runs in HBase master
• Create, Delete, Modify table
• Clone, Restore, Delete Snapshots
• Region splits and many more

Coprocessor Uses

Why use a coprocessor?
• Control behavior of Region Server
• Control behavior of data operations
• Server side processing
• Filters
• Aggregates
• Reduces pressure on the client side and network
• Less amount of data being sent from server
• NOT for complex data analysis
• Apache Phoenix (“We put the SQL back in NoSQL”)

Coprocessor Example
Post-Get example
Region Server
postGet
Key Col1 Col2 Col3 Col4 Col5
Key1Abc 1 4 5
Key1Def 2 2 2
Key1Xyz 10 11 12
Key1 Abc-
col1
Def-
col2
Abc-
col3
Abc-
col4
Xyz-
col5
Key1 1 2 4 5 12
Table Representation:
Coprocessor Result:

Apache Phoenix Coprocessors
• Apache Phoenix – OLTP and operational analytics for Apache Hadoop
• Phoenix jar runs on top of the Region Server
• Phoenix coprocessors map data model to raw bits and vice versa
• Coprocessor can ensure foreign key constraints
• Write to additional tables for global mutable secondary indices
• Server-side push down of many calculations – like filter, hash join

Development Abuses and Solutions
Application Developer Perspective

Challenge – Exceptions
• Exceptions (other than IOExceptions) in the coprocessor bring down Region Server
• 100% loss of service
• Manual intervention needed bring cluster back to health
• In other cases, the coprocessor silently unloads – depends on global settings

Solution – Catch all exception
public final void prePut(...)
throws IOException {
try {
prePutImpl(…);
}
catch(IOException ex) {
// Allow IOExceptions to propagate
// They won't cause an unload
throw ex;
}
catch(Throwable ex) {
// Wrap other exceptions as IOException
LOG.error("prePut: caught ", ex);
throw new IOException(ex);
}
}
Even better – create interface code for all coprocessors

Problem – Memory hog
• Memory is shared with Region Server memory and coprocessor memory
• Memory hogging slows down Region Server

Solutions – Defensive Java code
• Profile all coprocessor code for memory usage
• Use a generic profiler with a driver for your coprocessor
• i.e., JProfiler
• Use common Java tricks for limiting memory usage
• Use primitive types and underlying arrays where possible
• Use immutable objects
• StringBuilder vs. String concatenation

Logging and metrics tips
• Update log4j.properties file with a separate log parameter for coprocessors
• Use MDC (Mapped Diagnostic Context) context to pass parameters to all parts of the
coprocessor – across pre and post operations, for example
• http://www.slf4j.org/api/org/slf4j/MDC.html
• Create an extra column in a Result to pass back an object populated with metrics

Deployment Abuses and Solutions
Administrator Perspective

How to deploy a coprocessor?
• Known as coprocessor loading
• Static loading
• Dynamic loading

Static Loading
hbase-site.xml
hbase.coprocessor.region.classes
hbase.coprocessor.wal.classes
hbase.coprocessor.master.classes
Example:
<property>
<name>hbase.coprocessor.region.classes</name>
<value>com.bloomberg.hbase.coprocessor.endpoint.AverageEndPoint</value>
</property>
Property Type of Coprocessor
region.classes Region Observer
End Points
wal.classes Wal Observer
master.classes Master Observer

Static Loading
hbase-site.xml
RS1 RS2 RS3 RS4 RS5 RS6

Static Loading – Key points to remember
• Active on all Region Servers
— All the regions of all the tables
• Multiple co-processors can be deployed
— Provide a “,” separated list
• Update HBase’s class path
— In hbase-env.sh
— Copy jar to hbase/lib folder
• Must restart every Region Server
— hbase.dynamic.jars.dir is for filters

Static Unloading
hbase-site.xml
hbase.coprocessor.region.classes
hbase.coprocessor.wal.classes
hbase.coprocessor.master.classes
RS1 RS2 RS3 RS4 RS5 RS6

Dynamic Loading
• Modification to hbase-site.xml is not required
• Loading is per table basis
— Only available to the table they are loaded for
• Known as Table coprocessors
• Loaded via HBase shell
— Admin API via Java
• JAR is stored on shared location
— Usually HDFS
• Preferred method at Bloomberg

Dynamic Loading
HDFS
/hbase/coprocessors
coprocessor.jar MyRSObvserver.jar

Dynamic Loading - HBase shell – Admin/Developers
Image source: http://hbase.apache.org/book.htm

Dynamic Loading: Java API – Developers
Source: http://hbase.apache.org/book.htm

Dynamic Unloading: HBase shell – Admin/Developers

Dynamic Unloading: Java API - Developers
Note: In HBase 0.96 and newer, you can instead use the removeCoprocessor() method of the HTableDescriptor class.

Comparison
Description Static Loading Dynamic Loading
Changes to hbase-site.xml Yes No
Restart region servers Yes No
Jar location Local filesystem HDFS
Coprocessor availability Global Per table
Loaded via hbase shell No Yes
Loaded via java API No Yes
Read permissions HBase HBase
Management complexity High Low

Challenges
• Version compatibility can cause failures
• Rollout of non-backward-compatible coprocessor difficult
• Clean-up can be messy HBASE-14190 - Assign system tables ahead of user
region assignment
Solutions
• Compile code against version deployed on the cluster
• Keep your developers informed
• Work with developers
• Review their code
• Test version changes in development environment

Challenges
• User deployment is dangerous
• HDFS Permissions Changes
• Crashes entire cluster
• If hbase.coprocessor.abortonerror is set to true
• Bringing up cluster is challenging
• Requires manual intervention
• Missing jar will bring down the Region Server
ERROR
org.apache.hadoop.hbase.coprocessor.CoprocessorHost: The coprocessor
fooCoprocessor threw java.io.FileNotFoundException: File does not exist:
/path/to/corprocessor.jar
java.io.FileNotFoundException: File does not exist: /path/to/corprocessor.jar
Solutions
• Create a common shared directory under /hbase(/hbase/coprocessors)
• Use automation to deploy user coprocessor to shared location

Deployment Guidelines
• Enable user coprocessors
• Set hbase.coprocessor.user.enabled to true
• Phoenix coprocessors are treated as user coprocessor
• Set hbase.coprocessor.enabled to true
• Keeps system coprocessors enabled (AccessController)
• Enable coprocessor white listing (HBASE-16700)
• Set hbase.coprocessor.region.whitelist.paths
• Specify each directory individually
• Wildcards won’t include subdirectories (documentation says otherwise)
• Entire filesystem, i.e., hdfs:://Test-Laptop, won’t work (documentation says otherwise)

Deployment guidelines
• Create a bundled jar
• Includes all coprocessors
• Single entry in HBase classpath
• Use automation
• Chef/Puppet/Ansible
hbase.coprocessor.enabled hbase.coprocessor.user.enabled
hbase.coprocessor.regionserver.classes hbase.coprocessor.region.classes
hbase.coprocessor.user.region.classes hbase.coprocessor.master.classes
hbase.coprocessor.wal.classes hbase.coprocessor.abortonerror
hbase.coprocessor.region.whitelist.paths

Takeaways

Recap
• Coprocessors are necessary
• Phoenix
• HBase security
• User coprocessors are dangerous
• Write defensive code
• Be careful with deployment
• Make use of HBASE-16700
• Cleanup can be messy
• HBASE-14190 – Assign system tables ahead of user region assignment

Needed from the Community
• Story for coprocessor deployment
• Process isolation
• JMX metrics

Thank You!
Reference: http://hbase.apache.org
Chef Code: https://github.com/bloomberg/chef-bach.git
Connect with Hadoop Team: hadoop@bloomberg.net

We are hiring!
Questions?
https://www.bloomberg.com/careers

HBase coprocessors, Uses, Abuses, Solutions

More Related Content

What's hot (20)

Similar to HBase coprocessors, Uses, Abuses, Solutions (20)

More from DataWorks Summit (20)

Recently uploaded (20)

HBase coprocessors, Uses, Abuses, Solutions

Editor's Notes