HBase from the Trenches - Phoenix Data Conference 2015
- 1. HBase from the Trenches Avinash Ramineni Email: avinash@clairvoyantsoft.com LinkedIn: https://www.linkedin.com/in/avinashramineni
- 2. Agenda • Intro to HBase – Overview – Data Model – Architecture • Common Problems • Best Practices • Tools and Utilities
- 3. Intro to HBase • Non-relational distributed column-oriented database – Modeled after Google’s BigTable • Billions of Rows and Millions of Columns • Sparse, consistent, distributed sorted map • Built on top of HDFS • Tight integration with MapReduce • Supports Random CRUD Operations
- 4. Intro to HBase • Fault Tolerant • Horizontally Scalable • Real-time Random read-write access to data stored in HDFS • Millions of queries / second • Support for transactions at a single row level • Bloom filters • Automatic Sharding • Implemented in Java
- 5. Data Model • Data is stored in Tables • Tables contain rows – Rows are referenced by a unique key - Rowkey • Rows are made of columns which are grouped in column families • Rows are sorted • Everything is stored as a sequence of bytes • All entries are versioned and timestamped
- 7. HBase Cluster • HBase Master • Zookeeper • Region Servers • HDFS - Data Nodes
- 9. Logical View
- 10. HBase API • API is simple • Operations – Get,Put,Delete,Scan,MapReduce • Connection • Create this instance only once per application and share it during its runtime • Htable – Zookeeper • HBase:meta
- 11. Column Families • All columns that are accesed together need to be grouped into a Column Family • No need to access or load data that is not used • At the column family we can define the settings like – compression, version retention policy, cache priority – Understand the data, access pattern and group column family • Column Family and Column Qualifiers are stored as bytes – Avoid being verbose
- 12. HBase Write Path
- 13. HBase Compactions • HDFS does not support updates – HFiles are immutable – New HFiles are created • Minor Compactions – Small HFiles are merged into larger Hfiles – Deletes are not applied • Major Compactions – Hfiles with in column family are merged into Single Hfile – Deletes are applied
- 14. Rowkey • Immutable • Get it right the first time before a lot of data is loaded • What if we got it wrong ? – New table and load the data – If TTL set..let the data expire
- 15. Secondary Indexes • Querying / Accessing records other than by Rowkey • Map Reduce jobs to populate index table – Periodic update • Build a secondary index with dual writes • Co-processors
- 16. Region Hotspotting • Client traffic not equally distributed across the region servers – Performance degradation – Region unavailability • Poor rowkey design – Monotonically increasing RowKey • Timeseries or Sequence – Salting • Read Vs Writes • GET ? – Hashing • Salt with one-way hash of rowkey
- 17. Short Circuit Reads • Region Servers are co-located with data nodes • HMaster assigns Regions keeping data locality in to consideration (mostly) • dfs.client.read.short-circuit – Region Servers read the data directly from HDFS rather than going through Datanode • Locality Loss
- 18. Pre-Splitting • Region splitting – Grows untill it needs to be split – Region at a time is served by only 1 Region Server • Pre-split a table into regions at table creation time – Uniformly distribute write load across region servers – Understand the keyspace • Risk of uneven load distribution • Auto splitting – Constant size region split policy – IncreasingToUpperBoundRegionSplitPolicy
- 19. Bulk Loading • Native API – Disable WAL • MapReduce Job to generate Hfile – Load using completebulkload / importTSV tool • Loads into relevant region – Faster than going through normal write path • No writes to WAL and Memstore • No flushing and compacting
- 20. Troubleshooting • ulimit -n – Limits on number of files and processs • HBase is database and needs to open a number of files • dfs.datanode.max.transfer.threadsrr. • Network • OS Parameters
- 21. You are Dead Exception • Region Servers going down – Zookeeper • Distributed co-ordinated service – HBase Master asks the region server to shutdown – Garbage Collection – Zookeeper session timeout
- 22. Performance Tuning • Compression – Reduces data stored on disk and transferred – Compression speed over ratio • Load Balancing - Balancer • Merging Regions • Batch Writes – Client Write Buffer – AutoFlush • MemStore-local allocation buffers – Garbage Collection Issues
- 23. Tuning • Heavy Writes – Flushes, compacting,splitting increase IO and degrade cluster performance • Keep Region sizes larger • Keep Hfile size large • Heavy Sequential Reads • Higher block size • Avoid Caching on table • Heavy Random Reads • Higher Blocklevel cache • Lower Memstore limit • Smaller block size
- 24. Apache Phoenix • SQL over Hbase – Compiles into Hbase Scans – Orchetrates parallel execution – Aggregate queries • JDBC API’s over Native HBase API. • Salting Buckets PreSplitting • Trafodion – Transactional SQL on HBase
- 25. Hannibal • Monitor and maintain HBase Clusters • How well regions are balanced over the cluster? • How well regions are split for each table • How regions evolve over time • How long compactions take • Integration with HUE
- 26. Hannibal
- 27. Hannibal
- 28. Hannibal
- 29. Operational Aspects • Metrics – Master • Cluster requests,split time,split size – RegionServer • Blockcache,memstore,compaction,store,IO • Htrace – Trace tool for parallel distributed system • Monitoring – Nagios – Hannibal – Ganglia – Graphite – OpenTSDB • Backup – Export,CopyTable,Snapshot