High Availability for HBase Tables - Past, Present, and Future
9 likes2,690 views
This document summarizes different approaches to achieving high availability in HBase. It discusses HBase region replicas, asynchronous WAL replication, and timeline consistency introduced in HBase 1.1 to provide increased read availability. It also describes WanDisco's non-stop HBase implementation using Paxos consensus and Facebook's HydraBase which uses RAFT consensus, with HydraBase designating region replicas as active or witness. The document compares these approaches on attributes like consensus algorithm, read/write availability, strong consistency, and support for multi-datacenter deployments.
High Availability for HBase Tables - Past, Present, and Future
- 1. HBASEHIGHAVAILABILITY HBASE HIGH AVAILABILITYPAST, PRESENT AND FUTURE JUN // 11 // 2015 SUDARSHAN KADAMBI, BLOOMBERG DEVARAJ DAS, HORTONWORKS
- 2. HBASEHIGHAVAILABILITY HBASE BASICS – QUICK RECAP 2 Region Memstore RegionServer HDFS Client Region MemstoreRegion Memstore Get/Scan/Put Memstore
- 3. HBASEHIGHAVAILABILITY 3 HBASE MTTR – AN OVERVIEW
- 4. HBASEHIGHAVAILABILITY BEYOND MTTR …. 30 seconds downtime (aggressive MTTR time) is still too much CERTAIN CLASSES OF APPS ARE WILLING TO TOLERATE DECREASED CONSISTENCY GUARANTEES IN FAVOR OF AVAILABILITY » Especially for READs INTRODUCED REGION REPLICA & TIMELINE CONSISTENCY IN HBASE Community effort in the Apache jira HBASE-10070 4
- 5. HBASEHIGHAVAILABILITY HBASE-10070: “REGION REPLICA” 5 FOR EVERY REGION OF THE TABLE, THERE CAN BE MORE THAN ONE REPLICA » Each region replica is hosted by a different region server ONE REPLICA PER REGION IS THE “PRIMARY” » Only this can accepts WRITEs » All reads from this region replica return the most recent data OTHER REPLICAS, ALSO CALLED “SECONDARIES” FOLLOW THE PRIMARY » Via Asynchronous WAL replication » May not have received the recent data » Serve data from the same data files (hfiles) as primary
- 6. HBASEHIGHAVAILABILITY HBASE-10070: “ASYNC WAL REPL” Client1 X=1WAL MEM Replica_id=0 (primary) Replica_id=r1 Replica_id=r2 replication replication X=3 WAL MEM WAL MEM X=1X=1Write
- 7. HBASEHIGHAVAILABILITY HBASE-10070: “ASYNC WAL REPL” Client1 X=1 Client2 WAL MEM Replica_id=0 (primary) Replica_id=r1 Replica_id=r2 replication replication X=3 WAL MEM WAL MEM X=1 X=1 X=1 X=1 X=1 X=1Read X=1Read X=1Read
- 8. HBASEHIGHAVAILABILITY HBASE-10070: “ASYNC WAL REPL” Client1 X=1WAL MEM Replica_id=0 (primary) Replica_id=r1 Replica_id=r2 replication replication X=3 WAL MEM WAL MEM X=1 X=1 X=1 X=2Write X=2 X=2
- 9. HBASEHIGHAVAILABILITY HBASE-10070: “ASYNC WAL REPL” X=1 Client2 WAL MEM Replica_id=0 (primary) Replica_id=r1 Replica_id=r2 replication replication X=3 WAL MEM WAL MEM X=1 X=1 X=1 Read Read X=1Read X=2 X=2 X=2 X=2 X=2 X=2 Reads and Scans can be performed, returning possibly stale data
- 10. HBASEHIGHAVAILABILITY HBASE-10070: “TIMELINE CONSISTENCY” 10 PUBLIC ENUM CONSISTENCY { STRONG, TIMELINE } GET GET = NEW GET(ROW); GET.SETCONSISTENCY(CONSISTENCY.TIMELINE); ... RESULT RESULT = TABLE.GET(GET); … IF (RESULT.ISSTALE()) { ... }
- 11. HBASEHIGHAVAILABILITY THE CURRENT STATE OF HBASE-10070 • Available in HBase 1.1.0 • Provides HA reads but not writes • Simplest to implement • Consensus-based HA more work than having No consensus 11
- 12. HBASEHIGHAVAILABILITY EXTENSIONS TO HBASE-10070 • Standby-aware Assignment Manager • Standby-aware DataNode placement • Pluggable retry policies • Pluggable consistency models 12
- 13. HBASEHIGHAVAILABILITY HIGH AVAILABILITY – SOLUTION PARAMETERS • Where does the WAL go? • Are Region Servers Active-Active or Active- Standby? • Do you use quorum? • What consensus algorithm to achieve quorum? 13
- 15. HBASEHIGHAVAILABILITY WAN DISCO: ARCHITECTURE • WAL on Local file system • Multiple Region Servers host a region • Reads and writes to a key served by any replica • Paxos-based Consensus 15
- 16. HBASEHIGHAVAILABILITY WAN DISCO: WRITE PATH • All replicas equal for reads and writes • Client sends the Put to any one replica • Chosen replica co-ordinates with peers • Memstore and WAL updated after consensus is achieved 16
- 17. HBASEHIGHAVAILABILITY WAN DISCO: WRITE PATH Client WAL MEM Replica_id=R1 Replica_id=R2 Replica_id=R3 Networkpartition WAL MEM WAL MEM X=2 X=2 X=2 WRITE X=2 X=2 X=2
- 18. HBASEHIGHAVAILABILITY WAN DISCO: READ PATH • Reads go to any one replica in the set • On timeout, client does co-ordinated read to other replicas • Could potentially return stale data 18
- 19. HBASEHIGHAVAILABILITY WAN DISCO: READ PATH Client WAL MEM Replica_id=R1 Replica_id=R2 Replica_id=R3 WAL MEM WAL MEM X=2 X=2 X=2 READ X X=2
- 20. HBASEHIGHAVAILABILITY WAN DISCO: READ PATH Client WAL MEM Replica_id=R1 Replica_id=R2 Replica_id=R3 WAL MEM WAL MEM X=2 X=2 X=2 READ X X=2
- 21. HBASEHIGHAVAILABILITY WAN DISCO: REGION SERVER FAILURE • No downtime on reads or writes upon RS failure • Failed RS will not persist writes • Failed RS might be on the read path, client re-issues read on timeout 21
- 23. HBASEHIGHAVAILABILITY HYDRABASE: ARCHITECTURE • WAL on Local file system • Multiple Region Servers host a given region • Reads and writes to a key served by active leader only • RAFT-based Consensus 23
- 24. HBASEHIGHAVAILABILITY HYDRABASE: ARCHITECTURE • Replicas are in either active or witness mode • Active replicas write HFiles • Witness replicas only do WAL replication 24
- 25. HBASEHIGHAVAILABILITY HYDRABASE: ARCHITECTURE 25 cc, Image Courtesy https://code.facebook.com/posts/321111638043166/hydrabase-the-evolution-of-hbase-facebook/
- 26. HBASEHIGHAVAILABILITY HYDRABASE: FAILOVER 26 cc, Image Courtesy https://code.facebook.com/posts/321111638043166/hydrabase-the-evolution-of-hbase-facebook/
- 27. HBASEHIGHAVAILABILITY HYDRABASE: FAILOVER 27 cc, Image Courtesy https://code.facebook.com/posts/321111638043166/hydrabase-the-evolution-of-hbase-facebook/
- 28. HBASEHIGHAVAILABILITY COMPARISON MATRIX 28 INTER- CLUSTER REPLICATION HBASE-10070 WAN DISCO HYDRABASE CONSENSUS NO CONSENSUS NO CONSENSUS PAXOS RAFT READ AVAILABILITY YES YES YES YES WRITE AVAILABILITY YES NO* YES YES STRONGLY CONSISTENT HA READS NO NO NO YES
- 29. HBASEHIGHAVAILABILITY COMPARISON MATRIX 29 INTER- CLUSTER REPLICATION HBASE-10070 WAN DISCO HYDRABASE STRONGLY CONSISTENT HA WRITES NO NO YES YES MULTI- DATACENTER YES NO* YES YES PLUGGABLE CONSISTENCY MODEL NO NO NO NO LICENSE OPEN SOURCE OPEN SOURCE, 1.1 COMMERCIAL PROPRIETARY *
- 30. HBASEHIGHAVAILABILITY ACKNOWLEDGEMENTS • Ryan Rawson & Jagane Sundar, WanDisco • HBase team at Facebook 30