hbaseconasia2019 BDS: A data synchronization platform for HBase
- 1. THE COMMUNITY EVENT FOR APACHE HBASE™
- 2. BDS: A data synchronization platform for HBase 熊嘉男（侧⽥田） Ali-HBase 数据链路路负责⼈人
- 3. Requirement
- 4. • HBase support cross-version migration without downtime? • HBase support data backup to OSS or other storage? • HBase support replicate incremental data to MQ,ES,Solr? • Replicate incremental data from RDS to HBase? • HBase data can be archived to Spark cluster for offline analysis? • HBase High Availability …….
- 5. Challenges
- 6. • Table Structure transformation • Real-time data replication • Client double write • HBase Replication • Full data migration • DataX • CopyTable • Create Snapshot & Export Snapshot • Data consistency verification HBase clusters Migration • Cross-Version Migration compatibility issues • Impact on Business • Lack of integrated solutions Migration Step Defect
- 7. • Heterogeneous full data migration • DataX • Sqoop • Heterogeneous Real-time Data Replication • HBase Real-time Data export • Custom Replication Endpoint • Custom Replication Sink Heterogeneous Data Transmission
- 8. BDS
- 9. & & • Master & Slave • Stateless Slave • Plugin-in mode • Higher scalability and better performance High-Level Architecture
- 10. Technical Detail
- 11. HBase full data migration 3 3 . 3 3 . . . 3 . 3 . 3 . 3 1 3 35 3 2 4 3 5 3 4 3 2
- 12. HBase full data migration • Avoid the impact on business • Only access HDFS • Dynamic migration rate • Decoupled from HBase • One-click migration • Create table automaticlly • Perceive changes in region • Perceive HFiles compaction • Efficient • 100MB/s (single node) • Higher scalability
- 13. Data localization rate DataNode1 DataNode2 HFile HFile RegionServer Region Local read remote read • Data migration takes the issue of data localization rates into account • Avoid low localization rate after data migration
- 14. File split HFile1 HFile2 HFile3 HFile4 HFile1 HFile2-1 HFile2-2 HFile3 HFile4 Split Region1 Region2 Load • Migration will split HFiles according to the partitions of the original and target tables • Increase the speed of bulkload
- 16. Data pipeline 352 1162 5 3 4 • Using RingBuffer as a queue • AckQueue maintains offset • Write throughput support dynamic configuration
- 17. Impact on business 4 43 4 43 2 2 43 43 2 2 2 43 4 2 43 31 2 43 4 2 43 31 • Read and write affect data replication HBASE Replication BDS • Decoupled from HBase • Only access HDFS • Data Replication is not affected by HBase crash
- 18. Hotspot 4 43 4 43 2 2 43 43 2 2 2 43 4 2 43 31 2 43 4 2 43 31 2 3 2 1 1 1 HBASE Replication BDS • Hotspot • Round robin scheduling
- 19. Replication backlog 2 3 2 1 1 1 BDS • Add slave nodes • Slave throughput support dynamic configuration 增加Worker节点并发处理理⽇日志的数量量 增加AsyncWriter并发 Add Worker nodes
- 20. Operation and maintenance •BDS •Easy to expand •Easy to upgrade •monitor •alarm mechanism •HBase Replication •Bug fix •No alarm •Configuration modification and system upgrade requires RS to restart
- 21. BDS in Ali-Cloud
- 24. Data Backup
- 25. Archive data to Spark 1 0
- 26. RDS
- 27. About me
- 28. Thanks！