Harmonizing Multi-tenant HBase Clusters for Managing Workload Diversity
Download as PPTX, PDF3 likes4,817 views
The document discusses the management of multi-tenant HBase clusters at Yahoo, including application onboarding, automation for compaction, monitoring, and the challenges faced in workload diversity. It details the infrastructure, use cases, customer onboarding processes, namespace management, and region server group configurations. Additionally, it touches on compaction strategies, metrics analysis, and monitoring systems implemented for performance optimization.
Harmonizing Multi-tenant HBase Clusters for Managing Workload Diversity
- 1. Harmonizing Multi-tenant HBase Clusters for Managing Workload Diversity PRESENTED BY Dheeraj Kapur, Rajiv Chittajallu, Anish Mathew⎪ May 5, 2014
- 2. Agenda Topic Speaker(s) Overview of Hadoop stack and Grid Infrastructure at Yahoo Rajiv Chittajallu Application onboarding on Multi-Tenant HBase Dheeraj Kapur Automation for Compaction/Splits and Monitoring Anish Mathew Q&A All Presenters
- 4. Hadoop Usage at Yahoo HBaseCon 2014 Browsers Mobile Devices Web Crawl Knowledge Graph 3rd Party Yahoo Grid Business Intelligence Tools (e.g. Tableau, MicroStrategy) Data Collection Asynchronous Data Processing Synchronous Serving User Events WCC Entity Feeds Content Feeds Source of truth for data* Serving Systems Home Run Search Mail Mobile Flickr Media Stream Ads Native Ads Display Ads Content systems Y! NoSql …
- 5. Grid Infrastructure at Yahoo HBaseCon 2014 A multi-tenant, secure, distributed compute and storage environment, based on Hadoop stack for large scale data processing
- 7. Deployment Model HBaseCon 2014 DataNode NodeManager NameNode RM DataNodes RegionServers NameNode HBase Master Nimbus Supervisor Administration, Management and Monitoring ZooKeeper Pools HTTP/HDFS/GDM Load Proxies Applications and Data Data Feeds Data Stores Oozie Server HS2/ HCat
- 8. Network Architecture – 1G to node HBaseCon 2014
- 9. Network Architecture – 10G Node tor VC0 spine0 leaf0 spine1 spine7 . . leaf1 leaf31 . . VC1 spine0 leaf0 spine1 spine7 . . leaf1 leaf31 . . tor 40G (or 4 x 10G) Host Host Host . . . . . . 10G
- 10. Hbase @ Yahoo HBaseCon 2014 • 7 clusters, 1500 region servers, 6 PB of data • Diverse use cases, 500+ Tables, 100k regions • Rolling Major compaction & Split and Group Rebalancing • RegionServer groups, namespaces and multi region config System
- 11. Challenges HBaseCon 2014 • Customer onboarding and provisioning • Access management and Table provisioning • Deployments • Customizing group configs • Rolling Major Compaction and Splits • Group Balancing
- 12. Use Cases
- 13. Use Cases HBaseCon 2014 Search ▪ Web Cache ▪ Query Analysis ▪ Local Listings ▪ Analytics Y! Mail ▪ Anti-spam ▪ Log Analytics ▪ Metadata Mgmt. Cloud Platforms ▪ Performance ▪ Monitoring ▪ OpenStack Consumer Platforms ▪ CMS ▪ Social Data Online Ads ▪ Traffic Protection ▪ Ads Data Mgmt. P13N ▪ Content P13N ▪ Ad targeting Mobile ▪ Notifications ▪ Flickr Sales ▪ eCommerce Yahoo’s Global Business
- 14. Web Crawl Cache HBaseCon 2014 Developers/ Scientists Poller Fetcher Ingestor Extruder Processing Random Read poll fetch launch write Compute Clusters NM DN NM DN NM DN.....HDFS NN YARN RM Clusters RS DN RS DN RS DN..... HDFSNN HBaseHM r/w insert scan
- 15. Customer Onboarding & Multi Tenancy
- 16. Customer Onboarding & Provisioning HBaseCon 2014 • Two identical environments (Prod and Non-Prod) • Applications are on boarded to Non-Prod for performance/Integration testing • Once ready, provisioned on prod • Performance results help in production onboarding
- 17. Namespaces HBaseCon 2014 • Allow tenants to create/drop/modify their own tables • Only super admin used to do it before • Quota Management • Security administration • Commands : alter_namespace, create_namespace, describe_namespace, drop_namespace, list_namespace, list_namespace_tables
- 18. RegionServer Groups HBaseCon 2014 • Missing QoS in Hbase 0.94 • Isolation is required in Multi-tenant env • Multi configs are required for different apps • Commands : group_add, group_balance, group_get, group_list, group_list_tables, group_list_transitions, group_move_servers, group_move_tables, group_of_server, group_of_table, group_remove
- 19. Multi Region Configs HBaseCon 2014 SVN Jenkins Build Farm Master Repository Slave Repository Colo B Slave Repository Colo A HBase Cluster A HBase Cluster B Fetch Group List Generate Multi Configs Merge Default Config & Push multi config Sync Configs Download Host Maps and Multi Region Config
- 20. Compaction HBaseCon 2014 • Minor & Major • Minor picks up couple of smaller files and rewrite as one • Major drop deletes or expire cells and picks up all files and rewrite as one
- 21. Compaction file selection HBaseCon 2014 F I L E S I Z E Older File Age Younger minCompactSize ExcludedIncluded
- 23. Config Parameters HBaseCon 2014 • hbase.hstore.blockingStoreFile • hbase.hstore.compaction.max.size • hbase.hstore.compaction.min.size • hbase.hstore.compaction.ratio • hbase.hregion.max.filesize • hbase.hregion.memstore.flush.size • hbase.master.wait.on.regionservers.mintostart
- 24. Managed Compactions and Splits HBaseCon 2014 • Flexible Scheduling • Custom Logic per table and workload
- 25. Compaction and Splits Scheduler HBaseCon 2014 Metrics Mysql Metrics Analyze Region Specific Metrics Server Metrics Scheduling Parameters HBaseCtl: Scheduler HDFS Publish HBase Cluster A HBase Cluster B HBase Cluster CUpdate Compaction/Split Statistics Zookeeper Coordination & Intermediate Store
- 26. Group Balancing • Scheduled group balance followed by rolling major compaction • Based on Data Locality – Find data locality of each block of store files – Move region to server where the maximum blocks are located • Helps after cluster upgrades and restarts • After config changes for a region group HBaseCon 2014
- 27. Monitoring
- 28. Monitoring HBaseCon 2014 • Simon Metrics & Yahoo Monitoring As a Service (YMS) • OpenTSDB at Yahoo, replacing MySQL as backend for YMS
- 30. Monitoring Cont.. ( Metrics for Customers) HBaseCon 2014 Simon System Other Systems for Analysis & Reporting Jenkins Job : Merges and Formats Metrics HBase HBase Master HDFS Master Grid Snodes Customer Dashboards Upload data to HDFS Memory Dump from Master Region Server Metrics Push compiled metrics to snodes Fetch metrics
- 31. Monitoring cont.. ( OpenTSDB ) HBaseCon 2014 • Evaluating • Work required to make is production ready at Yahoo