Multi-tenant Hadoop - the challenge of maintaining high SLAS
Download as PPTX, PDF2 likes1,194 views
The document discusses the challenges faced by EDF-DST in maintaining high service level agreements (SLAs) in a multi-tenant Hadoop environment due to increased demand and operational complexities. It outlines the company's big data strategy, current infrastructure, and the necessity for improvements in technical support, organizational structure, and resource management. An action plan is presented to enhance application performance, communication, and monitoring to better cater to diverse business needs.
Multi-tenant Hadoop - the challenge of maintaining high SLAS
- 1. | 1 MULTI-TENANT HADOOP THE CHALLENGE OF MAINTAINING HIGH SLAS Edouard ROUSSEAUX EDF-DTEO-DSIT-ITO-DATACENTER Big Data Tech Lead
- 2. | 2 SUMMARY Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 1. EDF CONTEXT PRESENTATION 2. BIG DATA STRATEGY OF EDF-DSIT (IT PRODUCTION DEPARTMENT) HISTORY ARCHITECTURE CHOICES BIG DATA SERVICE OFFER 3. CHALLENGES TO TAKE UP CURRENT STATE DIAGNOSTIC FOCUS 4. ACTION PLAN TECHNICAL POINTS ORGANIZATIONAL POINTS
- 3. | 3 EDF CONTEXT Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 WORLD LEADER IN LOW-CARBON ENERGY, EDF GROUP BRINGS TOGETHER ALL THE BUSINESS OF PRODUCTION, TRADE AND ELECTRICITY NETWORKS. EDF-DSIT PROVIDES IT-SERVICES TO SUPPORT THE GROUP IN ITS DIGITAL TRANSFORMATION
- 4. | 4 BIG DATA STRATEGY OF EDF-DSIT A SHARED DATALAKE Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 ARCHITECTURE CHOICES Shared Platform for all Group businesses: - Centralization of data - Economic efficiency - Sharing / Cross Analyzing data between Business - Simplify operations Specific platform per type of use cases in order to guaranty SLA, performance and flexibility: - Development / Pre-production - Production (mainly used as a backend of production app) - Backup / Disaster Recovery - Analytics (soon) These architectural choices have a very strong impact on the performance of our infrastructures and applications HISTOIRY OF BIG DATA AT EDF We start in 2012 with a first cluster (Hadoop v1) - Trade Direction wanted to start cross analyzing data - 4 recycled hosts … exponential growth … Until today : - 3 physical environments (4th one soon) - We are using Hortonworks (HDP, HDF) - 200 hosts - HDFS ≈ 1,4 PB (usable space) - YARN ≈ 14,6TB of RAM / 4600 vCores - HBASE ≈ 8,2TB of RAM - Biggest HBASE table ≈ 90TB (8k Regions)
- 5. | 5 BIG DATA STRATEGY OF EDF-DSIT BIG DATA SERVICE OFFER Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 AN OFFER THAT FOUND ITS PUBLIC − 5 Business Directions 50 production applications − +10k Yarn jobs per day (in production) − 250 HBase Tables / 25k Hbase Regions / − 150+ Hive DB − +500 users − 24/7 applications − 1 HA applications − All kind of application type: • Batch (ELT) • Streaming / Real-time • OLAP • OLTP • Big volumes / Small volumes • Critical / Non-critical applications « BIG DATA » SERVICE OFFER Business oriented offer : - A price catalog - Very simple units of work : TB et vCores - Global SLA on shared services available on our clusters (HDFS, HBASE, KAFKA,…) - Organization, process,… « self-service » consumption by trades The design of this service offer also has a strong impact on the performance of infrastructures and applications OUR BIG DATA INFRASTRUCTURES HAS BECOME ESSENTIAL TO OUR BUSINESSES
- 6. | 6 CHALLENGES TO TAKE UP AN OFFER « VICTIM » OF ITS OWN SUCCESS Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 SITUATION LAST SUMMER Many critical use cases for businesses Hard time maintaining the expected level of service : - Instability / Unavailability - Difficulty to Communicate 99.5 100 100 100 99.35 99.86 99.1 99 96 97.44 98.85 99.8599.8 99.9 96.3 100 99.2 99.88 100 100 99.6 100 100 100 95 96 97 98 99 100 101 Juin Juillet Août Septembre Octobre Novembre Availability of services in 2017 Hive Hbase Hdfs Yarn
- 7. | 7 CHALLENGES TO TAKE UP DIAGNOSTIC Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 TECHNICAL ISSUES Improve the way we operate our shared Big Data Infrastructures: - Insufficient Metrics - Monitoring didn’t evolve as fast as application types - Complex diagnostics - Sizing anticipation were not accurate Some applications were developed with anti-patterns Lack of rigor when putting applications in production - Scale-out and Performance tests - No code review Internal billing based on storage and CPU (insufficient) : - Does not accurately reflect cluster usage - Does not encourage a virtuous use of clusters NEED FOR FURTHER TECHNICAL & PRODUCTION SUPPORT
- 8. | 8 CHALLENGES TO TAKE UP DIAGNOSTIC Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 ORGANIZATIONAL ISSUES « Self-Service » Offer: - Partial control of what runs on infrastructure - Lack of a global vision of clusters usage SLA not accurate enough: - No concept of degraded service - Feeling not equal between business and operation teams Improvement of our production skills - Shift in skill type required Shared infrastructure: - Governance challenge (change management, ...) - Businesses Use cases impact each other - Accurate resources allocation is essential (according to needs) Capacity Planning : - Not sufficiently detailed - Not easy to quantify the potential impact of business use cases (Restricted vision / Sizing) NEED MORE COORDINATION BETWEEN TEAMS
- 9. | 9 CHALLENGES TO TAKE UP IN HINDSIGHT Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 EVOLVING CLUSTER USAGE Until Summer 2017 : Only « ELT » usage New type of business use cases : - Intensive use of HBASE - Huge volumes - Critical use cases Lack of anticipation : - Impact of those new use cases not qualified - More than 1000 Regions par Region Servers - HDFS saturation - HBASE collapsed 99.5 100 100 100 99.35 99.86 99.1 99 96 97.44 98.85 99.8599.8 99.9 96.3 100 99.2 99.88 100 100 99.6 100 100 100 95 96 97 98 99 100 101 Juin Juillet Août Septembre Octobre Novembre Availability of services 2017 Hive Hbase Hdfs Yarn COMPELING EVENT FOR CHANGE
- 10. | 10 ACTION PLAN TECHNICAL POINTS EDITOR SUPPORT Expertise - Diagnostic aid - Improving operating aspect of infrastructure - Metrics - Tools - Sizing / Scaling - Improvement of application ops Internal skill improvement Communication - Facilitate internal communication between teams - Best practices - Shared Diagnostics - Facilitate dialogue with management - Investment decisions - Technical evolution decisions RESOURCES ISOLATION / SHARING Fine management of Yarn queues : - Queues per Business Directions - Sub-queues per applications or use cases ? - Overbooking ? Pre-emption ? Resources Isolation : - Node labels - Region Server groups - Containerization of Elasticsearch/SolR Protection mechanism of resources : - HDFS Quotas - HBASE Quotas - Kafka Quotas - Zookeeper Observers Evaluate multi-tenancy to match business requirements and enterprise strategy.
- 11. | 11 ACTION PLAN CONFIGURATION & ARCHITECTURE Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
- 12. | 12 ACTION PLAN SERVICES AVAILABILITY Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
- 13. | 13 ACTION PLAN MONITORING – HBASE TABLE Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
- 14. | 14 ACTION PLAN MONITORING – HBASE REGION SERVER Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
- 15. | 15 ACTION PLAN MONITORING – KAFKA Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
- 16. | 16 ACTION PLAN MONITORING – HBASE REPLICATION (EVOLUTION OF BAD ROWS) Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
- 17. | 17 ACTION PLAN ORGANIZATION COMMUNICATION Build and animate Enterprise Communities - Templates / GIT,… - Feedback Sharing Improve Cross team communication Share Development guides Create shared metrics - Dashboards - Health Check - Application Status
- 18. | 18 ACTION PLAN ORGANIZATION Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 MANAGEMENT Reorganization of Application Management : - Centralized team - Global vision of jobs - Centralization of application logs - Tools CAPACITY PLANNING Setup of an « industrial » Capacity Planning - Definition of the right metrics to predict - Tools - Regural meetup to review the planning SERVICE OFFER Modification of Billing to reflect actual usage - Memory and CPU Usage for HBASE - Number of Region - Markup for small files MANAGE EXPECTATIONS & ANTICIPATE REQUIREMENTS
- 19. | 19 ACTION PLAN POSITIVE EFFECTS Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 99.5 100 100 100 99.35 99.86 100 100 100 99.1 99 96 97.44 98.85 99.85 100 100 99.899.8 99.9 96.3 100 99.2 99.88 100 100 100100 100 99.6 100 100 100 100 100 100 95 96 97 98 99 100 101 Juin Juillet Août Septembre Octobre Novembre Décembre Janvier Février Availability of services Hive Hbase Hdfs Yarn
- 20. | 20 THING TO TAKE-AWAY Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 Service OFFER & Multi-tenancy must be managed globally Communication & coordination at all level is essential Get help !!
- 21. | 21Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018 THANK YOU ! QUESTIONS