Data Protection in Hybrid Enterprise Data Lake Environment
Download as PPTX, PDF0 likes375 views
This document discusses data protection in hybrid data lake environments using Cloudera's Data Lifecycle Manager (DLM) service. It provides an overview of DLM's capabilities for replicating data between on-premises and cloud environments, including HDFS, Hive, Ranger policies, and metadata. Key features highlighted are incremental replication of Hive metadata and data, HDFS snapshot-based replication between Hadoop clusters, and replication to cloud storage providers like AWS S3, Azure ADLS, and GCP. The document also demonstrates DLM's user interface and replication of data and security policies between on-prem and cloud clusters.
Data Protection in Hybrid Enterprise Data Lake Environment
- 1. © Cloudera, Inc. All rights reserved. DATA PROTECTION IN HYBRID DATA LAKE ENVIRONMENT Murali Ramasami, Staff Software Engineer
- 2. © Cloudera, Inc. All rights reserved. 2© Cloudera, Inc. All rights reserved. AGENDA Hybrid Data Lake Environment Data Lifecycle Manager Replicate data HDFS <-> Cloud Replicate data Hive <-> Cloud Demo
- 3. © Cloudera, Inc. All rights reserved. 3© Cloudera, Inc. All rights reserved. Digital content is doubling every 18 months Structured Data - Database - Data Warehouse - ERPs - CRMs Unstructured Data - Web blogs - Social media - Audio, Video - Software file-systems Source: Frost & Sullivan - World’s Top Global Mega Trends To 2025 and Implications to Business, Society and Cultures 1 2 3 BIG DATA TREND 90% of the data in the world has been created in the last two years alone
- 4. © Cloudera, Inc. All rights reserved. 4© Cloudera, Inc. All rights reserved. HYBRID ENTERPRISE DATA LAKE ENVIRONMENT ON- PREM Clusters CLOUD Dataplane Service HIVE SPARK YARN HDFS ATLAS, RANGER, HIVE METASTORE. KNOX AMBARI • LARGE • SHARED WORKLOADS • MULTI-TENANT CLUSTERS • SHARED SERVICES • MULTI-CLUSTER • MULTI-CLOUD Cloudbreak ATLAS, RANGER, HIVE, KNOX PUBLIC CLOUD COMPUTE PUBLIC CLOUD STORAGE HIVE LLAP AMBARI SPARK AMBARI
- 5. © Cloudera, Inc. All rights reserved. 5© Cloudera, Inc. All rights reserved. Cloudera DataPlane is a portfolio of data solutions that enable the enterprise to manage & discover data across hybrid environments. CLOUDERA DATAPLANE CLOUDERA DATAPLANE SERVICE MULTIPLE CLUSTERS AND SOURCES MULTIHYBRID DATA AT REST Cloudera Data Platform DATA IN MOTION Cloudera Data Flow
- 6. © Cloudera, Inc. All rights reserved. 6© Cloudera, Inc. All rights reserved. DATAPLANE: AT A GLANCE Core Capabilities Data Lakes, Clusters Platform Services Extensibility, Metering DP PLATFORM CDP CDF
- 7. © Cloudera, Inc. All rights reserved. 7© Cloudera, Inc. All rights reserved. DATAPLANE APPLICATIONS
- 8. © Cloudera, Inc. All rights reserved. 8© Cloudera, Inc. All rights reserved. DATA LIFECYCLE MANAGER (DLM) SERVICE A Portfolio of Service • Replication/failback to another cloud/on-prem site for Disaster Recovery • Backup & Restore of business critical data, for protection against accidental deletion • Auto Tiering of hot/warm/cold data to cloud object storage/on-prem for TCO reduction. Production Site Disaster Recovery Site Offsite Replication Failback Sunday Monday Tuesday Wednesda y Thursday Friday Saturday Sunday Full Backup Cumulative incremental backup Accidental Deletion Solid State Drive Hard Drive Archive Access to Data 0days 30days 90days Forever ProbabilityofReuse Time 100% 0% Disaster Recovery Backup & Restore Auto Tiering S3
- 9. © Cloudera, Inc. All rights reserved. 9© Cloudera, Inc. All rights reserved. DLM FEATURES • Incremental Hive replication & Hive metadata • HDFS snapshot based replication between HDP clusters • Ranger policy replication • Atlas tag/lineage replication • Cloud storage replication (AWS, Azure, GCP) • Active/standby behavior on DR site using Ranger • TDE & TLS support, Support multiple keys/KMS
- 10. © Cloudera, Inc. All rights reserved. 10© Cloudera, Inc. All rights reserved. DLM UI DLM App Dataplane Knox DLM Engine Yarn RangerHive HDFS Knox DLM Engine RangerHive HDFS Source Cluster Destination Cluster Atlas Atlas DLM ARCHITECTURE
- 11. © Cloudera, Inc. All rights reserved. 11© Cloudera, Inc. All rights reserved. DLM ENGINE INTERNALS • DLM engine is stateless server, state in external DB. • Schedules replication jobs using Quartz. • Job manager handles job concurrency, failures, retries and recovery. • Metric collection for job and data transfer • Specific plugins handles replication of the specific data from source service to target service. Knox Scheduler Job manager Data Store HDFS Hive Ranger REST API DLM Engine Plugin Manager
- 12. © Cloudera, Inc. All rights reserved. 12© Cloudera, Inc. All rights reserved. REPLICATING DATA ON-PREM <-> CLOUD
- 13. © Cloudera, Inc. All rights reserved. 13© Cloudera, Inc. All rights reserved. HDFS - CLOUD STORAGE REPLICATION • Replication to/from cloud storage directly, no cluster required on target • Supported on AWS S3, Azure ADLS, GCP GCS • Data is pushed/pulled from on-prem clusters using Hadoop Connectors for the filesystem • Supports cloud native encryptions • File ACLs, Ranger Policies and Atlas Metadata are copied in meta files for restore Knox DLM Engine Distcp (Yarn) Ranger HDFS On-prem Cluster Cloud Storage Replication Policy
- 14. © Cloudera, Inc. All rights reserved. 14© Cloudera, Inc. All rights reserved. REPLICATION—SECURITY POLICIES & GOVERNANCE METADATA • Copies resource and descendent policies for the directories/DBs replicated • Copies row filters and column masking rules • Adds deny policy on target • Atlas metadata, tags associated for the directories/DBs created will also get replicate on to the target
- 15. © Cloudera, Inc. All rights reserved. 15© Cloudera, Inc. All rights reserved. DEMO
- 16. © Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved. HIVE ON-PREM TO CLOUD REPLICATION • Minimal cluster on the cloud - shared services cluster • Push based replication to avoid opening data ports • Uses cloud SDK for secure data transfer • Supports cloud storage encryption • Uses Hadoop connector to transfer data Knox DLM Engine Distcp (Yarn) Ran ger HDFS On-prem Cluster Replication Policy HS2 Cloud Storage Knox DLM Engine Ran ger Cloud Cluster HMS
- 17. © Cloudera, Inc. All rights reserved. 17© Cloudera, Inc. All rights reserved. EVENT BASED HIVE REPLICATION ( CLOUD ) Master Cluster Cloud Metastore - RDBMS Event Table Metastore - RDBMS Cloud Storage HDFS HiveServer2 Distcp REPL DUMP REPL LOAD Serialize New Events Dump ( Data + Metadata )
- 18. © Cloudera, Inc. All rights reserved. 18© Cloudera, Inc. All rights reserved. DEMO
- 19. © Cloudera, Inc. All rights reserved. THANK YOU