Ozone: Evolution of HDFS scalability & built-in GDPR compliance
2 likes599 views
The document discusses the evolution of Apache Ozone as a scalable alternative to HDFS with built-in GDPR compliance. It outlines Ozone's architecture, deployment, and features, emphasizing its ability to handle large data stores efficiently while providing security and high availability. Additionally, it addresses GDPR requirements, including data deletion paths and encryption methods, while noting limitations and challenges that may arise during implementation.
![Notions
Volumes ~
user accounts
Buckets ~
directories (no
sub-buckets)
Keys ~ files
HDDS Notions
Containers
[Collection of
Blocks]
Pipeline](https://image.slidesharecdn.com/ozone-evolutionofhdfsscalabilitybuilt-ingdprcompliance-190916191914/85/Ozone-Evolution-of-HDFS-scalability-built-in-GDPR-compliance-6-320.jpg)
![Architecture
Ozone’s Microservices - Divide, Conquer, Scale
• Ozone Manager - namespace [~Namenodes]
• Storage Container Managers - blockspace [~BlockServer]
• Recon Server - Control Plane
• S3 Gateway
• Datanodes](https://image.slidesharecdn.com/ozone-evolutionofhdfsscalabilitybuilt-ingdprcompliance-190916191914/85/Ozone-Evolution-of-HDFS-scalability-built-in-GDPR-compliance-7-320.jpg)
![Ozone for Enterprise
High Availability
• Built-in HA
• Single HA Configuration mode
• Regular HA Configuration mode [3 instances of OM/SCM]](https://image.slidesharecdn.com/ozone-evolutionofhdfsscalabilitybuilt-ingdprcompliance-190916191914/85/Ozone-Evolution-of-HDFS-scalability-built-in-GDPR-compliance-16-320.jpg)
Ozone: Evolution of HDFS scalability & built-in GDPR compliance
- 1. Apache Ozone Evolution of HDFS Scalability & built-in GDPR compliance Hadoop, Ozone & Apache are trademarks of the Apache Software Foundation. Dinesh Chitlangia, Cloudera Ajay Kumar, Google
- 2. Agenda • Why, When, What • Notions, Architecture, Deployment • Ozone for Enterprise Ozone • Ozone – Delete Path • Ozone & GDPR GDPR Q & A
- 3. HDFS scalability limits 400M+ Future Make your HDFS healthy day Why
- 4. Object Store for Big Data •Scale both Objects & IOPS Set of Micro-services - Divide, Conquer, Scale Seamless transition for Yarn, MapReduce, Hive, Spark apps. Supports K8s, CSI and ability to run on K8s natively. Ozone
- 5. Scale beyond HDFS Large Data Store / Dedicated Storage Clusters Cloud like presence on-prem First class citizen on K8 When
- 6. Notions Volumes ~ user accounts Buckets ~ directories (no sub-buckets) Keys ~ files HDDS Notions Containers [Collection of Blocks] Pipeline
- 7. Architecture Ozone’s Microservices - Divide, Conquer, Scale • Ozone Manager - namespace [~Namenodes] • Storage Container Managers - blockspace [~BlockServer] • Recon Server - Control Plane • S3 Gateway • Datanodes
- 10. Ozone - Write Path Similar to DFS Write, Blocks are written directly to Datanodes
- 11. Ozone - Read Path Similar to DFS Read, Blocks are read directly from Datanodes
- 12. Using Ozone: Is it as painful as HDFS? We hear you and we have to setup Ozone every time we test. • Docker • docker-compose up -d • runs it on local machine • K8s • helm install ozone • Traditional tarball • Untar • Run genconfig • Update the configurations • If you are familiar with HDFS commands • dfs -ls hdfs://user • with ozone, it will become • dfs -ls o3fs://user • If you are familiar with S3 commands like • aws s3 ls -endpoint=us-west1. /bucketName • with Ozone s3 it becomes • aws s3 ls -endpoint=s3g.local. /bucketName Setup Usage
- 14. Ozone for Enterprise • 10 Billion Keys will be supported in first official release • Scale OM/SCM independently, without any disruption • Evenly distribute metadata across the cluster including Datanodes • RAFT Consensus Protocol via Apache RATIS • Tested with industry recognized off-the-shelf components • Blockade Tests - Tests to inject errors/failures in the clusters • Tested Apache Spark, YARN, Hive workloads • K8s based clusters, long running clusters, ephemeral clusters • Freon - custom load generator
- 15. Ozone for Enterprise Simplified Security • Similar to HDFS, relies on Kerberos / Delegation Token / Block Token • SCM comes with its own Certificate Authority and users DO NOT need to know about it. • Kerberos is only needed for OM/SCM, not for datanodes • Security is on by default, not an afterthought • Transparent Data Encryption • Selectively audit READ or WRITE events, switch configs without the need to restart.
- 16. Ozone for Enterprise High Availability • Built-in HA • Single HA Configuration mode • Regular HA Configuration mode [3 instances of OM/SCM]
- 17. ENFORCEMENTTRACKER.COM British Airways £183.39M Marriott International £100M Swedish School for facial tracking Dutch Hospital for unsecured patient data
- 18. GENERAL DATA PROTECTION REGULATION (GDPR) • Law for handling personal data • Imposes responsibility on Data Controllers • Enforces Accountability for Compliance • Grants rights to Data Entity • European Law: Spills outside of EU in Digital Era
- 19. STORAGE SYSTEMS & GDPR Territorial Scope Personal Data Right to Erasure (Right to be Forgotten) Notification Obligatan of the Controller
- 20. Delete Path - Overview
- 21. Delete Path – Under the hood
- 24. OZONE & GDPR • GDPR Enabled Bucket • During Ozone Key creation, generate Simple Encryption Key(SEK) • Client writes data to blocks, encoded by SEK under the hood • During read, the data is decoded using same SEK. • During delete, OM moves the KeyInfo to Deleted Keys Section. • SEK is irrevocable lost, Data cannot be decoded even if the actual blocks are deleted much later • Notification of Obligation is achieved
- 25. OZONE & GDPR -Limitations • Backups & Restore • Rapid Key Create/Delete cycles – false positives • Existing Buckets need manual copy
- 26. • Network Topology • HA Support • Disk Scanner • In-place upgrades for HDFS Clusters • Erasure Coding • Consistent Reads from Standby OM/SCM • Stability & Scale testing • TPC-DS, Chaos Monkey, Scale testing with Partners Road ahead
- 28. Q & A THANK YOU