What’s new in Alluxio 2: from seamless operations to structured data management
0 likes1,581 views
Alluxio 2.0 introduces advanced data management features, including seamless operations on AWS and Google Cloud. It allows easy deployment with one-click solutions and enables scaling to over one billion files using embedded RocksDB for efficient metadata storage. New services for cataloging and data transformation provide deeper integration with systems like Presto, enhancing policy-driven management of data.
What’s new in Alluxio 2: from seamless operations to structured data management
- 1. What’s new in Alluxio 2 Bin Fan & Calvin Jia | Founding Engineers | Alluxio
- 2. Seamless Operations Alluxio 2 Directions Advanced Data Management 2 Hyper-scale Architecture
- 4. Cloud Native on AWS: AMI, CFT, EMR Presto Hive Cluster Metadata & Data cache Presto Hive Metadata & Data cache Compute-driven Continuous sync Compute-driven Continuous sync 4 § Alluxio AMI in the Marketplace § Alluxio Cloud Formation Template for cluster deployment § AWS EMR with Alluxio with bootstrap script Enable one-click to deploy Alluxio on AWS
- 5. Cloud Native on Google Cloud: Dataproc Presto Hive Metadata & Data cache Presto Hive Metadata & Data cache Compute-driven Continuous sync Compute-driven Continuous sync 5 § Google Dataproc with Alluxio (init action integration available) Google Dataproc Cluster Enable one-click to deploy Alluxio on Google Cloud
- 6. Native Deployment with Kubernetes Alluxio Worker Kubernetes Cluster Host Machine 6 Alluxio Master Alluxio Worker Host Machine Journal Volume Application ApplicationApplicationApplicationApplication
- 7. Self-Managed Quorum 7 Available in 2.0.0 Distributed Storage (ie. HDFS) Alluxio Standby Master Distributed Quorum (Zookeeper) Alluxio Master Alluxio Standby Master Alluxio Standby Master Alluxio Master RAFT No major external dependencies
- 9. § Challenge: • 1 file metadata takes 1KB of on-heap storage • 1 billion files would take 1 TB of heap space, GC becomes a big problem § Solution: • Add new tier with embedded RocksDB to store inode tree • Keep an in-memory cache of frequently used inodes 9 Scaling to 1 Billion+ Files Scale to one billion files and beyond, with performance comparable to previous on-heap implementation
- 10. Scaling to 1 Billion+ Files 10 Available in 2.0.0 Alluxio Master Local Disk RocksDB (Embedded) ● Inode Table ● Edge Table ● Block Table ● Block to Worker Table ● Worker to Block Table On Heap ● Inode Cache ● Mount Table ● Locks Inode ID Metadata (Binary) 12392 010101101101 12393 110110110100 … … Edge (ID, name) Inode ID 12392,foo 12393 … …
- 11. Efficient cluster communication with gRPC 11 Available in 2.0.0 Thrift (Metadata) Netty (IO) Alluxio Master Alluxio Worker Alluxio Worker Alluxio Client Alluxio Master Alluxio Worker Alluxio Worker Alluxio Client gRPC (Metadata + IO)
- 13. § New Alluxio Catalog Service • Provides the Abstraction of Structured Data • Attaching a Hive MetaStore like Mounting a File system • Understand and Serve Schema of Files or Objects § New Alluxio Data Transformation Service • Tranform csv à parquet • Compact many files à fewer files Deeper Integration with Presto 13 Presto Alluxio Connector Based off the Hive Connector Now available as Developer Preview
- 14. Policy Driven Data Management 14 Available in 2.0.0 Alluxio Master Alluxio Policy Engine Example Policy Move files older than 90 days from HDFS to S3 Application Apps access the same path regardless of where the actual data is stored Decouple logical file system namespace with physical storage systems
- 15. Replicated Asynchronous Writes 15 RAM / SSD / HDD Network Speed Write of Data Application Alluxio Client Alluxio Worker RAM / SSD / HDD Alluxio Worker Under Store Available in 2.0.0 Fast and reliable writes to Alluxio, with data persisted in background
- 16. Questions?