What's New in Alluxio 2.3
0 likes1,761 views
Alluxio 2.3 introduces enhancements in hybrid cloud deployments, structured data services, and system scalability and operability, offering significant performance gains and operational improvements. New features include one-command deployment on Amazon EMR, better metadata synchronization, and support for Amazon Glue UDB. Case studies highlight performance improvements in query latency at Facebook and reduced training costs in Kubernetes environments.
What's New in Alluxio 2.3
- 1. What’s New in Alluxio 2.3 7/14 Office Hour - Bin Fan, Calvin Jia Please Join https://alluxio.io/slack for Q&A
- 2. Alluxio 2.3 Focus Areas ● Hybrid Cloud ○ Significant number of new use cases ○ Large amount of performance gain and/or cost savings ● Structured Data Services (SDS) ○ Presto continues to be a popular compute engine ○ More interest from cloud and hybrid use cases ● System Scalability & Operability ○ Large scale exposed limits in multi-tier storage management ○ Critical deployments require regular journal backups
- 3. Hybrid Cloud Features ● One Command Deployment on Amazon EMR and Dataproc ○ Try the demo hybrid use case - AWS or GCP ○ Use Terraform modules to customize your own cloud deployments ● Validate Env ○ Set of tools for connecting to remote storage and validating it works ○ New GUI for mounting remote storage which runs several diagnostic checks ○ New CLI commands for directly running the checks ■ runHdfsMountTests - checks HDFS connectivity and settings ■ runUfsIOTest - checks UFS I/O capabilities ■ runHmsTests - checks Hive Metastore connectivity and settings
- 4. Hybrid Cloud Features ● Concurrent Metadata Synchronization ○ Enable stricter metadata synchronization settings without performance loss ○ Improved synchronization algorithm to enable multiple paths in the same subtree to be synced at once ○ Order of magnitude improvement, especially for larger namespaces that can utilize the full parallelism ○ Improved performance of INotify based active sync, an important feature in hybrid deployments
- 5. SDS Features ● Support for Amazon Glue UDB ○ Use Amazon Glue to store table metadata instead of HMS ○ Many users in cloud requested Glue support ○ Will continue to provide compatibility in the cloud ● Full Transformation Support for ORC format ○ Benefit from transformation capabilities in SDS with ORC ○ Seeing a number of users on ORC and Parquet ○ ORC vs Parquet is mostly based on the compute framework and available optimizations
- 6. System Scalability ● Tiered Storage V2 ○ Use multi-tiered storage in read-write heavy workloads without performance loss ○ Eviction becomes O(1) operation, ensure write or cache path is not impacted ○ Evictors are now termed Annotators because it simply pre-marks the blocks to evict ○ Trade-off made is block placement is not 100% accurate when using multiple tiers ○ Background tasks are system resource aware and will operate only during quiet periods
- 7. System Operability ● Delegated Journal Backups for HA Deployments ○ Backups no longer impact master response latency ○ Instead of using primary master to backup, secondaries can run backups ○ Very useful for deployments running regular backups, can avoid performance impact ○ Window for performance hit is reduced to several seconds instead of minutes
- 8. Case Studies
- 9. Improving Presto Latency at Facebook With Alluxio data caching, for one of the Facebook internal Presto use cases: - query latencies improved by 33% (P50), and 48% (P95). - 57% improvement in IO for remote data source scans. https://prestodb.io/blog/2020/06/16/alluxio-datacaching Presto + Alluxio + Facebook HDFS
- 10. Memory-speed I/O for Model Training in K8s Alibaba Cloud K8s team is leveraging Alluxio Posix Interface to serve deep learning training in K8s in Kubernetes environment. - 40% reduction in training time & cost https://www.alluxio.io/blog/efficient-model-training-in-the-cloud-with-kubernetes-tensorflow-and-alluxio/ Tensorflow + Alluxio + OSS + K8s
- 11. Data Proxy for Hybrid Cloud WeRide, a self-driving-car company, deployed Alluxio to leverage Alluxio as a local cache in each offices, on top of a centralized S3 data lake - Saving $5 per issue ticket per engineer https://www.alluxio.io/blog/building-a-cross-region-hybrid-cloud-storage-gateway-for-engineers-at-weride/ DS tools + Alluxio + S3 (remote)