Burst Presto & Spark workloads to AWS EMR with no data copies
0 likes1,841 views
The Alluxio community office hour discusses the integration of Alluxio with AWS EMR and its benefits in enhancing big data workloads in hybrid cloud environments. The presentation highlights how Alluxio accelerates data access across on-premise and public cloud using various techniques like metadata synchronization, intelligent multi-tiering, and the use of transparent URIs. Performance benchmarks show significant query improvements, making it a valuable solution for data analytics in cloud environments.
Burst Presto & Spark workloads to AWS EMR with no data copies
- 1. Alluxio Open Source Community Office Hour: Burst Presto & Spark to AWS EMR
- 2. Open Source Project started at the UC Berkeley's AMP Lab, with an incredible Open Source Momentum with growing community 1,000+ contributors & growing 4000+ Git Stars Apache 2.0 Licensed Millions of downloads
- 3. Alluxio Use Cases Presto Alluxio *Burst big data workloads in hybrid cloud environments On-premise Public cloud Alluxio On-premise Presto Dramatically speed-up big data on object stores on premise Same container / machine or or Same instance / container Spark Alluxio Accelerate big data frameworks on the public cloud
- 4. Intro to EMR ▪ AWS Provided and Managed Hadoop Services ▪ Spark, HDFS, Presto, Hive ▪ Easy to configure and onboard ▪ Does the work for you ▪ Elastic and Flexible 4
- 5. EMR Service Integration: Bootstrap Actions ▪ EMR hooks into the main configuration files for Hadoop Services: ▪ hive-site.xml, core-site.xml, hadoop-env.sh, hive.properties ▪ Bootstrap Actions ▪ Alluxio can be deployed using a bootstrap action
- 6. Hybrid Cloud Architecture Burst workloads to EMR without manual copies and synchronization
- 7. A. Meta-data Locality with Active Sync Synchronize Metadata for On-premise Mutations Old File at path /file1 -> New File at path /file1 -> Alluxio Master Policies for pinning, promotion/demotion, TTL HDFS iNotify Based Metadata Synchronization Mutation
- 8. B. Data Locality with Intelligent Multi-tiering Local performance from remote data using multi-tier storage Hot Warm Cold RAM SSD HDD Read & Write Buffering Transparent to App Policies for pinning, promotion/demotion, TTL
- 9. C. No Hive Table Redefinitions in the Public Cloud Feature Highlight - Use Transparent URI • Begin with data and metadata on-premises • HDFS has data on-premises • Hive Metastore has meta-data with location hdfs://ns/table • Launch a Cluster in Public Cloud • Presto Catalog on EMR points to Hive Metastore On-premises • Configure Catalog to use Alluxio Transparent URI • Alluxio intercepts Presto calls to hdfs://ns/table • Start Querying in the Public Cloud • Accesses to HDFS on-premises are now served by Alluxio
- 10. Benchmark Report - TPC-DS ▪ Setup ▪ 10+1 r5.4xlarge instances in both clusters ▪ Latency: 175ms
- 11. Benchmark Report - TPC-DS ▪ Average Improvement
- 12. Benchmark Report - TPC-DS ▪ Maximum Improvement - All Queries ▪ q9 (7.1x) ▪ Maximum Improvement - By Class ▪ Reporting: q27 (3.1x) ▪ Interactive: q73 (3.9x) ▪ Deep Analytics: q34 (4.2x)
- 13. Additional Resources ▪ “Zero-Copy” Hybrid Cloud for Data Analytics - Strategy, Architecture and Benchmark Report https://www.alluxio.io/resources/whitepapers/zero-copy-hybrid-cloud-for-data-analytics-strategy-architecture-an d-benchmark-report/ ▪ Running Presto with Alluxio https://docs.alluxio.io/os/user/stable/en/compute/Presto.html ▪ Using Transparent URI https://docs.alluxio.io/ee/user/stable/en/operation/Transparent-Uri.html ▪ Top 5 performance tips running Presto with Alluxio https://www.alluxio.io/blog/top-5-performance-tuning-tips-for-running-presto-on-alluxio-1 ▪ Getting Started with EMR and Alluxio https://docs.alluxio.io/os/user/stable/en/cloud/AWS-EMR.html https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-gs.html
- 14. Questions? How are you using EMR? Welcome to join the Alluxio Open Source Community! www.alluxio.io | @alluxio