Interactive Analytics with the Starburst Presto + Alluxio stack for the Cloud
0 likes157 views
The document provides an overview of the Starburst and Alluxio stack for interactive big data analytics in cloud environments, highlighting the benefits of separating compute and storage for efficient data access. It covers the architecture and functionalities of both Presto and Alluxio, emphasizing their ability to scale independently and improve performance without vendor lock-in. Additionally, it presents use cases and performance tuning recommendations for users implementing this technology stack in their organizations.
Interactive Analytics with the Starburst Presto + Alluxio stack for the Cloud
- 1. Interactive Big Data Analytics with the Starburst + Alluxio Stack for the Cloud 1 Matt Fuller (matt@starburstdata.com) | Co-Founder, Starburst Bin Fan (binfan@alluxio.com) | Founding Engineer, Alluxio
- 2. Agenda 2 1. Why Presto+Alluxio Stack 2. Presto Overview 3. Alluxio Overview 4. Joint Use Cases 5. Best Practise
- 3. Motivation 3 Trends: - Running Interactive SQL Queries over Big Data - Cloud and object stores become the scalable and cost-effective way to serve massive amount of data Challenges: - How to efficiently access data across Multi cloud / Hybrid cloud - SLA w.r.t. slow or variant I/O performance
- 4. Starburst Presto + Alluxio 4 A truly separated compute and storage stack enabling interactive big data analytics: • on any object store • across clusters of HDFS • across multiple different storage systems • fast interactive SQL analytics
- 5. Download Starburst | www.starburstdata.com/presto-enterprise Starburst Overview
- 6. About Me Matt Fuller Co-Founder at Starburst Previously Teradata, Hadapt, Vertica 6 Email matt@starburstdata.com LinkedIn https://www.linkedin.com/in/mfuller/
- 7. Company Overview Founded 2017 • Founding team of largest committers to open source project Presto • Former Teradata, Vertica, Hadapt, Netezza, and Ab Initio Enterprise Presto Offering • AWS, Azure, On Premises GCP & Kubernetes (coming soon) Headquartered Boston • Locations in Boston, New York, and Central Europe Customers Globally
- 8. Starburst Offering • Enterprise Presto • Latest Cost Based Query Optimizer • Fully Tested, Stable Releases • Management • Starburst Mission Control • Presto Coordinator High Availability • Autoscaling with Graceful Shutdown • Presto Security Audit Logging 8 • Ecosystem • Apache Ranger Integration • Apache Sentry Integration • Enterprise ODBC & JDBC drivers • Support • 24x7 Support SLA from the Presto Experts • Long Term Presto Version Support • Hot fixes and Security Patches • Access to Customer Success team of Data Architects • Starburst & Presto Roadmap Influence
- 9. Starburst: SQL on Anything Query anything, anywhere 9
- 10. What is Presto? Community-driven open source project High performance ANSI SQL engine • New Cost-Based Query Optimizer • Proven scalability • High concurrency Separation of compute and storage • Scale storage and compute independently • No ETL or data integration necessary to get to insights • SQL-on-anything No vendor lock-in • No Hadoop distro vendor lock-in • No storage engine vendor lock-in • No cloud vendor lock-in
- 11. What is Presto? Community-driven open source project Separation of compute and storage • Scale storage and compute independently • No ETL or data integration necessary to get to insights • SQL-on-anything No vendor lock-in • No Hadoop distro vendor lock-in • No storage engine vendor lock-in • No cloud vendor lock-in High performance ANSI SQL engine • New Cost-Based Query Optimizer • Proven scalability • High concurrency
- 12. What is Presto? Community-driven open source project No vendor lock-in • No Hadoop distro vendor lock-in • No storage engine vendor lock-in • No cloud vendor lock-in High performance ANSI SQL engine • New Cost-Based Query Optimizer • Proven scalability • High concurrency Separation of compute and storage • Scale storage and compute independently • No ETL or data integration necessary to get to insights • SQL-on-anything
- 13. What is Presto? Community-driven open source project High performance ANSI SQL engine • New Cost-Based Query Optimizer • Proven scalability • High concurrency Separation of compute and storage • Scale storage and compute independently • No ETL or data integration necessary to get to insights • SQL-on-anything No vendor lock-in • No Hadoop distro vendor lock-in • No storage engine vendor lock-in • No cloud vendor lock-in
- 14. Nobody Knows Presto Like We Do Presto commits by company, 2017-2018 Source: Github
- 15. Many Well Known Presto Users See more at https://github.com/prestodb/presto/wiki/Presto-Users
- 16. Some key Presto contributions from our team Presto-Admin For easy installation & management of Presto Security Integrations Such as Kerberos, LDAP, and in-transit encryption ANSI SQL syntax Enhancements to fully support TPC-H and TPC-DS ODBC and JDBC drivers To enable BI tools such as Power BI, Tableau, Qlik, etc. Presto Connectors SQL Server, Cassandra, and Kafka Spill to disk Capabilities for large intermediate data sets Query Performance Cost-Based Query Optimizer Providing performance boost Improved performance such as Window Functions
- 17. “Syntactic Optimizer” (without Cost Based Optimizer) CUSTOMER ORDERS LINEITEMCROSS JOIN CROSS JOIN FILTER ... CUSTOMER ORDERS LINEITEM JOIN ON CUSTKEY JOIN ON ORDERKEY
- 18. Cost Based Optimizer ORDERS CUSTOMERS JOIN ON CUSTKEY LINEITEM JOIN ON ORDERKEY 61M ROWS 15M ROWS 1.3M ROWS FILTER LINEITEM 15M ROWS 61M ROWS 3K ROWS FILTER LINEITEM ORDERS JOIN ON ORDERKEY CUSTOMER JOIN ON CUSTKEY 1.3M ROWS 15M ROWS 3K ROWS 3K ROWS 3K ROWS 61M ROWS LINEITEM
- 19. Starburst Presto Architecture Processor Processor Processor COORDINATOR WORKER WORKER DATA SOURCES Parser Optimizer Scheduler Azure SQL Database ADLS Gen 1 & 2 Blob Storage S3
- 20. Query Execution Model STAGE 0STAGE 1 TASKS OPERATOR
- 21. Alluxio Overview Download Alluxio | www.alluxio.org/download Questions? | www.alluxio.org/slack
- 22. About Me • Bin Fan • PhD CS@CMU • Founding Engineer@Alluxio 22 Email: binfan@alluxio.com Github: @apc999 Twitter: @binfan
- 23. Company Overview • Founded Feb. 2015 – Haoyuan Li • PhD research at UC Berkeley AMPLab • Initially Tachyon Nexus • Venture backed: Andreessen Horowitz etc. • Open Source • Tachyon Open Sourced in Dec. 2012 • Open source v2.0-preview Mar. 2019 • 900+ Github contributors, 4000 Github stars • Office in San Mateo, CA • Team: Google, Palantir, Vmware, AMD, Cisco…
- 24. Data Ecosystem with Alluxio • Data Locality: move data to where it is needed • Data Abstraction: API translation to different file systems and object stores • Data Accessibility: Unified namespace across different storage systems Alluxio: a Virtual Distributed File System Java File API HDFS Interface S3 Interface REST API HDFS Connector S3 Connector Swift Connector NFS Connector POSIX Interface 24
- 26. Alluxio Architecture Alluxio Master Zookeeper Standby Master Alluxio Worker Alluxio Worker Under Store RAM / SSD / HDD RAM / SSD / HDD Control Path Data Path 26
- 27. Read Data not Cached in Alluxio + Caching 27 RAM / SSD / HDD Application Alluxio Client Alluxio WorkerUnder Store 12 3 4 4
- 28. Read Cached Data in Alluxio Alluxio Worker RAM / SSD / HDD Application Alluxio Client 28 1 2 3
- 29. Write data only to Alluxio Alluxio Worker RAM / SSD / HDD Application Alluxio Client 29 1 2 3
- 30. Write to Alluxio and Under Store Synchronously RAM / SSD / HDD Application Alluxio Client Alluxio Worker Under Store 30 12 2 3
- 31. Alluxio, Presto, the Cloud
- 32. A Common File System Abstraction 32 • Common interface across apps • HDFS-compatible interface: change hdfs://foo/bar to alluxio://foo/bar • Other interfaces: Native Alluxio Java FS, POSIX and S3. • Cloud storage becomes “hidden” to apps • Less vendor lock-in! Compute Zone Standalone or managed with Mesos or Yarn Storage in Different Availability Zone Either on-prem or cloud TensorflowPrestoMR HDFS API POSIX API
- 33. Data Path: Improved I/O Performance 33 • A New Tier Above Cloud Storage for Compute • Distributed buffer cache • Restore locality to compute • Read: • Cache-hit read: served by Alluxio workers (local worker preferred) • Cache-miss read: served by cloud storage, then cache to Alluxio worker • Write: • Burst buffer, then async propagate to S3 (Alluxio 2.0) • Challenges: • Locality: expose location information to applications; serve local apps through ramdisk (rather than network)
- 34. Metadata Path: Familiar Semantics 34 • Listing / renaming on object store can be expensive • Common operations for batch or SQL analytics • Overwriting Put is eventually consistent • Alluxio loads and manages metadata in master • Apps can continue assuming HDFS-like semantics and performance implication • Challenges • Data modification bypassing Alluxio: when and how to re-sync • Slow lists in object store: batch operations • Too many objects: off-heap metadata (Alluxio 2.0)
- 35. Performance Tuning Tips: Presto + Alluxio 35 • Data Locality • Enable Locality Aware Scheduling • Hostname matching • Higher Parallelism • Tune worker threads • Tune number of splits in a batch • Tune Alluxio client timeout • Increase Netty timeout for Alluxi 1.8 https://www.alluxio.com/blog/top-5-performance-tuning-tips-for-running-presto-on-alluxio-1
- 36. Case Study: - Leading Online Retailer (NASDAQ: JD) - Building Ad-hoc SQL Query Engine - Pain Point: - Presto workers may read remotely from HDFS datanodes - Large query variance https://www.slideshare.net/Alluxio/alluxio-in-jd 36
- 37. Solution: Colocate Alluxio with Presto 37
- 38. Query Time 38
- 39. 39 Query Time
- 40. Case Study: 40 - Leading Online Gaming Service Company (NASDAQ: NTES) - Partner with Blizzard to operate service of “WoW”, “Hearthstone” - Coming “Diablo Immortal” - Building Ad-hoc SQL Query Engine - Large data volume: ~30 TB raw data daily - A separate satellite compute cluster - Pain Point: - Requirement in response time: < 15s - Large startup latency on submitting SQL jobs as YARN app https://www.alluxio.com/blog/presto-on-alluxio-how-netease-games-leveraged-alluxio-to-boost-ad-hoc-sql-on- hdfs
- 41. Solution: Presto + Alluxio 41
- 42. Result: Smoother Response During Peak Time 42 Response time (ms) Presto w/ Alluxio Presto w/o Alluxio
- 43. - Presto + Alluxio as the Stack - Truly separated compute and storage - Improve data and metadata performance on cloud storage - Alluxio Architecture and Data Flow - Master, Worker, Under Storage - Cache-{hit, miss} reads, Sync/Async writes - Use Cases on Presto + Alluxio Conclusion 43
- 45. Metadata Path: Efficient Renames 45 • Rename files on S3 can be expensive • Common operations for MR in commit phase • Write results to tmp paths • Rename tmp files to final paths (another copy, slow) • Rename with Alluxio async writes • t0: writes to tmp paths in Alluxio: near-compute, fast writes • t1: rename tmp paths to final path in Alluxio: cheap renames • t2: persist files in final paths in Alluxio to S3: 2PC to avoid partial data • Speculative execution allowed
- 46. Data Transformation 46 • Pressure in all industries to be “data driven” • Majority of companies still figuring out the transformation • Increased collection of numerous, low-value data • Challenge of overcoming data silos to convert data into business value • Limited success of Data Warehouse, Mart, and Lakes – cost of copying/moving data is substantial • Single Data Plane for Business value
- 47. Migration to Cloud 47 • Decoupling of compute and storage • Enterprise move from turnkey solution to self managed data platforms on IaaS • Lacking agility at Data Storage level • Requires Storage Abstraction
- 48. Data Path: Async Persist to S3 (Alluxio 2.0) 48 RAM / SSD / HDD Application Alluxio Client Alluxio Master Alluxio Worker Under Store • Async Writes • Step1: App writes to Alluxio • Step2: Alluxio writes to UFS • Benefits • Apps writes in Alluxio speed • Data gets persisted • Challenges • File rename/delete before persist: 2PC • Fault-tolerance: journal async requests
- 49. Alluxio 49 • Our implementation of the data access layer – a virtual distributed file system • Open source project with over 900 contributors from 100s of organizations worldwide • Deployed in many top internet and financial companies
- 50. The Data Access Layer 50 • Abstraction layer between applications and storage systems • Present a stable storage interface to applications, including semantics, security, and performance • Eliminate weakness of data silos instead of data silos themselves • Enable transparent migration of underlying storage systems • Enable application API to storage API translation in a single layer