Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration between Presto & Alluxio
0 likes159 views
The document discusses optimizing latency-sensitive queries at Facebook using Presto and Alluxio, focusing on architecture, re-architecture, and performance enhancement techniques. It details data caching strategies, including file handle caching and metadata caching, to reduce I/O latency and improve query execution times. Benchmark results highlight significant data read savings and cache hit rates achieved through a collaborative approach with Alluxio.
Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration between Presto & Alluxio
- 1. Optimizing Latency-Sensitive Queries at Facebook with Presto & Alluxio Ke Wang (Facebook) Bin Fan (Alluxio) December 2020
- 2. • Overview • Architecture and Problems • Re-architecture and Solutions • Performance • Alluxio Deep-dive 2
- 3. Presto is Open Source
- 4. 40K Servers ~ 1 EB data scan per day > 80% new ETL Presto @ Facebook Scale 4
- 5. • Overview • Architecture and Problems • Re-architecture and Solution • Performance • Alluxio Deep-dive 5
- 7. • Overview • Architecture and Problems • Re-architecture and Solution • Performance • Alluxio Deep-dive 7
- 8. • Metadata cache at various levels • schemas • ACLs • Partitions info • HDFS • File handle caching: avoid file open calls • File stripe/footer caching: avoid multiple redundant RPC calls to HDFS • File data caching: avoid network or HDFS latency. • Compute • Plan • Partitial Result Caching 8
- 9. • An optimization technique is to cache working dataset closer to the compute node. • Less trips to remote storage should help with latencies and IO. Data Caching 9
- 10. Driver Driver Planner/ Optimizer Scheduler Worker Worker Driver Worker HDFS Hive Metastore getFiles openFile/footer cache read/writeBlock soft affinity Data Cache Local SSD Metadata Caching Low-overhead coordinator KV store file location/stats Presto with Data Caching 10
- 11. • Random Node Scheduler • Best efforts to assign the same split to the same worker Affinity Scheduling 11
- 12. • Blocked --> Secondary Preference --> Least busy Soft Affinity 12
- 13. • Facebook internal caching libraries • Open source solutions • Build our own Various Options 13
- 14. • Naïve solution • Copying files from remote storage on local storage • Merging files in the local storage to keep file count low File Merge Caching 14
- 16. • Segment Based data caching • Pluggable eviction policies • Configuration of various aspects like sizes, resources usage, eviction policies, etc. • • A Java based OSS library • Provide detailed stats regarding cache usage. • Caching should not become a point of failure. • Asynchronous operations. • Files management at the disk level. • Flash throughput limiter to avoid endurance issues. Learnings & Alluxio Collaboration 16
- 17. • Overview • Architecture and Problems • Re-architecture w/ Presto+Alluxio • Performance • Alluxio Deep-dive 17
- 18. • Two full days worth of queries from the production cluster was shadowed to the test cluster. • Query Count: 17320 • 600 nodes cluster • 460GB per node was configured for data caching. • LRU eviction policy. • 1MB as the block size, meaning data is read, stored, and evicted in the 1 MB size. Benchmark Configuration 18
- 19. Benchmark Results Query Execution Time 19
- 20. • Data Size read for master branch run: 582 T Bytes • Data Size read for caching branch run: 251 T Bytes • Savings in Scans: 57% Benchmark Results IO Savings 20
- 21. Benchmark Results Cache hit rate 21
- 22. Production
- 23. • Overview • Architecture and Problems • Re-architecture and Solutions • Performance • Alluxio Deep-dive 23
- 24. Alluxio Overview Translate access to optimal storage APIs over a slow network Data Orchestration for the Cloud Java File API HDFS Interface S3 Interface REST APIPOSIX Interface HDFS Driver Swift Driver S3 Driver NFS Driver 24
- 25. Local cache storage Alluxio Caching File System On Cache Hit External Storage Presto Worker On Cache Miss HDFS API Calls Alluxio Cache Manager External File System Presto Server JVM Presto & Alluxio Local Cache Architecture 25
- 26. • Cache files in fix-sized segments (called pages) • configurable, 1MB by default • Store pages off-heap • avoid using JVM memory resource but with SSDs • Highly-concurrent & thread-safe • Light-weight & fine-grained locking if cacheManager.hasPage(pageId): page = cacheManager.readPage(pageId) else: readFromExternalFS(page, offset, len) cacheManager.writePage(pageId, page) Implementation & Optimization 26
- 27. • Pluggable cache replace policies: • LRU, LFU • Pluggable cache storage options: • Local file system store: each page -> one file • Rocksdb store: page -> one value associated with pageId • Async cache writes • to handle bursty cache write ops, queue writes in background • Failure Recovery • disks are expected to fail when running at Facebook scale Implementation & Optimization 27
- 28. • (WIP) Support Schema/Table/Partition level Cache Quota • (WIP) Performance optimizations for small files • (Future work) Semantics-aware caching Ongoing Development 28
- 29. • Edit etc/catalog/hive.properties • More details in the blog cache.enabled=true cache.type=ALLUXIO cache.base-directory=/tmp/alluxio-cache cache.alluxio.max-cache-size=500GB hive.node-selection-strategy=SOFT_AFFINITY Enable Alluxio Local Cache w/ Presto 29 https://prestodb.io/blog/2020/06/16/alluxio-datacaching
- 30. • Fine-grained control on working set • free / pin data in cache, set data TTL in cache etc • Metadata caching and syncing • Automatically sync data b/w Alluxio cache and persisted data • Data Transformation Services • e.g., convert csv files into parquet format in cache • Data Migration services • e.g., migrate from HDFS to S3 based on access time policy • Familiar Filesystem CLIs • e.g., alluxio fs ls /my/path Alluxio File System Enhancements 30 Alluxio Doc: https://docs.alluxio.io/os/user/stable/en/Overview.html
- 31. Twitter.com/alluxio Linkedin.com/alluxio Website www.alluxio.io Slack http://slackin.alluxio.io/ @ Social Media A recording of this talk will be available soon Q & A www.prestodb.io https://prestodb.io/blog/2020/06/16/alluxio-datacaching https://prestodb.slack.com https://alluxio.io/slack