Alluxio Architecture and Performance
0 likes825 views
Gene Pang presented on Alluxio architecture and scaling performance for large deployments. He discussed Alluxio's high-level components including the master, workers, jobs masters and workers, and proxies. He then covered techniques for improving Alluxio scaling including parallelizing metadata sync and catalog sync, handling slow external storage reads asynchronously, rearranging blocks asynchronously, and adding timeouts for disk operations to avoid unexpected hangs. The goal is to make Alluxio faster, more predictable, and support higher concurrency even with interactions with slow external storage systems.
Alluxio Architecture and Performance
- 1. DATA ORCHESTRATION SUMMIT 2020 Alluxio Architecture and Scaling Performance Gene Pang | Founding Engineer & Head Architect @ Alluxio, Inc.
- 4. DATA ORCHESTRATION SUMMIT Alluxio Overview Data Orchestration for the Cloud Java File API HDFS Interface S3 Interface REST APIPOSIX Interface HDFS Driver Swift Driver S3 Driver NFS Driver
- 6. DATA ORCHESTRATION SUMMIT Worker Job Worker Fuse Process Proxy Worker Job Worker Fuse Process Proxy Worker Job Worker Fuse Process Proxy Job Master Alluxio Master Alluxio Processes • Manages filesystem namespace • Handles authentication and authorization • Manages structured data catalog • Maintains Alluxio worker membership
- 7. DATA ORCHESTRATION SUMMIT Job Master Job Worker Fuse Process Proxy Job Worker Fuse Process Proxy Job Worker Fuse Process Proxy Alluxio Master Worker WorkerWorker Alluxio Processes • Manages file/block data cache • Writes and reads data to and from UFS • Interacts with clients for data transfer
- 8. DATA ORCHESTRATION SUMMIT Worker Job Worker Fuse Process Proxy Worker Job Worker Fuse Process Proxy Worker Job Worker Fuse Process Proxy Alluxio Master Job Master Alluxio Processes • Jobs are asynchronous and distributed tasks (load, persist, policy-based movements) • Manages execution and state of jobs • Maintains Job worker membership
- 9. DATA ORCHESTRATION SUMMIT Job Master Fuse Process Proxy Fuse Process Proxy Fuse Process Proxy Alluxio Master Worker WorkerWorker Job Worker Job Worker Job Worker Alluxio Processes • Executes distributed tasks for jobs • Uses the Alluxio client to interact with Alluxio data
- 10. DATA ORCHESTRATION SUMMIT Job Master ProxyProxy Proxy Alluxio Master Worker WorkerWorker Job Worker Job Worker Job Worker Fuse Process Fuse Process Fuse Process Alluxio Processes • Enables Alluxio Fuse (Filesystem in User Space) • Uses Alluxio client to interact with other components • Enables mounting Alluxio namespace to local filesystem • Any machine with this process can interact with Alluxio
- 11. DATA ORCHESTRATION SUMMIT Job Master Alluxio Master Worker WorkerWorker Job Worker Job Worker Job Worker Fuse Process Fuse Process Fuse Process ProxyProxy Proxy Alluxio Processes • Exposes REST and S3-compatible endpoints • Uses Alluxio client to interact with other components
- 12. DATA ORCHESTRATION SUMMIT 2020 Scaling Alluxio Performance Improving Alluxio for Large Scale Deployments
- 15. DATA ORCHESTRATION SUMMIT ▪ External Storage Systems (UFS) • HDFS • Object stores • Cloud blob stores ▪ External Catalogs (UDB) • Hive Metastore • AWS Glue ▪ Local Disks • SSDs • HDDs Sources of Slow Pauses
- 16. DATA ORCHESTRATION SUMMIT Metadata Sync Lock Contention UFS Thread-1 Thread-2 Thread-3 Thread-4 • Metadata sync is single-threaded • Requires write lock for entire sync • Blocks other users/threads Before
- 17. DATA ORCHESTRATION SUMMIT Metadata Sync Lock Contention UFS Thread-1 Thread-2 Thread-3 Thread-4 • Shorten critical section which requires write lock • Parallelize UFS access and sync with multiple threads • Enable more concurrent users, and faster syncs After
- 18. DATA ORCHESTRATION SUMMIT Slow UFS Data Reads Alluxio Worker Client UFS • High concurrency saturates network bandwidth and UFS • Client times out even though the reads eventually complete • Applications may fail unnecessarily Client Client Client Before
- 19. DATA ORCHESTRATION SUMMIT Slow UFS Data Reads Alluxio Worker UFS • Expect slow IO, adjust and handle timeouts better • Improve logging to observe slow IO • Prevent unnecessary timeouts for users and applications, during high concurrency Client Client Client Client After
- 20. DATA ORCHESTRATION SUMMIT Slow Catalog Sync Alluxio Catalog UDBClient • Alluxio catalog syncs from the UDB with a single thread • Syncing databases with many tables can take a long time Before
- 21. DATA ORCHESTRATION SUMMIT Slow Catalog Sync Alluxio Catalog UDBClient • Parallelize catalog syncing with multiple threads • Accelerate syncing large databases for users After
- 22. DATA ORCHESTRATION SUMMIT Synchronous Disk IO in Worker Storage Tiers MEM SSD HDD • CACHE_PROMOTE requires moving block to top tier • Finding free space may require cascading eviction • User reads of cached data may block on many disk IOs Before
- 23. DATA ORCHESTRATION SUMMIT Synchronous Disk IO in Worker Storage Tiers MEM SSD HDD • Make default read type as CACHE, to avoid synchronous disk IO • Rearrange blocks asynchronously to match caching policy • Avoid blocking on disk IO for user requests, for faster and more predictable Alluxio IO performance After
- 24. DATA ORCHESTRATION SUMMIT Unlimited Waiting for Disk Operations Local Disk Client Data Cache get() • Client-side cache stores data on local disk • Some disk operations can take an unlimited amount of time • Applications can get stuck when accessing cache Before