Open Source Memory Speed Virtual Distributed Storage
1 like1,639 views
Alluxio is an open source memory-speed virtual distributed storage system. It enables applications to access data closer to the CPU and improves performance by serving data from memory and burst buffer before spilling to persistent storage. The document outlines key updates and improvements in Alluxio versions 1.0 and 1.1, including new integrations with storage systems, usability enhancements, and significant performance optimizations that increase throughput for file creation, listing directories, and scalability of the Alluxio master and workers. Benchmark results show up to 23x faster file creation throughput and 7x faster directory listing throughput compared to prior versions.
Open Source Memory Speed Virtual Distributed Storage
- 1. Alluxio (formerly Tachyon) Open Source Memory Speed Virtual Distributed Storage Haoyuan Li CEO, Alluxio, Inc.
- 2. 2 Rebranded from Tachyon to Alluxio! Tachyon Alluxio
- 3. 3 Rebranded from Tachyon to Alluxio! http://www.alluxio.com/blog/
- 4. About Alluxio • Team – Alluxio Creators and Top Developers/Committers (all top 8 committers). • Investors
- 5. Performance Trend: Memory is Fast • RAM throughput increasing exponentially • Disk throughput increasing slowly • Memory-locality key to interactive response times
- 6. Price Trend: Memory is Cheaper Source: jcmit.com
- 7. The Big Data Ecosystem Today
- 8. What is Alluxio? • Alluxio: Memory Speed Virtual Distributed Storage • Enables Virtualized Data Across Multiple Types of Storage
- 9. 9 Open Source Community Growth 0 50 100 150 200 250 300 350 # Contributors (gitcommit history) v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 v0.8
- 10. 10 Open Source Community Growth 0 50 100 150 200 250 300 350 # Contributors (gitcommit history) v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 v0.8 v1.0 v1.1
- 11. Open Source Alluxio System • The fastest growing open source project in big data • Over 250 contributors from over 100 organizations
- 12. Alluxio Benefits • Flexibility – Enable new workloads across any storage systems – Unified Name Space enable application to access data in any storage system • Agility – Work with the framework of your choice – Work with the storage of your choice • Performance – High performance data access • Cost – Grow Storage and Compute independently • Any application accesses any data from any storage at memory speed.
- 13. New Features and Improvements in Alluxio 1.0 and 1.1 Gene Pang @ Alluxio, Inc. June 15, 2016 @ Alluxio Meetup (hosted by Intel)
- 14. About Me • Gene Pang - Software Engineer @ Alluxio, Inc. • One of the core maintainers of Alluxio Open Source Project • Ph.D. @ AMPLab, UC Berkeley • Worked at Google before UC Berkeley • Twitter: @unityxx 14
- 15. 15 Outline Performance Improvement Results in Alluxio 1.1 New Developments in Alluxio Alluxio Architecture Overview
- 17. 17 Architecture Overview Alluxio Master Alluxio Worker Alluxio Worker Alluxio Worker Under File System Under File System Journal Manages metadata Serves data blocks Mount multiple storage systems
- 19. 19 Releases Tachyon 0.8 – Oct 22, 2015 Alluxio 1.0 – Feb 23, 2016 Alluxio 1.1 – Jun 7, 2016
- 20. 20 New Developments New Integrations Usability Improvements Performance Improvements Access Control (Alpha)
- 21. 21 New Integrations Native OpenStack Swift Driver Alluxio to FUSE Connector Google Cloud Storage Aliyun Object Storage Service Google Compute Engine improve performance, reduce complexity manage data on Alibaba Cloud mount Alluxio to local file system manage data on Google Cloud Platform deploy Alluxio on Google Cloud Platform
- 22. 22 Access Control (Alpha) User/Group Support Command-line Permission Tools Configuration Parameter File System Permissions similar to POSIX permission model chown, chgrp, chmod alluxio.security.authorization.permission.enabled similar to POSIX permission model
- 23. 23 Usability Improvements Write Location Policies Simplified Configuration Automatic Metadata Loading configure how to write data to Alluxio load metadata automatically customize with properties
- 24. 24 Performance Improvements Improved Alluxio Master Scalability Better Support for Random I/O Workloads Improved Alluxio Worker Scalability fine-grained locking, efficient journaling improved data structures, improved locking cache blocks during random I/O (e.g., parquet files)
- 25. 25 Alluxio 1.1 Performance Improvement Results
- 26. 26 Create File Throughput Throughput Test Duration 1.0.1 (Local Journal)
- 27. 27 Create File Throughput Throughput Test Duration 1.0.1 1.1.0 1.8x improvement (Local Journal)
- 28. 28 Create File Throughput (Remote Journal) Throughput Test Duration 1.0.1
- 29. 29 Create File Throughput (Remote Journal) Throughput Test Duration 1.0.1 1.1.0 23x improvement
- 30. 30 List Directory Throughput Throughput Test Duration 1.0.1
- 31. 31 List Directory Throughput Throughput Test Duration 1.0.1 1.1.0 7x improvement
- 32. 32 Worker Scalability Write Latency # Blocks on Worker 1.0.1
- 33. 33 Worker Scalability Write Latency # Blocks on Worker 1.0.1 1.1.0
- 34. 34 Try out Alluxio 1.1.0 http://www.alluxio.org/releases