Alluxio - Virtual Unified File System
2 likes1,824 views
Alluxio provides a virtual unified file system that allows for unified access and accelerated performance of data across multiple storage systems and tiers. It addresses challenges of separating compute and storage in modern data architectures by providing a global namespace, server-side API translation between storage systems, and intelligent multi-tiering of data across RAM, SSDs and HDDs. Alluxio has been deployed in over 100 production environments across financial services, retail, telecom and other industries to accelerate analytics, machine learning and other workloads.
Alluxio - Virtual Unified File System
- 1. Alluxio –Virtual Unified File System Li Haoyuan – Founder and CEO at Alluxio haoyuan@alluxio.com
- 2. The Global Datasphere will grow from 33 ZB in 2018 to 175 ZB by 2025 China’s Datasphere is expected to grow 30% on average over the next 7 years & will be the largest Datasphere of all regions by 2025 Source: IDC White Paper – #US44413318
- 3. We are in the era where Data is your biggest asset
- 4. Extracting maximum value from your data The Data Ecosystem Evolution
- 5. Data Ecosystem - Beta Data Ecosystem 1.0 COMPUTE STORAGE STORAGE COMPUTE
- 6. Data Ecosystem 1.0 – The Challenges STORAGE COMPUTE Complex Low performance Expensive
- 7. 3 big trends driving the need for a new architecture Separation of Compute & Storage Hybrid –Multi cloud environments Self-service data across the enterprise
- 8. The Data Architecture for the Digital Future
- 9. Core requirements of 2.0 data ecosystem Unified Memory-first Native APIs Multi-hybrid cloud
- 10. AVirtual Unified File System
- 11. Virtual Unified File System Java File API HDFS Interface S3 Interface REST APIFUSE Interface HDFS Driver Swift Driver S3 Driver NFS Driver
- 12. Unified Namespace Bring all files into a single interface Interact with data using any API Accelerate & tier data transparently API Translation Intelligent Multi-tiering Key Innovations of theVirtual Unified File System
- 13. Unified Namespace: Global Data Accessibility FUSE Interface makes all enterprise data available locally SUPPORTS • HDFS • NFS • OpenStack • Ceph • Amazon S3 • Azure • Google Cloud IT OPS FRIENDLY • Storage mounted into Alluxio by central IT • Security in Alluxio mirrors source data • Authentication through LDAP/AD • Wireline encryption HDFS #1 Object Store NFS HDFS #2
- 14. Server-side API Translation: From legacy to modern Convert from Client-side Interface to native Storage Interface Java File API HDFS Interface S3 Interface REST APIFUSE Interface HDFS Driver Swift DriverS3 Driver NFS Driver
- 15. Intelligent Multi-tiering: Get high-value data faster Local performance from remote data using multi-tier storage Hot Warm Cold RAM SSD HDD Read & Write Buffering Transparent to App Policies for pinning, promotion/demotion,TTL
- 16. Real world Use cases
- 17. Virtual Data Lake § Accelerate batch, micro- batch & streaming jobs § Slowly transition to lower cost object stores § Run in hybrid cloud environment with compute in the cloud § Accelerate ML jobs running on object stores or file systems § Provide consistent performance to data scientists § Provide unified interface to access all data § Accelerate & tier data transparently across storage tiers § Co-locate remote data with compute for performance Machine Learning Productivity Self-service data across hybrid cloud Popular Technical Use Cases
- 18. 100+ Known Production Deployments Massive clusters deployed, many with 500+ nodes
- 19. Financial Services Case Study Machine Learning Use Case Challenge – Gain end to end view of business with large volume of data Queries were slow / not interactive, resulting in operational inefficiency Solution – ETL Data from Teradata to Alluxio Impact – Faster Time to Market – “Now we don’t have to work Sundays” SPARK TERADATA SPARK TERADATA
- 20. Retail Case Study Customer Analytics Use Case Challenge – Bottleneck in Trend Analysis of mission critical daily sales and inventory management Queries were slow / not interactive, resulting in operational inefficiency Solution – With Alluxio, data queries are 10X faster Impact – Higher operational efficiency SPARK HDFS SPARK HDFS
- 21. Telecom Case Study Customer 360 Insights Challenge – Desired a central view of consumer information in near real time for proactive support. Many HDFS, different distributions, many incompatible versions. On- prem & cloud. Integration through heavy ETL. Solution – Alluxio integrates data into central catalog for fast access to consumer interaction records. Impact – Reduced integration time Faster data speed & freshness HADOOP ML HADOOP HDFS HDFS HDFS ML ETL HDP HDFS CDH HDFS MAPR HDFS HDFS
- 22. Machine Learning / Deep Learning – Maximizes GPU investment: • Self-serve data access for data scientists • Rapid integration of new data sources • Improved memory management & performance
- 23. Incredible Open Source Momentum with growing community 920+ contributors & growing 3760+ Git Stars Apache 2.0 Licensed Hundreds of thousands of downloads Download Alluxio today @ www.alluxio.org
- 24. ThankYou Join the Alluxio Community www.alluxio.org | www.alluxio.com | Twitter: @alluxio