A Reliable Memory-Centric Distributed Storage System
1 like674 views
This document provides an overview of Tachyon, an open-source memory-centric distributed storage system. It discusses Tachyon's motivation of leveraging faster memory storage, architecture that manages data across tiers of storage, and status as an Apache project used in production at over 50 companies. The document outlines Tachyon's ability to improve data sharing and fault tolerance across different compute frameworks. It shares several customer use cases and previews new features around lineage and tiered storage. The overall goal of Tachyon is to better assist other components with distributed memory-centric storage.
A Reliable Memory-Centric Distributed Storage System
- 1. Haoyuan Li, Tachyon Nexus haoyuan@tachyonnexus.com September 22, 2015 @ SDC 2015 A Reliable Memory-Centric Distributed Storage System
- 2. • Team consists of Tachyon creators, top contributors, people from UC Berkeley, Google, CMU, VMware, Stanford, Facebook, etc. • $7.5 million Series A from Andreessen Horowitz • Committed to Tachyon Open Source 2
- 3. 3
- 4. Outline • Overview – Motivation – Tachyon Architecture – Using Tachyon • Open Source – Status – Production Use Cases • Roadmap 4
- 5. Outline • Overview – Motivation – Tachyon Architecture – Using Tachyon • Open Source – Status – Production Use Cases • Roadmap 5
- 6. Tachyon: Born in UC Berkeley AMPLab 6 Cluster manager Parallel computation framework Reliable, distributed memory-centric storage system
- 8. Memory is Fast • RAM throughput increasing exponentially • Disk throughput increasing slowly 8 Memory-locality key to interactive response times
- 9. Memory is Cheaper source: jcmit.com 9
- 12. 12 Missing a Solution for the Storage Layer
- 13. An Example: - • Fast, in-memory data processing framework – Keep one in-memory copy inside JVM – Track lineage of operations used to derive data – Upon failure, use lineage to recompute data map filter map join reduce Lineage Tracking 13
- 14. Issue 1 14 Data Sharing is the bottleneck in analytics pipeline: Slow writes to disk Spark Job1 Spark mem block manager block 1 block 3 Spark Job2 Spark mem block manager block 3 block 1 HDFS / Amazon S3 block 1 block 3 block 2 block 4 storage engine & execution engine same process (slow writes)
- 15. Issue 1 15 Spark Job Spark mem block manager block 1 block 3 Hadoop MR Job YARN HDFS / Amazon S3 block 1 block 3 block 2 block 4 Data Sharing is the bottleneck in analytics pipeline: Slow writes to disk storage engine & execution engine same process (slow writes)
- 16. Issue 2 16 Spark Task Spark memory block manager block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 execution engine & storage engine same process Cache loss when process crashes
- 17. Issue 2 17 crash Spark memory block manager block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 execution engine & storage engine same process Cache loss when process crashes
- 18. HDFS / Amazon S3 Issue 2 18 block 1 block 3 block 2 block 4 execution engine & storage engine same process crash Cache loss when process crashes
- 19. HDFS / Amazon S3 Issue 3 19 In-memory Data Duplication & Java Garbage Collection Spark Task1 Spark mem block manager block 1 block 3 Spark Task2 Spark mem block manager block 3 block 1 block 1 block 3 block 2 block 4 execution engine & storage engine same process (duplication & GC)
- 20. Tachyon Reliable data sharing at memory-speed within and across cluster frameworks/jobs 20
- 21. Technical Overview Ideas • A memory-centric storage architecture • Push lineage down to storage layer • Manage tiered storage Facts • One data copy in memory • Re-computation for fault-tolerance 21
- 22. Eco-System 22
- 26. Issue 1 revisited 26 Memory-speed data sharing among jobs in different frameworks execution engine & storage engine same process (fast writes) Spark Job Spark mem Hadoop MR Job YARN HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Tachyon" in-memory block 1 block 3 block 4
- 27. HDFS / Amazon S3 block 1 block 3 block 2 block 4 Tachyon" in-memory block 1 block 3 block 4 Issue 2 revisited 27 Spark Task Spark memory block manager execution engine & storage engine same process Keep in-memory data safe, even when a job crashes.
- 28. Issue 2 revisited 28 HDFS disk block 1 block 3 block 2 block 4 execution engine & storage engine same process Tachyon" in-memory block 1 block 3 block 4 crash HDFS / Amazon S3 block 1 block 3 block 2 block 4 Keep in-memory data safe, even when a job crashes.
- 29. Issue 3 revisited 29 No in-memory data duplication, much less GC Spark Task Spark mem Spark Task Spark mem HDFS / Amazon S3 block 1 block 3 block 2 block 4 execution engine & storage engine same process (no duplication & GC) HDFS disk block 1 block 3 block 2 block 4 Tachyon" in-memory block 1 block 3 block 4
- 30. Comparison with In-Memory HDFS 30
- 31. Outline • Overview – Motivation – Tachyon Architecture – Using Tachyon • Open Source – Status – Production Use Cases • Roadmap 31
- 32. Open Source Status • Started at UC Berkeley AMPLab in Summer 2012 • Apache License 2.0, Version 0.7.1 (August 2015) • Deployed at > 50 companies (July 2014) • 30+ Companies Contributing 32
- 33. Contributors Growth 33 v0.4" Feb ‘14 v0.3" Oct ‘13 v0.2 Apr ‘13 v0.1 Dec ‘12 v0.6" Mar ‘15 v0.5" Jul ‘14 v0.7" Jul ‘15 1 3 15 30 46 70 111
- 34. Codebase Growth 34 v0.4" Feb ‘14 v0.3" Oct ‘13 v0.2 Apr ‘13 v0.6" Mar ‘15 v0.5" Jul ‘14 v0.7" Jul ‘15 465 commits 696 commits 1080 commits 1610 commits 2884 commits 5021 commits
- 35. Thanks to Our Contributors! 35
- 37. Under Filesystem Choices (Big Data, Cloud, HPC, Enterprise) 37
- 38. Use Case: Baidu • Framework: SparkSQL • Under Storage: Baidu’s File System • Storage Media: MEM + HDD • 100+ nodes deployment • 1PB+ managed space • 30x Performance Improvement 38
- 39. Use Case: a SAAS Company • Framework: Impala • Under Storage: S3 • Storage Media: MEM + SSD • 15x Performance Improvement 39
- 40. Use Case: an Oil Company • Framework: Spark • Under Storage: GlusterFS • Storage Media: MEM only • Analyzing data in traditional storage 40
- 41. Use Case: a SAAS Company • Framework: Spark • Under Storage: S3 • Storage Media: SSD only • Elastic Tachyon deployment 41
- 42. Outline • Overview – Motivation – Tachyon Architecture – Using Tachyon • Open Source – Status – Production Use Cases • Roadmap 42
- 43. New Features • Lineage in Storage (alpha) • Tiered Storage (alpha) 43
- 44. New Features • Lineage in Storage (alpha) • Tiered Storage (alpha) • Data Serving • Support for New Hardware • … • Your New Feature! 44
- 46. Distributed Memory-Centric Storage: Better Assist Other Components Welcome Collaboration! 46 JIRA New Contributor Tasks
- 47. • Website: http://tachyon-project.org • Github: https://github.com/amplab/tachyon • Meetup: http://www.meetup.com/Tachyon • News Letter Subscription: http://goo.gl/mwB2sX • Email: haoyuan@tachyonnexus.com 47