Using Spark with Tachyon by Gene Pang
8 likes3,031 views
This document summarizes a presentation about Tachyon, an open source memory-centric distributed storage system. It introduces Tachyon and how it can be used with Spark to resolve issues around slow data sharing, in-memory data loss during crashes, and data duplication. The presentation outlines new features in Tachyon 0.8.0 like tiered storage, pluggable data management policies, and a unified namespace across storage systems. It concludes by inviting users and collaborators to try, develop, and get involved with the Tachyon community.
Using Spark with Tachyon by Gene Pang
- 1. Using Spark with Tachyon: An Open Source Memory-Centric Distributed Storage System Gene Pang, Tachyon Nexus gene@tachyonnexus.com October 29, 2015 @ Spark Summit Europe
- 2. Who Am I? • Gene Pang • PhD from UC Berkeley AMPLab • Software Engineer at Tachyon Nexus
- 3. • Team consists of Tachyon creators, top contributors • Series A ($7.5 million) from Andreessen Horowitz • Committed to Tachyon Open Source Project • www.tachyonnexus.com
- 5. Outline • Introduction to Tachyon • Using Spark with Tachyon • New Tachyon Features • Getting Involved
- 6. Outline • Introduction to Tachyon • Using Spark with Tachyon • New Tachyon Features • Getting Involved
- 7. History of Tachyon • Started at UC Berkeley AMPLab – From Summer 2012 – Same lab produced Apache Spark and Apache Mesos • Open sourced on April 2013 – Apache License 2.0 – Latest Release: Version 0.8.0 (October 2015) • Deployed at > 100 companies
- 9. Contributors Growth 150+ Contributors 50+ Organizations
- 10. One of the Fastest Growing Big Data Open Source Projects
- 11. Thanks to Contributors and Users!
- 13. What is Tachyon?
- 15. Tachyon Stack
- 16. Why Use Tachyon?
- 17. Performance Trend: Memory is Fast • RAM throughput increasing exponentially • Disk throughput increasing slowly Memory-locality is important!
- 18. Price Trend: Memory is Cheaper source: jcmit.com
- 19. These Memory Trends are Realized By Many…
- 20. Is the Problem Solved? Missing a Solution for the Storage Layer
- 21. enables reliable data sharing at memory-speed within and across computation frameworks/jobs
- 22. How Does Tachyon Work? Memory-Centric Storage Architecture Lineage in Storage Layer
- 25. Outline • Introduction to Tachyon • Using Spark with Tachyon • New Tachyon Features • Getting Involved
- 26. Fast and general engine for large-scale data processing What are some potential issues?
- 27. Issue 1 Data Sharing bottleneck in analytics pipeline: Slow writes to disk Spark Job1 Spark Memory block 1 block 3 Spark Job2 Spark Memory block 3 block 1 HDFS / Amazon S3 block 1 block 3 block 2 block 4 storage engine & execution engine same process
- 28. Issue 1 Spark Job Spark Memory block 1 block 3 Hadoop MR Job YARN HDFS / Amazon S3 block 1 block 3 block 2 block 4 Data Sharing bottleneck in analytics pipeline: Slow writes to disk storage engine & execution engine same process
- 29. Issue 1 resolved with Tachyon Memory-speed data sharing among different jobs and different frameworks Spark Job Spark mem Hadoop MR Job YARN HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Tachyon in-memory block 1 block 3 block 4 storage engine & execution engine same process
- 30. Issue 2 Spark Task Spark Memory block manager block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 In-Memory data loss when computation crashes storage engine & execution engine same process
- 31. Issue 2 crash Spark Memory block manager block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 storage engine & execution engine same process In-Memory data loss when computation crashes
- 32. HDFS / Amazon S3 Issue 2 block 1 block 3 block 2 block 4 crash storage engine & execution engine same process In-Memory data loss when computation crashes
- 33. HDFS / Amazon S3 block 1 block 3 block 2 block 4 Tachyon in-memory block 1 block 3 block 4 Issue 2 resolved with Tachyon Spark Task Spark Memory block manager storage engine & execution engine same process Keep in-memory data safe, even when computation crashes
- 34. Issue 2 resolved with Tachyon HDFS disk block 1 block 3 block 2 block 4 Tachyon in-memory block 1 block 3 block 4 crash HDFS / Amazon S3 block 1 block 3 block 2 block 4 storage engine & execution engine same process Keep in-memory data safe, even when computation crashes
- 35. HDFS / Amazon S3 Issue 3 In-memory Data Duplication & Java Garbage Collection Spark Job1 Spark Memory block 1 block 3 Spark Job2 Spark Memory block 3 block 1 block 1 block 3 block 2 block 4 storage engine & execution engine same process
- 36. Issue 3 resolved with Tachyon No in-memory data duplication, much less GC Spark Job1 Spark mem Spark Job2 Spark mem HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Tachyon in-memory block 1 block 3 block 4 storage engine & execution engine same process
- 37. Tachyon Use Case: Baidu • Framework: SparkSQL • Under Storage: Baidu’s File System • Tachyon Storage Media: MEM + HDD • 100+ Tachyon nodes • 1PB+ Tachyon managed storage • 30x Performance Improvement
- 38. Tachyon Use Case: An Oil Company • Framework: Spark • Under Storage: GlusterFS • Tachyon Storage Media: MEM only • Analyzing data in traditional storage
- 39. Tachyon Use Case: A SAAS Company • Framework: Spark • Under Storage: S3 • Tachyon Storage Media: SSD only • Elastic Tachyon deployment
- 40. Outline • Introduction to Tachyon • Using Spark with Tachyon • New Tachyon Features • Getting Involved
- 41. Tachyon 0.8.0 Just Released! http://tachyon-project.org/
- 42. Use different frameworks to enable workloads on different storage 1. Growing Ecosystem
- 43. MEM SSD HDD Faster Greater Capacity 2. Tiered Storage Tachyon manages more than DRAM
- 44. MEM only MEM + HDD SSD only 2. Tiered Storage Configurable storage tiers
- 45. Evict stale data to lower tier Promote hot data to upper tier 3. Pluggable Data Management Policy
- 46. Tachyon Storage System (HDFS, S3, …) tachyon://host:port/ Data Users Reports Sales Alice Bob s3n://bucket/directory/ Data Users Reports Sales Alice Bob 4. Transparent Naming • Persisted Tachyon files are mapped to under storage • Tachyon paths are preserved in under storage
- 47. Tachyon Storage System A tachyon://host:port/ Data Users Alice Bob hdfs://host:port/ Users Alice Bob Storage System B s3n://bucket/directory/ Reports Sales Reports Sales 5. Unified Namespace • Unified namespace for multiple storage systems • Share data across storage systems • On-the-fly mounting/unmounting
- 48. Additional Features Remote Write Support Easy deployment with Mesos and Yarn Initial Security Support One Command Cluster Deployment Metrics for Clients/Workers/Master
- 49. Outline • Introduction to Tachyon • Using Spark with Tachyon • New Tachyon Features • Getting Involved
- 50. Welcome users and collaborators! Memory-Centric Distributed Storage System
- 51. Try Tachyon: http://tachyon-project.org Develop Tachyon: https://github.com/amplab/tachyon Meet Friends: http://www.meetup.com/Tachyon Tachyon Nexus: http://www.tachyonnexus.com Email: gene@tachyonnexus.com Thank you!