How to Build a Cloud Native Stack for Analytics with Spark, Hive, and Alluxio on S3
2 likes767 views
The document discusses efficient data engineering using Apache Spark, Hive, and Alluxio on AWS S3, introduced during a data engineering meetup. It highlights the performance enhancements and architecture strategies for managing large datasets, including tiered-storage solutions for faster access and data locality. The document also provides benchmarks comparing the performance of standard AWS S3 with improved systems using ZFS and Alluxio, showcasing significant speed improvements in data processing.
How to Build a Cloud Native Stack for Analytics with Spark, Hive, and Alluxio on S3
- 1. Efficient Data Engineering with Apache Spark, Hive, and Alluxio on S3 + August 14th, 2019
- 2. Inaugural Cloud, Data & Orchestration Meetup! ● Welcome! ● First Meetup ● Looking for future presenters in Data Engineering/Ops Community ● Let us know on the Meetup group or talk to Bin, Thai, & Tim
- 3. Your Hosts: ● Thai Bui - Senior Staff Big Data Engineer, Bazaarvoice ● Bin Fan - VP, Founding Member, Alluxio ● Tim Kelly - Engineering Manager, Bazaarvoice ● Amelia Wong - Co-Founder, Alluxio
- 5. Creating the World’s Smartest Network of Consumers, Brands & Retailers
- 6. Core Product: SaaS Ratings & Reviews
- 9. 2018 - Holiday Shoppers in the Bazaarvoice Network
- 10. Confidential and Proprietary. © 2017 Bazaarvoice, Inc.10 Tools
- 11. Confidential and Proprietary. © 2017 Bazaarvoice, Inc.11 Architecture 11 Confidential and proprietary. Copyright 2015 Bazaarvoice.
- 12. Confidential and Proprietary. © 2017 Bazaarvoice, Inc.12 Cloud Compute Tech 12 Confidential and proprietary. Copyright 2015 Bazaarvoice.
- 13. Confidential and Proprietary. © 2017 Bazaarvoice, Inc.13 Bazaarvoice Data Lake Stats: AWS S3 with Parquet & ORC Registered Data on S3 500 TB Clickstream Data 250 TB Number of Clients 5700 Targetable Shoppers 214 M Active Products 125 M Total Reviews 900 M
- 14. Confidential and Proprietary. © 2017 Bazaarvoice, Inc.14
- 16. Accelerating S3 with ZFS, tiered-storage & Alluzio Thai Bui
- 17. AWS S3 : The Good An object storage service provided by Amazon ● Really cheap ● Highly available ● Fully-managed service ● Scales really well ● Integrates with virtually all tools
- 18. AWS S3 : The Bad When you have 100s of TB of data and millions of files ● Just object listing is slow ● Download speed is limited by network bandwidth ● No concept of cache ● No concept of data locality
- 19. AWS S3 : The Need For Speed ● Add tiered-storage to S3 ○ Hot, warm, cold storage (fastest, fast, and not so fast) ○ Metadata cache ○ Data cache ● Keep data local ○ In the same machine, not via the network cable ● Compatible with existing services ○ Hadoop, Spark, Hive, Presto, etc. ● Adaptive & highly configurable ○ Symlink for S3
- 20. ZFS Hive Spark Alluxio S3 Hot & Warm Cold Overview Hive ● Alluxio ○ Compatibility-layer ○ Tiered storage layer ● ZFS ○ OS-level file system ○ Volume manager ○ Acceleration layer ● Both are open-source metastore metastore metastore
- 21. Alluxio : The tiered-storage layer ● Support for native filesystem and Hadoop filesystem ● Distributed but can be installed in every node ○ Provides data locality ● Mount S3, HDFS, etc. to Alluxio ○ Think symlink. No data movement. ● Use RAM, NVMe, SSD, HDD to define hot, warm, cold data tier ● LRU, LFU policies for caching data at different layers ● Not enough space -> evict or move least used files to the next tier
- 22. ZFS : The acceleration layer ● Both a filesytem & a volume manager ○ Works with RAM to accelerate read/write ○ Auto promote/demote blocks from RAM to other storage ○ Used with local NVMe SSD if data is not in RAM ○ Mirror write to 2 SSDs -> 2x read speed ● Works at the kernel-space ● Extremely reliable ○ Automatic block checksum & repair
- 23. ZFS + NVMe: Micro benchmark I3.4xlarge, up to 10 Gbit network, 2 x 1.9 NVMe SSD, single-threaded ● Baseline w/ EBS ○ 135 MB/s write (dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync) ○ 157 MB/s read (dd if=/tmp/test1.img of=/dev/zero bs=8k) ● ZFS + 2 mirrored NVMe SSD ○ 820 MB/s write (dd if=/dev/zero of=/alluxio/fs/test1.img bs=1G count=1) ○ 1.7 GB/s read (dd if=/alluxio/fs/test1.img of=/dev/zero bs=1G count=1) ● 4x write, 10x read compared to EBS ● 8-14x compared to S3 (120 MB/s both read/write)
- 24. All together ZFS Hot Warm Kernel-space User-space Alluxio RAM NVMe SSD promote demote Native/Hadoop Filesystem API
- 25. Hive Metastore Last 30 days Alluxio > 30 daysS3 Hot & Warm Cold With Hive
- 26. Example query SELECT .. FROM .. WHERE year = 2019 AND month = 1 GROUP BY .. ORDER BY .. LIMIT 100;
- 27. Example query plan SELECT .. FROM .. WHERE year = 2019 AND month = 1 GROUP BY .. ORDER BY .. LIMIT 100
- 28. Without tiered-storage ● 50s for split calculations ○ Listing files on S3 ○ Sub-dividing the tasks amongst workers ● 12s for scanning data on S3 ● 70s to complete the query
- 29. With tiered-storage ● 1.7s for split calculations ○ 30x improvement ● 3s for scanning data on tiered-storage ○ 3x improvement ● 6s to complete the query ○ 10x improvement overall
- 30. Result ● 5-10X read improvement in Hive ○ Worker can short-circuit and read directly from ZFS instead of S3 ○ Move compute to the data ● Should give the same result in Apache Spark ● Good for iterating over the same data set multiple times ○ Machine learning ○ Exploratory analysis ● Give us control over S3 ○ More recent data should be faster to access
- 31. Question?