Storage for big-data by Joshua Robinson
2 likes211 views
The document discusses the use case of Pure Storage's FlashBlade in handling big data analytics efficiently and effectively. It outlines the infrastructure requirements, challenges with test failures, and the automation of performance metrics extraction in real-time. Additionally, it highlights the benefits of a scalable system that integrates seamlessly with Apache Spark to enhance data processing capabilities.
Storage for big-data by Joshua Robinson
- 1. JOSHUA ROBINSON FLASHBLADE ENGINEERING, PURE STORAGE From Big Data To Big Intelligence: Spark Meets Flashblade A Pure Engineering Use Case
- 2. © 2017 PURE STORAGE INC. 2 ALL-FLASH STORAGE FOR DATA-INTENSIVE COMPUTING
- 3. © 2017 PURE STORAGE INC. 3 FLASHBLADE FOR BIG DATA ANALYTICS FAST DATA BIG DATA X AGILE DATA X = DATA ADVANTAGE
- 4. © 2017 PURE STORAGE INC. 4 TRADITIONAL DATA ANALYTICS EXAMPLE SALES FEED/ PIPELINE CRM ENGINEERING TICKETS EXTRACT AGGREGATE ANALYTICS PARAMETERS PRODUCT LOGS RAW LOG STORAGE GREP, AWK, ETC STORAGE COMPUTE
- 5. © 2017 PURE STORAGE INC. 5 MODERN BIG DATA ANALYTICS EXAMPLE SALES FEED/ PIPELINE CRM ENGINEERING TICKETS PRODUCT LOGS EXTRACT AGGREGATE
- 6. © 2017 PURE STORAGE INC. 6 INFRASTRUCTURE OF BIG DATA WAREHOUSES >6PBs ACROSS 100s OF HETEROGENEOUS DATA SILOS
- 7. © 2017 PURE STORAGE INC. 7 BIG DATA WAREHOUSES INFRASTRUCTURE Compute Storage BIG FAST SIMPLE
- 8. © 2017 PURE STORAGE INC. 8 INTRODUCING FLASHBLADE™ ALL-FLASH FILE AND OBJECT STORAGE BIG Up to 8 PBs FAST 75 GBps / 8M IOPS SIMPLE Seamlessly scalable BLADE PURITY FABRIC
- 9. Automating triage of test failures in SW development A Pure Engineering Use Case
- 10. © 2017 PURE STORAGE INC. 10 THE PROBLEM Handful 1 Test coordinator (Jenkins) Handful Handful 100s of tests
- 11. © 2017 PURE STORAGE INC. 11 THE PROBLEM 1,000 test failures 20,000+ tests / day 20 Engineers 2x in the next 12 months 1000+ VMs 120+ FBs 20+ Jenkins 400+ clients 100+ Engineers
- 12. © 2017 PURE STORAGE INC. 12 THE DREAM 1. Automate triaging of failures as much as possible 2. Extract performance metrics from the logs 3. Save our logs for future use 4. Do all of this in a scalable system 5. Real-time results!
- 13. © 2017 PURE STORAGE INC. 13 OUR DATA ANALYTICS PIPELINE 10 FB 20 clients 100+ tests 12 12 12 12 rsyslog 12 12 12 12 12 12 12 12
- 14. © 2017 PURE STORAGE INC. 14 OUR DATA ANALYTICS PIPELINE 100 FB 200 clients 1,000+ tests 12 12 12 12 rsyslog 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12
- 15. © 2017 PURE STORAGE INC. 15 OUR DATA ANALYTICS PIPELINE 100 FB 200 clients 1,000+ tests 12 12 12 12 rsyslog 12 12 12 12 12 12 12 12 12 12 12 12 12 12
- 16. © 2017 PURE STORAGE INC. 16 OUR DATA ANALYTICS PIPELINE 120+ FB 400+ clients 4,000+ tests 12 12 12 12 rsyslog 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12
- 17. © 2017 PURE STORAGE INC. 17 OUR DATA ANALYTICS PIPELINE 1,000+ VMs 120+ FBs 20+ Jenkins 400+ clients 16 16 16 16 rsyslog 12 12 12 12 12 12 12 12 12 12 6G 40 40 40 40 18T 18T 6T 6G 12 Custom code ü Duplicate bug ü Infrastructure failure ü Performance regression 20,000+ tests
- 18. © 2017 PURE STORAGE INC. 18
- 19. © 2017 PURE STORAGE INC. 19 Processed: 18 TB 30 Billion events per day Extracted: 6 GB 8 Million events per day
- 20. © 2017 PURE STORAGE INC. 20 THE POWER OF DATA ANALYTICS 20,000+ tests 1,000 test failures 20,000+ tests ~30 distinct test failures Data Analytics Pipeline
- 22. © 2017 PURE STORAGE INC. 22 SCALING STORAGE
- 23. © 2017 PURE STORAGE INC. 23 FLASHBLADE GUI
- 24. © 2017 PURE STORAGE INC. 24 SCALING COMPUTE
- 25. © 2017 PURE STORAGE INC. 25 OUR DATA ANALYTICS PIPELINE 1,000+ VMs 120+ FBs 20+ Jenkins 400+ clients 16 16 16 16 rsyslog 12 12 12 12 4G 40 40 40 40 18T 18T 6T 4G 12 Custom code 20,000+ tests 12 12 12 12
- 26. © 2017 PURE STORAGE INC. 26 ANALYTICS PIPELINE SCALING COMPUTE 1. Download docker image 2. Mount FlashBlade on container 3. Hot-add to Spark cluster ALL OF THIS CAN BE DONE IN A SINGLE COMMAND WITHOUT DISRUPTING YOUR SPARK JOBS!
- 27. © 2017 PURE STORAGE INC. 27 OUR DATA ANALYTICS PIPELINE 1000+ VMs 120+ FBs 20+ Jenkins 400+ clients 16 16 16 16 rsyslog 12 12 12 12 12 12 12 12 12 12 6G 40 40 40 40 18T 18T 6T 20,000+ tests 6G 12 Custom code
- 28. © 2017 PURE STORAGE INC. 28 INFRASTRUCTURE AGILITY AD-HOC AND BURSTY ANALYTICS rsyslog 1000-CORE SPARK CLUSTER ON FLASHBLADE
- 29. © 2017 PURE STORAGE INC. 29 INFRASTRUCTURE SIMPLICITY ⎯ Physical Consolidation ⎯ Density: Multiple racks to a single FlashBlade ⎯ Management Consolidation ⎯ Non-disruptive upgrades ⎯ Storage capacity planning, data access, security ⎯ Backups and Restores
- 30. © 2017 PURE STORAGE INC. 30 FLASHBLADE File & Object AND 2.5 PBs (1:1) N+2 REDUNDANCY Purity PLUS Pure1 17TB 52TB BLADES Power 1150Watt/PB 8M IOPs AND 75 GB/s PERFORMANCE
- 31. © 2017 PURE STORAGE INC. 31 RESOURCES Apache Spark White Papers: 1) Guide to Supporting On-Premise Spark Deployments with a Cloud-Scale Data Platform 2) Engineering Unplugged: A Discussion with Pure Storage's Brian Gold on Big Data Analytics for Apache Spark Big Data Analytics: purestorage.com/analytics FlashBlade Product Info: purestorage.com/flashblade