Spark & Hadoop at Production at Scale
Download as PPTX, PDF12 likes2,423 views
This document discusses how Spark can be used for production scale applications. It provides examples of companies using Spark and MapR in production for tasks like security analytics, genomics research, and customer analytics. It also outlines key issues to consider when taking Spark to production and describes how MapR provides the performance, reliability, support and data services needed for mission critical Spark applications.
Spark & Hadoop at Production at Scale
- 1. © 2015 MapR Technologies 1© 2015 MapR Technologies Taking Your Spark To Production Scale
- 2. © 2015 MapR Technologies 2 The Journey To Production Scale Trials, science projects Large mission-critical, operational deployments © 2015 MapR Technologies 2
- 3. © 2015 MapR Technologies 3 Companies with Spark & MapR in Production GLOBAL TELECOM HEALTHCARE GLOBALFINANCIAL SERVICES
- 4. © 2015 MapR Technologies 4 Key Issues To Plan For 3
- 5. © 2015 MapR Technologies 5 Global Managed Security Services delivered on Hadoop Spark Stream processing used to first check for known threats Data next processed on Hadoop using MLLib and GraphX Additional SQL querying done via Spark SQL Security Intelligence Operations
- 6. Delivers Lightning Fast Analytics for Clients Building largest Hadoop cluster in Australia Real-time analytics using Spark on MapR–reducing data loading time from hours to minutes Leverage multi-tenancy, high-performance and reliability of MapR
- 7. © 2015 MapR Technologies 7 Next-Gen Genomics Develop flexible platform to keep up with fast changing research techniques POSIX file access lets bio-informaticians use existing tools with open source tools (Spark) Graph manipulations can be done reliably and at scale using Spark
- 8. © 2015 MapR Technologies 8 Real-Time Customer Analytics • MapR Data Lake stores both online and archive data • Spark on MapR reduced ETL processing • NFS moved data into the cluster seamlessly • 1/10th Total Cost of Ownership vs. old way • New customer onboarding cut from months to weeks
- 9. © 2015 MapR Technologies 9 Databricks & MapR Strategic Partnership (since April 2014) Support for the complete Spark stack Engineering & roadmap collaboration Back-end support +
- 10. © 2015 MapR Technologies 10 The Most Complete Spark Environment Spark SQL (SQL) Spark Streaming (Streaming) MLlib (Machine learning) GraphX (Graph computation) Foundation For Enterprise-Grade Spark
- 11. © 2015 MapR Technologies 11 DB Operations Real-Time and Actionable Analytics Operations + Analytics on One Hadoop Platform with SQL Access Mobile application server Customer 360 dashboard Churn analysis Product/service optimization and personalization Real-time ad targeting Web application server Data exploration (SQL) • User profiles and state • User interactions • Real-time location data • Web and mobile session state • Comments/rankings
- 12. © 2015 MapR Technologies 12 Spark + MapR = Ready For Production Success World-record performance on disk High Performance SLA-Driven Applications • High availability • Data protection • Disaster recovery Reliability for Production Strategic partnership with Databricks to ensure enterprise support for the entire stack 24/7 Best-in-class Global Support MapR-DB + Spark = real-time analytics Operational Data Store
- 13. © 2015 MapR Technologies 13 Free On-Demand Training www.mapr.com/training
- 14. © 2015 MapR Technologies 14 Self-Service Data Exploration Data Agility with Less IT Required Single SQL Interface for Structured and Semi-Structured Data
- 15. © 2015 MapR Technologies 15 MapR Introduces 3 New Spark-Based Quick Start Solutions Real-Time Security Log Analytics Time Series Analytics Genome Sequencing
- 16. © 2015 MapR Technologies 16 Get Your Tattoo In The MapR Booth! Show off your Kickstart My Heart skills and enter to win Xbox 360 & Guitar Hero
- 17. © 2015 MapR Technologies 17 Top-Ranked NoSQL Top-Ranked Hadoop Distribution Top-Ranked SQL-on Hadoop Solution
Editor's Notes
- #3: JN: I’d make the arrow not absolutely vertical tilt steeply to the right
- #6: http://www.thinkstockphotos.com/image/stock-photo-businessman-selecting-a-futuristic-padlock/179694007/popup?sq=Datacenter%20security/f=CPIHVX/s=Popularity
- #8: http://www.thinkstockphotos.com/image/stock-photo-scientist-holding-test-tube/461208473/popup?sq=drug%20discovery/f=CPIHVX/s=Popularity a. Interested in Adam project - runs on top of Spark - for nextgen genomics - good whiteppaer - search for APche spark Adam Git hub for Adam - Notes - links to the whitepaper - AMPLab - http://www.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-207.html Genomics realignment tool - crux - next ten gemonic medicine - allows you to much more quickly access and manage the alignment of data. b. Existing genomics pipeline - many weeks to realign - drill down augment and working through their chemical compounds - genetics can come and test their compound against the alignment c. Not the exact sections of then they cannot focus - back out and zoom back in on the right set of sequence. that process 6 weeks - geneticist - 1 day and shift it - 6 weeks to get to the change - Adam in a matter of hours - geneticists can do it themselves - whole team of HPC experts otherwise d. this is the case for almost all pharma companies - Novartis is also the same e. One tool in a bigger framework - several other use cases as well.
- #9: Objectives: Razorsight's cloud-based predictive analytics software delivers insights to help communications service providers (CSPs) and media companies optimize operations and offer superior customer experiences. “As we grew as a company and big data evolved, there was a lot more data available,” explains Razorsight CTO Suren Nathan. “Today’s data has higher volumes and different structures. There are new types of devices generating data for the Internet of Things, mobile phones using broadband for apps, and VoIP.” Challenges Razorsight’s old technology platform could not keep up with the demands and opportunities of this increasing volume of data. Storage costs were exploding and they wanted to be able to maintain performance and scalability at a lower cost. The prior platform was based on IBM Pure Data (Netezza) as a data warehouse appliance and couldn’t support high-speed data ingestion and reporting. Additionally it required having a separate database for each customer. Solution MapR came out on top for several reasons and is being used as the primary data store for online and archive data Having the flexibility of the full Spark stack as part of the Hadoop distribution was very important. Spark helps transition a large part of ETL processing 2. MapR provided production-class Hadoop with enterprise support 3. The NFS gateway was critical for easy, high-speed access. Business Impact Customers such as Virgin Mobile LA are reducing churn by tailoring campaigns to the right subscriber at the right time. Much lower TCO compared to IBM Netezza ( 1/8th the cost). Razorsight is able to also reduce on-boarding time for new customers from 4-6 months to 8-12 weeks.
- #11: Spark execution engine rides on the Mesos framework and any distributed file system (in this case HDFS)
- #15: Data Agility with No IT Intervention
- #18: Just Thought You Might Want To Know….