Democratizing AI with Apache Spark
Download as PPTX, PDF4 likes2,523 views
This document discusses democratizing AI using Apache Spark. It summarizes that while AI is advancing rapidly, it has not been fully democratized due to challenges with data management, developing productive teams, and establishing production-ready applications. Databricks aims to close these gaps with its just-in-time data platform that provides integrated workspaces, automated Spark management, and supports deep learning use cases across industries.
Democratizing AI with Apache Spark
- 1. Democratizing AI with Apache Spark Ali Ghodsi Co-Founder and CEO
- 2. AI is changing the world 2 Why now? AlphaGoSIRI/assistantsSelf-driving cars
- 3. Data is the catalyst 3 AI hasn’t been democratized Better training, tuning, validation More data Clickstreams Sensor data (IoT) Video Speech Handwriting …
- 4. The hardest part of AI isn’t AI 4 “Hidden Technical Debt in Machine Learning Systems “, Google NIPS 2015 How do we democratize AI?
- 5. 5 “Hidden Technical Debt in Machine Learning Systems “, Google NIPS 2015 + AI FLEXIBLE FAST BIG DATA
- 6. Some gaps remain 6 Manage Data infrastructure • Create, configure, monitor resilient big data clusters. • Securely access silos of disparate data sources. • Enforce proper data governance. •1 Empower teams to be productive • Interactively explore data and prototype ideas. • Securely share big data clusters among analysts. • Debug, troubleshoot, version-control big data applications.• 2 Establish Production- Ready Applications • Setup robust ML data pipelines for ETL/ELT. • Productionize real-time applications with HA, FT. • Build, serve, maintain advanced machine learning models. • 3
- 7. Databricks: Closing the gap 7 • Separate compute & storage • Integrate existing data stores • Efficient cache on first access Just-in-Time Data Platform1 Agile + Low TCO • Interactive notebooks, dashboards, reports • Real-time exploration, machine learning, graph use cases Integrated Workspace2 Accelerate Time to Value • Workflow scheduler for ML, streaming, SQL, ETL • Performance-optimized, high availability, fault- tolerant Automated Spark Management3 Performance
- 8. Enterprise AI use-cases 8 Predict credit score, credit limit, anomalies Predict energy demand based on massive weather data Natural language processing to extract author graph Predict player churn, predicting network outages Predict machine equipment failure
- 9. New Frontier of AI: Deep Learning 9 Detect cancer Understand speech Infer location Identify landmarks in photos Recognize Mandarin and English Improve cancer detection
- 10. Faster and easier deep learning with Databricks 10 GPUs • TensorFlow: The most popular deep learning framework. • TensorFrames: Makes TensorFlow computations faster and easier to program on Spark. TensorFlow on TensorFrames and GPUs support out-of-the-box Massive parallelism
- 11. Deep Learning on Databricks 11 Data Ingest Feature extractio n Model Training Product- ionize Clusters Jobs & Workflows TensorFrames + GPUs Interactive exploration Just-in-time data platform Automated management
- 12. Thank you.
- 13. Deep Learning references • Image recognition (Geo ID): – https://www.technologyreview.com/s/600889/google-unveils- neural-network-with-superhuman-ability-to-determine-the-location- of-almost/ • Cancer screening: – http://www.popsci.com/how-deep-learning-technology-could-be- next-step-in-cancer-detection – https://blogs.nvidia.com/blog/2016/09/19/deep-learning-breast- cancer-diagnosis/ • Speech translation: – https://www.technologyreview.com/s/544651/baidus-deep- learning-system-rivals-people-at-speech-recognition/ 13
- 14. PHARM A MEDIA INDUSTRIA L Generating programs based on Nielsen ratings Predictive Analytics Analytics Transforming Industries 15 Real-time detection of failing wind-turbines Anomaly Detection Predicting Diabetes in Rural Counties Next-Gen Product R&D Real-time Data-Driven Analytics Applications
- 15. DATA WAREHOUSE S HADOOP / DATA LAKESYour Storage CLOUD STORAGE Orchestrated Spark In The Cloud 16 Databricks Just-in-Time Data Platform Integrated WorkspaceDASHBOARD S Reports NOTEBOOKS github, viz, collaboration EnterpriseSecurity AccessControl,Auditing,Encryption BI Tools Open Source Your Custom Spark Apps • Clusters: Auto-scaled, resilient, multi-tenant • Data Integration: Universal secure and fast • Interfaces: BI tools & REST API Databricks Managed Services PRODUCTION JOBS + Powered by Apache Spark
- 16. Data Warehouses Hadoop / Data Lakes YOUR STORAGE Cloud Storage Orchestrated Spark In The Cloud Databricks Just-in-Time Data Platform Integrated Workspace Dashboards Reports Notebooks github, viz, collaboration BI Tools Open Source Your Custom Spark Apps • Clusters: Auto-scaled, resilient, multi-tenant • Data Integration: Universal secure and fast • Interfaces: BI tools & REST API Production Jobs Powered by Apache Spark + Managed Services Your Storage Enterprise SecurityAccess Control, Auditing, Encryption
- 17. Today’s Data Reality 18 Siloed, Unstructured, Fast-Growing Data Data Warehouses Hadoop / Data LakesCloud Storage
- 18. Databricks Just-in-Time Data Platform Powered by Apache Spark® ™ Integrated Enterprise Security Framework Integrated Workspace Notebooks Dashboards Your Custom Spark Apps Production Jobs BI Tools Orchestrated Apache® Spark™ in the Cloud Open Source Managed Services+ Your Storage Cloud Storage | Data Warehouses | Data Lakes OPTIONAL
Editor's Notes
- #3: THIS IS NOT JUST SCI-FI
- #6: BUT SPARK DOESN’T GET YOU ALL THE WAY THERE
- #8: 1: Global publisher Elsevier – team in US and EU perform natural language processing on all their content to experiment with new product ideas. 2: Energy analytics company DNV GL – Databricks sped up analytics of IoT data from weather and grid sensor by 100x. 3: Financial service provider LendUp– Databricks enabled them to update their machine models daily instead of weekly.
- #9: TRANSITION TO NEXT: A NEW USE CASE COMPANIES HAVE BEEN ASKING FOR
- #10: THERE IS A LOT OF INTEREST IN DEEP LEARNING, THE MACHINE LEARNING TECHNIQUE THAT HAS ACHIEVED REMARKABLE RESULTS ** PICTURE IS BRUSSELS **
- #11: IN RESPONSE TO THE GROWING DEMAND FOR DEEP LEARNING ON SPARK FROM THE COMMUNITY, DATABRICKS HAS BEEN WORKING HARD, AND IS PROUD TO ANNOUNCE TODAY THAT…
- #12: In summary, Databricks mission has always been to make big data simple. With the new developments we are taking a next step in this journey, DEMOCRATIZING DEEP LEARNING by making it simple for all
- #15: WHY DID WE START DATABRICKS? THIS IS WHERE THE WORLD IS GOING
- #17: The DatabricksEnterprise Spark Platform Built by the creators of Spark Contribute 75% of the code Trained 20,000+ users Largest # of customers deploying Spark Spark is becoming the de facto standard for big data Started at UC Berkeley AMPLab in 2009 Most active open source project in data processing (1000+ contributors) Today, the de facto technology standard for big data processing is Apache Spark. Spark is an open source data processing framework that was built for speed, ease of use, and scale. It has been adopted by thousands of companies for every type of workload because it solves some of the most pressing data challenges: Ability to process a wide variety of data types. Unlimited scalability, for organizations with rapidly growing data. Flexible enough to be customized for many use cases and with a wide variety of programming languages. Databricks was founded by the creators Apache Spark. In addition to creating Spark, we continue to be the driving force behind Spark contributing over 75% of the code every release. (10X more than any other vendor); trained 20K Spark users on Databricks (more users trained than any other vendor) and has the largest number of customers deploying Spark (>200)…more than any other vendor. Additional info on Spark if required: Much of its benefits are due to how it unifies critical data analytics capabilities such as SQL, machine learning and streaming in a single framework. This enables enterprises to simultaneously achieve high performance computing at scale while simplifying their data processing infrastructure by avoiding the difficult integration of many disparate and difficult tools with a single powerful yet simple alternative.
- #18: The DatabricksEnterprise Spark Platform Built by the creators of Spark Contribute 75% of the code Trained 20,000+ users Largest # of customers deploying Spark Spark is becoming the de facto standard for big data Started at UC Berkeley AMPLab in 2009 Most active open source project in data processing (1000+ contributors) Today, the de facto technology standard for big data processing is Apache Spark. Spark is an open source data processing framework that was built for speed, ease of use, and scale. It has been adopted by thousands of companies for every type of workload because it solves some of the most pressing data challenges: Ability to process a wide variety of data types. Unlimited scalability, for organizations with rapidly growing data. Flexible enough to be customized for many use cases and with a wide variety of programming languages. Databricks was founded by the creators Apache Spark. In addition to creating Spark, we continue to be the driving force behind Spark contributing over 75% of the code every release. (10X more than any other vendor); trained 20K Spark users on Databricks (more users trained than any other vendor) and has the largest number of customers deploying Spark (>200)…more than any other vendor. Additional info on Spark if required: Much of its benefits are due to how it unifies critical data analytics capabilities such as SQL, machine learning and streaming in a single framework. This enables enterprises to simultaneously achieve high performance computing at scale while simplifying their data processing infrastructure by avoiding the difficult integration of many disparate and difficult tools with a single powerful yet simple alternative.
- #20: The DatabricksEnterprise Spark Platform Built by the creators of Spark Contribute 75% of the code Trained 20,000+ users Largest # of customers deploying Spark Spark is becoming the de facto standard for big data Started at UC Berkeley AMPLab in 2009 Most active open source project in data processing (1000+ contributors) Today, the de facto technology standard for big data processing is Apache Spark. Spark is an open source data processing framework that was built for speed, ease of use, and scale. It has been adopted by thousands of companies for every type of workload because it solves some of the most pressing data challenges: Ability to process a wide variety of data types. Unlimited scalability, for organizations with rapidly growing data. Flexible enough to be customized for many use cases and with a wide variety of programming languages. Databricks was founded by the creators Apache Spark. In addition to creating Spark, we continue to be the driving force behind Spark contributing over 75% of the code every release. (10X more than any other vendor); trained 20K Spark users on Databricks (more users trained than any other vendor) and has the largest number of customers deploying Spark (>200)…more than any other vendor. Additional info on Spark if required: Much of its benefits are due to how it unifies critical data analytics capabilities such as SQL, machine learning and streaming in a single framework. This enables enterprises to simultaneously achieve high performance computing at scale while simplifying their data processing infrastructure by avoiding the difficult integration of many disparate and difficult tools with a single powerful yet simple alternative.