IBM Strategy for Spark
Download as PPTX, PDF1 like765 views
This document provides an introduction and overview of Spark: - Spark is an open-source in-memory data processing engine that can handle large datasets across clusters of computers using an API in Scala, Python, or R. - IBM is heavily committed to Spark, contributing the most code and fixing the most issues reported by other organizations to continually improve the full analytics stack. - An example is presented on using Spark to predict hospital readmissions from diabetes patient data, obtaining AUC scores comparable to other published models.
IBM Strategy for Spark
- 2. Introductions Garrett Young (sgyoung@us.ibm.com) 1) Introduction to Spark (10 mins) 2) IBM's Commitment to Spark (5 mins) 3) How Predictive Analytic Lifecycles Typically Work (10 mins) 3) Using Spark to Predict Hospital Readmissions (15 mins) 4) How you can get a free-trial Spark environment from IBM (5 mins) 5) Q&A (15 mins)
- 3. What is Spark? • In-memory data processing engine • Open Source Apache Project • Cluster Computing Framework • Can use Scala, Python or R Languages • Horizontally/Vertically Scalable • Not a data store
- 4. IBM | SPARK – The Analytics Operating System “Enabling New Classes of Intelligent Applications Embedded with Analytics” • Spark unifies data, enabling real-time insights • Spark processes and analyzes data from any data source • Spark is complementary to Hadoop, but faster with in-memory performance • Build models quickly. Iterate faster. Apply intelligence .
- 5. • Traditional Approach: MapReduce jobs for complex jobs, interactive query, and online event-hub processing involves lots of (slow) disk I/O How Spark Works HDFS Read HDFS Write HDFS Read HDFS Write Input ResultCPU Iteration 1 Memory CPU Iteration 2 Memory
- 6. • Solution: Keep more data in-memory with a new distributed execution engine HDFS Read Input CPU Iteration 1 Memory CPU Iteration 2 Memory faster than network & disk Zero Read/Write Disk Bottleneck How Spark Works Chain Job Output into New Job Input
- 7. General Spark Architecture Overview • Driver Uses Spark Context to talk to the Cluster Manager • Executors run their own JVM Processes • Cluster manager distributes the workload based on information from the Worker
- 8. Key Reasons for the Interest in Spark Performant In-memory architecture greatly reduces disk I/O Anywhere from 20-100x faster for common tasks Productive Concise and expressive syntax, especially compared to prior approaches Single programming model across a range of use cases and steps in data lifecycle Integrated with common programming languages – Java, Python, Scala, R New tools continually reduce skill barrier for access (e.g. SQL for analysts) Leverages existing investments Works well within existing Hadoop ecosystem Improves with age Large and growing community of contributors continuously improve full analytics stack and extend capabilities
- 9. What is SparkML? MLlib is Spark’s machine learning (ML) library. Its goal is to make practical machine learning scalable and easy. At a high level, it provides tools such as: 1. ML Algorithms: common learning algorithms such as classification, regression, clustering, and collaborative filtering 2. Featurization: feature extraction, transformation, dimensionality reduction, and selection 3. Persistence: saving and load algorithms, models, and Pipelines 4. Utilities: linear algebra, statistics, data handling, etc.
- 10. What is scikit-learn? • Used for Data Mining and Data Analysis • Open Source • Various classification, regression and clustering algorithms
- 11. Watson Machine Learning • Uses both Spark ML and Scikit-Learn plus others • Built on SPSS plaform • Can pull from many different data sources • Integrates with DSX (Beta)
- 12. Web Service Data Access: • Easily connect to Behind- the-Firewall and Public Cloud Data • Catalogued and Governed Controls through Watson Data Platform Creating Models: • Single UI and API for creating ML Models on various Runtimes • Auto-Modelling and Hyperparameter Optimization Web Service: • Real-time, Streaming, and Batch Deployment • Continuous Monitoring and Feedback Loop Intelligent Apps: • Integrate ML models with apps, websites, etc. • Continuously Improve and Adapt with Self-Learning IBM DSX Machine Learning IMS
- 13. IBM Machine Learning in Data Science Experience API for Jupyter Notebooks Wizard GUI IBM Machine Learning is provisioned by default in Data Science Experience • Enables Data Scientists to deploy machine learning models as web services • Single UI for creating, collaborating, deploying, monitoring, and feedback • Accessible via API, Wizard GUI, and Canvas
- 14. IBM's Commitment to Spark
- 15. Spark Tech Center (STC): IBM’s Commitment to Spark 0 100 200 300 400 500 600 700 800 900 1000 Databricks IBM Hortonworks Cloudera Intel IVU Traffic Technologies Tencent Top 7 Contributing Companies to Spark 2.0.0 25,600 Spark LOC 606 Spark JIRAs 253 SystemML JIRAs 64 Speakers at events … and all that with 1 Team 1.5 Years Databricks Hortonworks Cloudera Intel Tencent NTT Other
- 16. IBM Spark Technology Center – San Francisco, CA As of March 10, 2016 See what we’re up to … IBM Spark Technology Center http://www.spark.tc/blog/ Fixing lot’s of issues reported by others
- 17. Using Spark to Predict Hospital Readmissions & How Predictive Analytic Lifecycles Typically Work
- 18. Reducing Hospital Readmissions with Predictive Analytics An Example ‘Proof of Concept’ Using Open Data
- 21. Problem : 30-Day Hospital Readmissions costs $41B Annually Source: http://www.hcup-us.ahrq.gov/reports/statbriefs/sb172-Conditions-Readmissions-Payer.pdf
- 22. Medicare HRRP – Penalties to Hospitals Source: Kaiser Family Foundation http://kff.org/medicare/issue-brief/aiming-for-fewer-hospital-u-turns-the-medicare-hospital-readmission-reduction-program/
- 24. Get Data: Diabetes Readmissions Dataset • University of California Irvine – Machine Learning Repos. • Open Data • 130 Hospitals, 1999-2008 • 101,766 rows, 50 columns of data • Diabetes Readmissions • Top ten for Medicaid, Private Insurance and Uninsured • Not in top ten for Medicare https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008
- 25. Build a Predictive Model : Conceptual View Step 1: Model Development Step 2: Perform Predictions Historical Data Machine Learning (Mathematical Algorithm) Model Model PredictionNew Case
- 26. IBM Bluemix • Bluemix • Infrastructure, Watson, software and services on Bluemix Cloud Platform • Services such as Big Insights (Hadoop), Data Connect (ETL), and Spark can be almost instantly provisioned
- 27. Data Science Experience (DSX) • Data Science Experience (DSX) • Easily execute scala, python and R notebooks • Share notebooks with your data science team
- 28. Bluemix Services Architecture in the Cloud BigInsights HDFS (Hadoop) Data Connect DashDB Data Science ExperienceCloudantNode.js Web Form Training Data Convert to CSV Predictions New Records Predictions
- 30. A Look at The Raw Data
- 31. Data Science Experience – Python Code
- 33. First Pass Results – Are they any good? AUC = Area Under the Curve AUC Score 0.6514 0.50 = Random Guessing 1.00 = Perfect Prediction
- 34. 2nd Pass Results – Are they any good? AUC = Area Under the Curve AUC Score 0.6750 0.50 = Random Guessing 1.00 = Perfect Prediction
- 35. How Do Other Readmission Models Perform? “A comparison of models for predicting early hospital readmissions” Journal of Biomedical Informatics Volume 56, August 2015, Pages 229– 238 Source: http://www.sciencedirect.com/science/article/pii/S1532046415000969
- 36. Which Factors Affect Diabetes Readmission? Data: Feature Importance from Random Forest Algorithm The Algorithm can tell us which features (columns) it found important during the training process. 22 columns from original 50
- 38. Summary • Readmissions Prediction is an important area of research for using Predictive Analytics in Healthcare • Patient: Improved Outcome • Hospital Providers: Avoid Penalties • Payers: Reduce Costs • In a short amount of time we were able to develop results comparable to leading research studies
- 39. How you can get a free-trial Spark Cluster from IBM
- 40. Sign Up for Free Account Data Science Experience with IBM ML https://ibm.box.com/s/y2zvpzk8pje56lto0oja0372tnbydbomhttp://datascience.ibm.com/ Notebook Samples
Editor's Notes
- #4: 3