IP EXPO Nordic: Data Science in the Cloud
2 likes331 views
The document discusses data science in the cloud, highlighting its key components such as data collection, storage, exploration, and result publication. It emphasizes the importance of utilizing various data sources, including traffic collisions and weather data, and the benefits of cloud computing for scalability and collaboration. Additionally, it presents tools and frameworks like Jupyter notebooks and PixieDust for data analysis and visualization.
IP EXPO Nordic: Data Science in the Cloud
- 1. Data Science in the Cloud Margriet Groenendijk IP EXPO Nordic Stockholm - 20 September 2017
- 3. Data Science?What is Data Science? @MargrietGr http://visual.ly/exports-and-imports-scotland
- 4. Data Science?What is Data Science? @MargrietGr
- 5. Data Science?What is Data Science? @MargrietGr
- 6. Data Science is⌠Big Data? @MargrietGr
- 7. Data Science is⌠Extracting new insights from data @MargrietGr
- 8. Data Science is⌠Using the Scientific Method @MargrietGr
- 10. @MargrietGr Discover Data Use Data Publish Data Socialize Data The ideal workflow
- 13. Data Science in the Cloud @MargrietGr
- 14. Normal Day 325TB per day 2.4 Billion API requests every 15 minutes 50,000 videos played @MargrietGr The Weather Company Hurricane Harvey 500TB per day 3.4 Billion API requests every 15 minutes 750,000 videos played
- 15. Example @MargrietGr Traffic Collisions and Weather www.pexels.com/photo/blur-cars-dew-drops-125510/
- 16. Collect Data Store Data Explore Data Publish Results @MargrietGr
- 17. Collect Data Explore Data Publish Results @MargrietGr Store Data
- 18. NYPD Traffic Collisions @MargrietGr 812,526 Collisions since April 2014 https://data.cityofnewyork.us/Public-Safety/NYPD- Motor-Vehicle-Collisions/h9gi-nx95
- 20. Collect Data Traffic Collisions Historic Weather Store Data Explore Data Publish Results @MargrietGr
- 21. Store Data Object Store Relational Database NoSQL Database @MargrietGr
- 22. Store Data Object Store Static data Unstructured data Relational Database NoSQL Database @MargrietGr
- 23. Store Data Object Store Static data Unstructured data Relational Database Tables with a fixed schema NoSQL Database @MargrietGr
- 24. Store Data Object Store Static data Unstructured data Relational Database Tables with a fixed schema NoSQL Database Everything else @MargrietGr
- 25. Collect DataCollect Data Traffic Collisions Historic Weather Store Data Object-Store Explore Data Publish Results @MargrietGr
- 28. Explore data in Jupyter notebooks @MargrietGr
- 29. Notebooks in the Cloud @MargrietGr https://datascience.ibm.com/
- 30. @MargrietGr Organized in Collaborative Projects
- 31. Explore data in notebooks in the Cloud @MargrietGr
- 33. Collisions Data @MargrietGr Load into Pandas DataFrame
- 34. Collisions Data @MargrietGr Load into Pandas DataFrame
- 35. Explore Data with PixieDust @MargrietGr Load into Pandas DataFrame
- 36. Explore Data with PixieDust @MargrietGr
- 37. @MargrietGr Explore Weather Data with PixieDust
- 38. Hypothesis The number of traffic collisions is influenced by the weather @MargrietGr
- 39. Combine collisions and weather data @MargrietGr
- 40. Combine collisions and weather data @MargrietGr
- 42. Get the data ready for analysis @MargrietGr
- 43. @MargrietGr
- 44. Other models @MargrietGr Dimensionality Reduction (PCA) Stepwise Linear Regression Decision Trees Random Forest Kmeans clustering
- 45. Collect DataCollect Data Traffic Collisions Historic Weather Store Data Object-Store Explore Data New hypothesis Add more data Aggregate and summarize in different ways Publish Results @MargrietGr
- 46. Collect DataCollect Data Traffic Collisions Historic Weather Store Data Object-Store Explore Data New hypothesis Add more data Aggregate and summarize in different ways Publish Results @MargrietGr
- 47. Publish Results @MargrietGr Use Weather forecast Data Model as a RESTful API - Flask app deployed in Bluemix Watson Machine Learning Service
- 48. Publish Results â PixieApp in a notebook @MargrietGr PixieApps Dashboards within a notebook
- 49. Why Data Science to the Cloud? @MargrietGr Scales up â unlimited resources All tools connected Github integration Local development, easy to move Collaboration!
- 50. Thank you! Dr. Margriet Groenendijk Developer Advocate mgroenen@uk.ibm.com @MargrietGr Slides https://www.slideshare.net/MargrietGroenen dijk/presentations Blog https://medium.com/ibm-watson-data-lab @MargrietGr IBM Data Science Experience https://datascience.ibm.com PixieDust https://ibm-cds-labs.github.io/pixiedust/ Notebooks https://github.com/ibm-cds-labs/python- notebooks Weather Data https://business.weather.com/products/weather- data-packages IBM Bluemix https://console.ng.bluemix.net/
- 51. @MargrietGr