Data Science in the Cloud
1 like230 views
Margriet Groenendijk gave a presentation on data science in the cloud. She discussed her background working with large datasets and using tools like Python, Spark, R, and IBM's cloud services. She then outlined the typical data science workflow of collecting and storing data, exploring and cleaning it, creating predictive models, and presenting results. Finally, she demonstrated an example of analyzing weather and Twitter sentiment data using various IBM cloud tools.
Data Science in the Cloud
- 1. @MargrietGr Margriet Groenendijk Developer Advocate for IBM Cloud Data Services SW Cloud meetup Bristol 24 November 2016 Data Science in the Cloud
- 2. @MargrietGr About me • Developer Advocate at IBM Cloud Data Services, UK •Data science •Python, Spark, R, Cloudant, dashDB • Research Fellow at University of Exeter, UK •Worked with very large observational datasets and the output of global scale climate models • PhD at Vrije Universiteit Amsterdam, the Netherlands •Explored large observational datasets of carbon uptake by forests
- 6. @MargrietGr 1960s http://www.climatecentral.org/news/first-climate-model-video-19007
- 7. @MargrietGr Data Engineers Data Scientists Business Analysts App Developers Data Science is a Team Effort Data
- 12. @MargrietGr Collect Data APIs Open Data Maps Web Scraping Time Series
- 13. @MargrietGr Store Data Object Store - binary files Relational database Document store - json Bluemix https://console.ng.bluemix.net/
- 17. @MargrietGr The Spark Stack from Karau et al.: Learning Spark
- 18. @MargrietGr RDDs : Resilient Distributed Datasets • Data does not have to fit on a single machine • Data is separated into partitions • Creation of RDDs •Load an external dataset •Distribute a collection of objects • Transformations construct a new RDD from a previous one (lazy!) • Actions compute a result based on an RDD
- 19. @MargrietGr Run Spark locally in a Python notebook https://www.continuum.io/downloads http://spark.apache.org/downloads.html Create a new kernel to use in a Jupyter notebook
- 20. @MargrietGr Jupyter Notebooks! • Server-client application to edit and run notebook documents via a web browser • Cells with: •Code •Figures and tables •Rich text elements • Different kernels: Python, R, Scala, Spark In the Cloud:
- 22. @MargrietGr
- 23. @MargrietGr
- 24. @MargrietGr
- 26. @MargrietGr Define Question What will the weather be next weekend? https://unsplash.com/search/autumn?photo=LSF8WGtQmn8 https://unsplash.com/search/rain?photo=19tQv51x4-A
- 28. @MargrietGr Explore Data Python packages • requests and json •API credentials and latitude/longitude of Bristol •json data returned • pandas, numpy and datetime •convert json to pandas DataFrame (table with multiple indices) •add time as index
- 29. @MargrietGr Weather forecast for Bristol https://developer.ibm.com/ clouddataservices/2016/10/06/ your-own-weather-forecast-in-a- python-notebook/ Visualize Data Python packages • pandas - rolling mean • matplotlib • Basemap Demo
- 30. @MargrietGr Weather map https:// developer.ibm.com/ clouddataservices/ 2016/10/06/your-own- weather-forecast-in-a- python-notebook/ Python packages • matplotlib • Basemap • itertools • urllib
- 31. @MargrietGr
- 32. @MargrietGr
- 34. @MargrietGr Weather, Twitter and Sentiment • Where to find the data? • Where to store the data? • Where to analyse the data? • Quick tools to explore
- 36. @MargrietGr Add sentiment - example
- 37. @MargrietGr • watson tone analyser Emotion Language style Social propensities Analyze how you are coming across to others
- 38. @MargrietGr Workflow Weather Company Data crontab -e 0 23 * * * /path/to/file/do_something.sh python do_something.py Tweets Weather Sentiment Watson Tone Analyser Insights for Twitter Cloudant NoSQL
- 39. @MargrietGr PixieDust https://github.com/ibm-cds-labs/pixiedust Simpler Workflow
- 40. @MargrietGr PixieDust: an Open Source Library that simplifies and improves Jupyter Python Notebooks • PackageManager • Visualizations • Cloud Integration • Scala Bridge • Extensibility • Embedded Apps https://developer.ibm.com/clouddataservices/2016/10/11/pixiedust-magic-for- python-notebook/ @DTAIEB55
- 41. @MargrietGr Install Spark packages or plain jars in your Notebook Python kernel without the need to modify configuration file Uses the GraphFrame Python APIs Install GraphFrames Spark Package
- 42. @MargrietGr One simple API: display() Call the Options dialog Panning/Zooming options Performance statistics
- 43. @MargrietGr Easily export your data to csv, json, html, etc. locally on your laptop or into a cloud-based service like Cloudant or Object Storage
- 44. @MargrietGr Scala Bridge Define a Python variable Use the Python var in Scala Define a Scala variable Use the Scala var in Python
- 45. @MargrietGr Easily extend PixieDust to create your own visualizations using HTML/CSS/JavaScript Customized Visualization for GraphFrame Graphs
- 46. @MargrietGr Encapsulate your analytics into compelling User Interfaces better suited for Line of Business Users
- 47. @MargrietGr
- 48. @MargrietGr IBM Watson Data Platform • Data Science Experience • Watson Data Platform • Machine Learning • Sign up for beta: http://datascience.ibm.com/features#machinelearning
- 49. @MargrietGr
- 50. @MargrietGr https://developer.ibm.com/clouddataservices/author/ mgroenen/ Thanks! Slides will be here: http://www.slideshare.net/MargrietGroenendijk