CFSummit: Data Science on Cloud Foundry
0 likes819 views
Ian Huston and Alexander Kagoshima are data scientists at Pivotal Labs who use Cloud Foundry to deploy data science projects. They discuss using Cloud Foundry to easily control incoming data and bind external data services. They also provide examples of building prediction APIs and deploying machine learning models on Cloud Foundry to power data-driven applications.
CFSummit: Data Science on Cloud Foundry
- 2. Data Science on Cloud Foundry Ian Huston @ianhuston Alexander Kagoshima @akagoshima
- 3. Who are we? • Data Scientists at Pivotal Labs • Using Cloud Foundry since 2013 • Working with enterprises to get value out of their data
- 4. Image by Drew Conway: http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram
- 5. Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician. - Josh Wills
- 8. Data Services Easy control of incoming data
- 9. Data Services Bind and scale system services – Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME Add User Provided Services – Standalone Hadoop or Apache Spark cluster, Big Data System $ cf cups SERVICE_INSTANCE -‐p "host, port, username, password" Data Service App App App App App App
- 10. Deploy a Model Prediction API Control distributed computation
- 11. h"ps://github.com/ihuston/python-‐conda-‐buildpack Install PyData packages with binary builds using conda
- 12. h"ps://github.com/alexkago/cf-‐buildpack-‐r R interpreter and package setup, ready for RShiny
- 13. Siloed Data Siloed Systems Distributed Big Data Platform HOW TO DEPLOY MODELS? Data Extract ? (Model development happens here!) (Business needs model predictions here!)
- 14. App App App App App Big Data Platform Big Data Storage
- 15. R E S T A P I Send data as JSON Data Ingest Model Create Model Redis Kicking off periodic retraining Save training data Save model object Send JSON data without label Receive prediction from trained model instance Deployed at: http://dsoncf.cfapps.io Code: https://github.com/pivotalsoftware/ds-cfpylearning PREDICTION API ARCHITECTURE $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME
- 16. MODEL INTERFACE
- 18. SIMPLE HTML + JS MODEL PREDICTIONS http://ds-demo-transport.cfapps.io
- 20. Show off your data science related Cloud Foundry apps: Twitter: @dsoncf http://dsoncf.com
- 22. R E S T A P I Send data as JSON Data Ingest Model Create Model Redis Kicking off periodic retraining Save training data Save model object Send JSON data without label Receive prediction from trained model instance Deployed at: http://dsoncf.cfapps.io Code: https://github.com/pivotalsoftware/ds-cfpylearning Visualization PREDICTION API ARCHITECTURE
- 23. Data Services Bind and scale system services – Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME Add User Provided Services – Standalone Hadoop or Apache Spark cluster, Big Data System $ cf cups SERVICE_INSTANCE -‐p "host, port, username, password"