Smart App@Pivotal by Dat Tran
1 like501 views
This document discusses smart apps and how Pivotal uses data science to build them. It describes three key components of smart apps: data, a smart system that uses data science to understand user behavior, and a user interface. It then provides examples of smart apps Pivotal has developed for logistics and automotive customers, describing how machine learning models were used to predict delivery locations and road conditions. The document emphasizes an API-first approach and using cloud platforms like Cloud Foundry to operationalize models and deliver insights through predictive APIs.
Smart App@Pivotal by Dat Tran
- 1. Smart Apps @ Pivotal Korea Big Data Roadshow – 05/04/2017 Dat Tran (Senior Data Scientist)
- 2. 2 echo $(whoami) Senior Data Scientist @datitran
- 3. 3 That’s me! A digital native…
- 4. 4 My top three apps...
- 5. 5 What do these apps have in common? They are smart…
- 7. 7 Traffic Prediction and ETA
- 9. 9 What are Smart Apps? High value for users Less value for users High value through data Smart Apps Less value through data
- 10. 10 Why Smart Apps? Users expect an app to be easy to use and personalized * o An audience of one is the new normal o Our smart apps adapt to a customer’s personal likes and dislikes o By capturing the feedback, we continually improve * The Rise of the Data Natives by Monica Rogati
- 11. 11 Three Components of a Smart App Data of all sizes and formats including audio and video forms the backbone of a smart app Data A smart system uses data science to understand and predict user behavior Smart System User Interface + + Data Smart System User Interface ACTIONS An easy to use interface presents the results of the smart system to drive the desired action
- 12. 12 How we do data science @ Pivotal
- 13. 13 Data Science Product Management Product Design Engineering Continuous Improvement
- 14. 14 Radically Agile Data Science @ Pivotal Labs Pair Programming Retros Test Driven Development Continuous Integration / API First Tracker Standups
- 15. 15 API First Source: What is hardcore data science—in practice? (Mikio Braun)
- 16. 16 Live Demo
- 17. 17 Smart Data-Driven Apps Handwritten Digit Recognition Next Likely Location
- 18. 18 Use Cases
- 19. Largest Logistic Services in the Middle East
- 20. Delivery in a Land with No Post Codes Customer: One of the largest logistics services company based in the middle east Problem: No postcodes in middle east and hence the addresses were confirmed by making a phone call to every consignee Data Science Solution: Use machine learning models to predict the right location for delivery using historic data Business Impact: Potential saving the client ~ $2.3M by avoiding phone calls
- 21. Technology and Data Overview ~5TB data for ~57M customers
- 22. DS Workflow Diagram Using DBSCAN on GPS locations per consignee Consignees with one location cluster Consignees with multiple location clusters Use multinomial logistic regression to predict the most likely cluster location Already sure about the cluster location Built personalized delivery models each for ~57M customers
- 23. Impact Potentially saving ~ $2.3M No Phone Calls Call API to get delivery location App Integration
- 25. 25 Saving Human Lives with the IoT Customer: A traditional German automobile manufacturer specializing on sport cars Problem: Developed a new sensor which can detect road conditions well but with no predictive elements and operationalization is unclear Data Science Solution: Use machine learning models to predict certain road conditions based on weather data and other car data Challenges: No technical infrastructure and BDS not possible due to financial constraints
- 26. 26 Cloud Native Architecture Predictive API Redis for Pivotal CF Deep Learning Spring XD FTP Source Shell Processor Tap S3 Feature Engineering Dashboard Enricher Model Pipeline Orchestration EC2 S3Sinkforpersistentstorage Redis Sink Measured data Weather data λ-Architecture Modelpersistence JSON JSON CSVCSV CSV Real-Time Layer Batch Layer Models
- 27. 27 Spring Cloud Data Flow (Formerly Spring XD) Unified, distributed, and extensible open-source system for data ingestion, real time analytics, batch processing and data export o Data Ingestion and Pipeline Processing o Real Time Analytics o Rapid Dashboarding o Batch Workflow Orchestration + ETL Redis for Pivotal CFSpring XD FTP Source Shell Processor Tap S3 S3Sinkforpersistentstorage Redis Sink
- 28. 28 Batch Layer Deep Learning S3 Feature Engineering Model Pipeline Orchestration CSVCSV
- 29. 29 Real-Time Layer Predictive API Redis for Pivotal CF Dashboard Enricher Measured data Weather data JSON JSON CSV Real-Time Layer Models
- 30. 30 Pivotal Cloud Foundry PaaS more… http:// Push App> cf
- 31. Modeling
- 32. 32 Short Introduction into Deep Learning Input Layer Output Layer Hidden Layer 1 Hidden Layer 2 Positive Neutral Negative
- 33. 33 Types of Neural Network Recurrent Neural NetworkFeed-Forward Neural Network o Models dynamic temporal behaviors o Many variants: LSTM, GRU, Bi- directional RNNs etc. o Applications: Handwriting and speech recognition and many more o Ideal for functional mapping problems o Architectures: Multi-layer perceptron, CNNs etc. o Many applications in supervised learning
- 34. 34 Recurrent Neural Network Image classification Image captioning Sentiment analysis Machine translation Video classification Source: The Unreasonable Effectiveness of Recurrent Neural Networks (Andrej Karpathy)
- 35. 35 Key Learnings o Class imbalance problem o Use simple RNNs over LSTM o Use GPUs! o Time-consuming to find the optimal network o Many data is needed
- 36. 36 Key Takeaways
- 37. 37 Key Takeaways o API First: Bringing the models into production as fast as possible helps to minimize risk o Clients can test it and give early/regular feedback o Fast ROI o Cloud Foundry enables us to reliably expose models as scalable predictive APIs o Cloud Native Data Science is crucial for Smart Apps