The More the Merrier: Scaling Model Building Infrastructure at Zendesk
0 likes1,396 views
The document outlines the evolution and challenges faced by Zendesk in building machine learning models for customer satisfaction prediction, including their experiences using Hadoop and the introduction of deep learning models such as the Answer Bot. Key lessons learned emphasize the importance of scalability, dynamic resource allocation, and automatic model validation. The text details the technological infrastructure and processes adopted to improve model building efficiency and manage large volumes of training data.
The More the Merrier: Scaling Model Building Infrastructure at Zendesk
- 1. Lessons Learnt Scaling Model Building @ Zendesk The More the Merrier
- 2. Wai Chee Yau Staff Software Engineer Pepper
- 3. Model Building @ Zendesk How we built machine learning models initially What challenges we faced How we evolved the model building infrastructure What lessons we learnt
- 4. Scaling Can be Painful 4
- 5. A Few Models
- 7. Slide types and sample layouts What is Zendesk?
- 8. Zendesk ML Products Journey Customer Satisfaction Prediction 2016 3000 models per week 2017 20192018
- 10. Building Satisfaction Prediction Models in Hadoop Hadoop HDFS Training data (Tickets) Training data (Tickets) Training data (Tickets) Training data (Tickets) Map Reduce Jobs ModelsModelsModelsModels 400 models per job run
- 11. Hadoop Pains 11
- 12. Slide types and sample layouts Second ML Product Answer Bot
- 13. Second ML Product - Answer Bot Customer Satisfaction Prediction Answer Bot 2016 3000 models per week 1 deep learning model a few months 2017 2018 2019
- 14. Answerbot - Zendesk’s First Deep Learning Adventure
- 15. Embedding: Turning words into numbers [ 7.25020394e-02, -1.19434139e-02, 2.35390533e-02, 9.40115377e-03, 8.13035890e-02, 6.50805384e-02, -4.03035507e-02, -6.47375807e-02, 2.81035509e-02, -1.87401652e-01, 1.12001531e-01, -2.67665803e-01, 6.60590157e-02, 2.46239230e-02, -3.72320563e-02, 3.12019400e-02, -7.69012272e-02, -1.70350112e-02, -4.82226498e-02, -8.59876275e-02, 4.28824723e-02, -9.28599089e-02, -6.01094738e-02, -8.52334574e-02, 8.72100666e-02, 1.91824064e-01, 1.05177149e-01, -1.12113327e-01, -1.71761960e-01, -1.66820228e-01, -1.36309946e-02, -3.36700417e-02, 3.18476819e-02, -1.26342744e-01, -8.29755142e-03, 8.12109783e-02, -1.25934565e-02, 1.49573416e-01, 2.69240364e-02, “We are all prisoners of our phones, thus they are called cell phones!” Embed!
- 16. Embedding Space, where no one has gone before Men with heels They is not necessarily plural Short haired women Tenderloin is the best part of SF Smell of weed
- 17. Embedding Space, where no one has gone before Ticket
- 18. Slide types and sample layouts Third ML Product Content Cues
- 19. Third ML Product - Content Cues Customer Satisfaction Prediction Content Cues (early access) Answer Bot 2016 3000 models per week 1 deep learning model a few months Up to 50000 models per day 2017 2018 2019
- 20. Content Cues - Summarizing Tickets into Topics
- 21. Content Cues - Summarizing Tickets into Topics
- 22. Scaling Challenges with Content Cues Satisfaction Prediction 3k+ models weekly Content Cues Up to 50k models daily
- 23. Slide types and sample layouts Generate training data for Content Cues
- 24. Feature Generation with Spark MySQLMySQLMySQLMySQLMySQL Kafka Data Lake (S3) Tickets (snapshot) ML Features (S3) EMR Spark Filter and Partition by Account Training Features (account 1) Training Features (account 1) Training Features (account 1) Training Features (account 2) Training Features (account 2) Training Features (account 2)
- 25. Slide types and sample layouts Group Tickets into Topics
- 26. How to Group Tickets into Topics? Add frequent flyer number Change flight Cancel Flight
- 27. Ticket Summarisation Process Embed Text Cluster Generate Titles + Keywords Topic 1 Topic 2 Topic n Input Ticket Cluster snapshots
- 28. How to present the UI to the user? Challenge
- 29. Content Cues - Summarizing Tickets into Topics
- 30. t-SNE Plot of Ticket Clusters
- 32. How to scale model building? Challenge
- 33. Generic Solution for Offline Batch Model Building ML Features (S3) Training Features (account 1) Training Features (account 1) Training Features ML Models (S3) Training Features (account 1) Training Features (account 1) Models Scalable Compute To Build Models
- 34. Requirements for Model Building ● Scalable and elastic ● Support building batches of models on a recurrent basis ● Support on demand building of models ● Flexibility to use CPU or GPU instances
- 36. AWS Batch for Building Models AWS Batch Job Queue (low) Job Queue (medium) Job Queue (high) Compute Environment (spot) Model Build JobModel Build Job Compute Environment (on demand) Model Build JobModel Build Job
- 37. Building Models with AWS Batch AWS Batch Job Queues Job Queues Compute Environments Model Build JobModel Build Job Model Serving Service Submit Job ML Features (S3) Training Features (account 1) Training Features (account 1) Training Features ML Models (S3) SNS + SQS Models Models Airflow
- 38. Select Suitable Instance Types For Jobs Lesson Learnt
- 39. How to deal with different data distribution across accounts? Challenge
- 40. Distribution of account and ticket count NumberofAccounts Number of Tickets
- 42. Allocate Resources Based on Job Size Small Medium Large Container resources: vCPU, memory vCPU: 2 Memory: 2GB AWS Batch vCPU: 4 Memory: 5GB vCPU: 8 Memory: 8GB
- 44. 12xFaster to build 50k models Dynamic vs Static Resource Allocation
- 46. How to prove scalability? Challenge:
- 47. Elastic Compute With Job Queues Works Well Lesson Learnt
- 48. How to fix the 0.03% failed jobs? Challenge
- 49. Timing matters Build latest clusters Upload Clusters Input Ticket Input Ticket Input Ticket Input Ticket Previous Cluster snapshots ML Models (S3) Clusters info Publish clusters Upload Snapshot
- 50. Timing matters Build latest clusters Upload snapshots Upload clusters Input Ticket Input Ticket Input Ticket Input Ticket Previous Cluster snapshots ML Models (S3) Clusters info Publish clusters
- 51. Keep the ML Code Idempotent Lesson Learnt
- 52. Overcome out of memory errors Challenge
- 53. Try It Till You Make It AWS Batch SNS + SQS Model Building Service If job failed due to out of memory, resubmit with higher memory limit Trigger job Job status events
- 54. Be Prepared to Handle Outlier Memory Usage Lesson Learnt
- 55. How to validate ML models? Challenge
- 56. Model Performance Concerns Tickets in the topic not related to each other Incorrect grammar of title Topic title not related to tickets
- 57. Cluster Quality Checks Checks Overlap between : ● title and keywords ● title and tickets ● tickets and the keywords
- 58. Always Build in Automatic Model Validation Lesson Learnt
- 59. Balance model complexity vs system performance Lessons Learnt Select suitable instance types for jobs Elastic compute with job queues works well Dynamic resource allocation optimizes costs Keep the ML code idempotent Be prepared to handle outlier memory usage Always build in automatic model validation
- 60. TM and © 2018 Zendesk Inc. All rights reserved.