Continuous Delivery of Deep Transformer-Based NLP Models Using MLflow and AWS SageMaker for Enterprise AI Scenarios
1 like570 views
The document discusses the continuous delivery of NLP models using MLflow and AWS SageMaker within the context of enterprise AI implementation challenges. It identifies specific issues like the dev-prod divide, arbitrary uniqueness in model deployment, and challenges in provenance, while outlining a full machine learning lifecycle at Outreach. The presentation concludes with a successful deployment case of an intent classification model and future steps to enhance model development through feedback loops and active learning.
Continuous Delivery of Deep Transformer-Based NLP Models Using MLflow and AWS SageMaker for Enterprise AI Scenarios
- 2. Continuous Delivery of Deep Transformer-based NLP Models Using MLflow and AWS Sagemaker for Enterprise AI Scenarios Yong Liu Principal Data Scientist Outreach Corporation Andrew Brooks Senior Data Scientist Outreach Corporation
- 3. Presentation Outline ➢ Introduction and Background ➢ Challenges in Enterprise AI Implementation ➢ Full LifeCycle ML Experience at Outreach ➢ Conclusion and Future Work
- 6. Sales Engagement Platform (SEP) ▪ SEP encodes and automates sales activities into workflows ▪ Enables reps to perform one-on-one personalized outreach to up to 10x Day 1 Phone Call Day 1 Email Day 3 LinkedIn Day 5 Phone Day 5 Email
- 7. ML/NLP/AI Roles in Enterprise Sales Scenarios ▪ Continuous learning from data ( emails, phone calls, engagement logs etc.) ▪ Reasoning from knowledge to create a flywheel for the continual success of Reps
- 8. Challenges in Enterprise AI Implementation
- 9. Implementation Challenges: the Digital Divide Outline Dev-Prod Divide Dev-Prod Differences Challenge 2Challenge 1 Arbitrary Uniqueness Challenge 3 Provenance Challenge 4
- 10. Challenge 1: Dev-Prod Divide ▪ can’t test on “live” data ▪ can’t verify model invoked correctly ▪ can’t reproduce bugs or issues reported by users ▪ can’t reuse prod code for model development Isolated prod environment Source: Winderresearch
- 11. Challenge 2: Dev-Prod Differences Dev-Prod ▪ training data != prod data ▪ production scoring requires logic not used during model training When training & prod pipelines are and need to be different Source: MoneyUser.com
- 12. when the “whole” is not greater than the “sum of its parts”. Challenge 3: Arbitrary Uniqueness ▪ deploying each model feels like a “special case” ▪ gates and deploy mechanisms are ad hoc ▪ pipeline maintenance is costly Source: Rowperfect UK
- 13. Challenge 4: Provenance ▪ don’t know what exactly is running in prod ▪ inability to repro and debug model issues reported by customers ▪ model/pipeline changes = 😬 ▪ undocumented model/code changes compromise metric drifting. ▪ model experiments wasted Provenance from models to source code and data. Source: Slane Cartoons
- 14. Full-life Cycle ML Implementation Experience at Outreach
- 15. A Use Case: Guided Engagement powered by an intent classification model ▪ ML model predicts the intent of prospect’s email reply and then recommends the right template to respond.
- 16. Six Stages of ML Full Life Cycle
- 17. Six Stages of ML Full Life Cycle
- 18. Model Development and Offline Experimentation MLFlow tracking server to log all offline experiments Tracking experiments
- 19. Creating a transformer flavor model A new MLflow model flavor (transformer) & TransformerClassifier (sklearn pipeline) Transformer MLflow model Flavor code TransformerClassifier
- 20. Saving and Loading Transformer Artifacts An example of a fully saved and reloadable MLflow “Transformer”-flavor Model Load: mlflow.pyfunc.load_model(model_uri) Save: mlflow_transformer.log_model( transformer_classifier=trained_model, artifact_path="transformer_classifier", conda_env=CONDA_ENV, )
- 21. Productionizing Code and Git Repos ▪ MLProject Conda.yml IDE Dev Environment, MLflow MLProject, Github repo structure, flake8
- 22. Flexible Execution Mode MLProject allows using code and execution environments either locally or remotely (1)mlflow run ./ -e train (1)mlflow run git+ssh://git@github.com/model-repo -e train --version 1.1 (1)mlflow run ./ -e train.py --backend databricks --backend-config gpu_cluster_type.json (1)mlflow run git+ssh://git@github.com/model-repo -e train.py -- version 1.1 --backend databricks --backend-config gpu_cluster_type.json
- 23. Models: trained, wrapped, private-wheeled To support deployment specific logic and environment, we create three progressively evolved models for final deployment in a host (Sagemaker) Fine-tuned Trained Transformer Classifier Pre-score filter Post-score filter Wrapped sklearn pipeline model Private wheeled model No need to access github
- 24. Continuous Integration through CircleCI
- 25. Continuous Delivery/Rollback through Concourse Gated by two human gates: 1) start the full model deployment 2)promote the model from staging to production; and one regression test gate: accuracy must not be lower than previous version
- 26. Model Registry to Track Deployed Model Provenance
- 27. How Well Did We Do? Dev/Prod Divide Dev/Prod Differences Arbitrary Uniqueness Provenance + + _
- 28. Conclusions and Future Work
- 29. Conclusions and Future Work ▪ We highlight four typical enterprise AI implementation challenges and how we solve them with MLflow, Sagemaker and CICD tools ▪ Our intent classification model has been deployed in production and in operation using this framework ▪ Next steps: Incorporating model in-production feedback loop into annotation and model dev cycle We are further improving the annotation pipeline to have seamless human-in-the-loop active learning and model validation
- 30. Acknowledgements Data Science Group | Outreach.io Contact: yong.liu@outreach.io andrew.brooks@outreach.io