Scale your Testing and Quality with Automation Engineering and ML - Carlos Kidman
Download as PPTX, PDF0 likes109 views
The document outlines a comprehensive approach to testing and ensuring the quality of machine learning systems, emphasizing the importance of defining problems, assessing risks, and designing effective architectures. It advocates for continuous testing from model training to post-deployment, addressing challenges like data quality, adversarial attacks, and fairness. Additionally, it highlights the need for comprehensive behavior testing and iterative improvements to enhance machine learning models and their applications.
Scale your Testing and Quality with Automation Engineering and ML - Carlos Kidman
- 1. Testing & Quality of ML Systems @CarlosKidman
- 2. SLIDESMANIA.COM ● Head of Engineering ● Test Automation University ● Open Source (ie Pylenium, PyClinic) ● International Keynote Speaker ● Twitch and YouTube ● Founder of QAP
- 3. “ Test early. Test often. Tulsee Doshi and Jacqueline Pan at Google I/O 2019 Conference. 3
- 4. PREDICTION @JayAlammar Predicting (estimating, calculating) values based on patterns in other, existing values.
- 5. Predict how much customers would spend How much would 3 people spend? Photo by Greta Hoffman from Number of people Purchase Amount 1 10 2 20 4 40 10 🎉
- 7. Photo by Greta Hoffman from Number of people Purchase Amount 1 10 2 20 4 40 10 🎉 Dataset Weight Features Labels
- 8. Shift Left Quality starts at the beginning @CarlosKidman
- 9. 1. Define the Problem ╸ What are we trying to solve? ╸ Who is the ML System for? ╸ Does this even need ML?
- 10. 2. Define Success and Assess Risk ╸ Define initial baselines ╸ Consider Privacy and Security risks ╸ Create a Proof of Concept Do we have the proper resources to do this?
- 11. 3. Design the Initial Architecture ╸ Which DB or tables will we pull the data from? ╸ What does our ELT Process look like? ╸ REST Service or Mobile Device? ╸ How will we monitor and measure our model?
- 12. 4. Collect Data ╸ Where is the data coming from? ╸ Is this different than STAGE or DEV? ╸ Where are we storing the data? ╸ Is the data streamed and/or batched? Understand the Data Journey
- 14. 5. Prepare Data ╸ What shapes does the data need to be in? ╸ What are the data types? ╸ Are there missing values or errors? 💩 data = 💩 models
- 15. 6. Train and Validate Models ╸ Experiment and compare multiple models ╸ Capture training and validation metrics ╸ Visualize results for further analysis ╸ Tune weights and parameters ╸ Use tests (exploratory and automated)
- 16. VALIDATIO N How well does the model perform (using metrics like accuracy) against a dataset it has NOT seen? TESTING TRAINING VALIDATION 95% 76% ***
- 17. 7. Test the Models ╸ What behaviors does the model show? ╸ Does it demonstrate harmful biases? ╸ Does it meet Privacy and Security Requirements? ╸ Is it performant and reliable? ╸ Can it withstand things like Adversarial Attacks? ╸ Does it solve the problem(s) we set out to solve? ╸ Do our customers enjoy using our ML System?
- 18. 8. Deploy the Model ╸ REST or GraphQL Service? ╸ Mobile Device? ╸ Airplane or Self-Driving Car? ╸ Serverless Function?
- 19. 9. Observe and Iterate ╸ Monitor ╸ Measure ╸ Alert ╸ Insights ╸ Learn
- 23. Lessons Learned ╸ Poor data quality ╸ Easy to game the system ╸ Too dependent on the ML System ╸ Bias from Selective Focus ╸ Risk from External Factors ╸ Good at first, worse over time Source: https://medium.com/geekculture/what-can-ds-ml-engineers-learn-from-zillows-flippinggate-b820a1a5c8ef
- 24. Shift Right Quality continues post-deployment @CarlosKidman
- 26. MLOps requires good Processes, Testing, and Automation @CarlosKidman
- 27. Adversarial Attacks Fooling and exploiting ML Models @CarlosKidman
- 28. Attacks can cause harm 😨 Sources: https://www.youtube.com/watch?v=YXy6oX1iNoA https://arstechnica.com/cars/2017/09/hacking-street-signs-with-stickers-could-confuse-self-driving-cars/
- 29. Invisible to the Security AI Source: https://www.theverge.com/2019/4/23/18512472/fool-ai-surveillance-adversarial-example-yolov2-person-detection
- 30. Become the Attacker ╸ Not only for Computer Vision ╸ You don’t have to be a Data Scientist ╸ You don’t have to know how to code ╸ Be a part of Threat Modeling ╸ Be creative and design tests and attacks
- 31. Behavioral Testing Beyond accuracy & loss metrics @CarlosKidman
- 32. Test behaviors and capabilities ╸ Define desired capabilities during design ╸ Testing behaviors gives a more accurate picture of model performance ╸ “...created twice as many tests and found almost three times as many bugs” Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (Ribeiro et al., 2020) https://aclanthology.org/2020.acl-main.442.pdf
- 33. What does it look like? 🤔 Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (Ribeiro et al., 2020) https://aclanthology.org/2020.acl-main.442.pdf
- 34. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (Ribeiro et al., 2020) https://aclanthology.org/2020.acl-main.442.pdf
- 35. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (Ribeiro et al., 2020) https://aclanthology.org/2020.acl-main.442.pdf
- 36. Fair and Responsible AI Test for Harmful Biases @CarlosKidman
- 38. Fairness ● By Data ● By Measurement & Modeling ● By Design
- 40. Source: Google I/O 2019 Machine Learning Fairness: Lessons Learned https://youtu.be/6CwzDoE8J4M
- 41. Source: Google I/O 2019 Machine Learning Fairness: Lessons Learned https://youtu.be/6CwzDoE8J4M
- 42. Source: Google I/O 2019 Machine Learning Fairness: Lessons Learned https://youtu.be/6CwzDoE8J4M
- 43. Recap ╸ Shift Left ╸ Shift Right ╸ Adversarial Attacks ╸ Test Behaviors and the overall experience ╸ Test for Fairness and Harmful Biases
- 44. We need holistic testing and quality ╸ Test early and test often ╸ Apply current testing and quality techniques and strategies ╸ We need better testing!
Editor's Notes
- #34: NER = Named Entity Recognition