Evan Giordanella: The Testing of Machine Learning and Artificial Intelligence Algorithms
- 1. THE TESTING OF MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE ALGORITHMS Evan R. Giordanella
- 2. QUICK INTRODUCTION • Python Engineer • Tester • QA Engineer • Manager • Quality Expert • Mentor • Coach • Musician • Cyclist Who am I?
- 3. THE TEAM • Where • Comcast Advanced Advertising • What we do • Intelligent ad sales, planning, and execution • The Team • Build e2e testing systems: simulate system behavior • Experts on big/small picture code (many languages, technologies, environments, etc)
- 4. MY FIRST MACHINE LEARNING TESTING EXPERIENCE
- 5. NEW PRODUCT - NEW TEAM • New product to “intelligently” deliver ads • Machine learning techniques were implemented to predict how many people will be watching an ad at any given moment • New data science team for new product • New test engineers for new product
- 6. NEW PRODUCT - NEW TEAM • New product to “intelligently” deliver ads • Machine learning techniques were implemented to predict how many people will be watching an ad at any given moment • New data science team for new product • New test engineers for new product My task was to assess whether this new product intelligently delivered ads.
- 7. CHALLENGES • Inexperience with ML applications • Stochastic output • Computationally intense validation • Communication barriers with the data science team • Statistics, math!? • Data scientists’ confidence in testing • Buy-in from product organization re: Machine Learning • Buy-in from industry stakeholders (slow moving local cable businesses) • New testing can be intimidating
- 8. WHAT IS MACHINE LEARNING? • Field of Computer Science • An application of artificial intelligence (AI) that analyzes data without the computer being explicitly programmed to do so
- 9. WHAT IS MACHINE LEARNING? • Field of Computer Science • An application of artificial intelligence (AI) that analyzes data without the computer being explicitly programmed to do so
- 10. SKILLS NEEDED FOR ML TESTING There are a lot of them!
- 11. STATISTICS / MATH • All testers needed to have a foundational knowledge of data analytics • Characterization the data • mean, standard deviation, variance • Performance Metrics • mean square error (MSE), classification accuracy, loss, area under the curve (AUC)
- 12. TERMINOLOGY • We had to learn basic Machine Learning terminology • Testing and Training • Regression and Classification • Validation • Features • Dimensions • Random Forest
- 13. TERMINOLOGY • We had to learn basic Machine Learning terminology • Testing and Training • Regression and Classification • Validation • Features • Dimensions • Random Forest Speaking the same language is important!
- 14. TOOLS AND METHODS WE USED some worked and some didn’t…
- 15. TOOLS •We wanted to use same tools as Data Science Team •What tools do computational scientists commonly use? Python, Pandas, SciPy, NumPy, Docker
- 16. VISUALIZATION THEN • How do we assess the application? • Visualization! • 1st strategy was to plot model performance over time using excel • Visualization very helpful • Worked only in the beginning • This did not scale! Who would have thought! Ha!
- 17. VISUALIZATION NOW • 2nd strategy was to automate test and output visualization • Created tools using Docker and D3 to graph results of various models using various testing data sets
- 18. HOW • Looked at how models were implemented in order to find ways to exploit them • Examine the code • Demonstrated how models would behave given common data problems • missing values • anomalies • small vs large data • real vs synthetic data • Creates predictable data sets with injected noise in an attempt to break the algorithm How we devised our tests
- 19. LESSONS LEARNED • Need to speak the same language with those that implement the ML/AI • Conflicting technical nomenclature can create unnecessary slow downs • Need to interpret and present the results in a way that stakeholders and PMs can digest • Visualize when possible • Know your audience! • Knowing what fears your stakeholders have about ML can help you build their confidence in it • Do not be afraid!
- 20. THANK YOU • Questions? • evang@visibleworld.com