Lessons Learned from Testing Machine Learning Software
Download as PPT, PDF0 likes238 views
1) Machine learning software is difficult to test compared to traditional software because it is monolithic rather than modular, so changing one part affects the whole system. 2) When testing machine learning models, you need to understand what the models have been taught from the training data, including potential issues like spurious correlations, rather than just checking inputs and outputs. 3) Testing machine learning software effectively requires a good mathematical foundation as well as an understanding of different machine learning techniques and how to implement them.
![Traditional software
programming• How we do testing, basically:
– Inputs a=[X1,X2,…,Xn]
– Expect results b=[Y1,Y2,…,Yn]
– We use assertions to validate the specification
– f(Xi)==Yi](https://image.slidesharecdn.com/eurostar-christianramirez-161214041112/85/Lessons-Learned-from-Testing-Machine-Learning-Software-9-320.jpg)
Lessons Learned from Testing Machine Learning Software
- 1. Lessons Learned from Testing Machine Learning SoftwareChristian Ramírez @chrix2 @formiik
- 2. About me • Computer Engineer • M.Sc. (Astronomy) • Ex-googler • Python lover • Actually formiiker (principal researcher at formiik and formiiklabs)
- 3. About formiik and how we use MLFormiik is a platform designed to improve productivity of on-site staff. – Route optimization – Workload balancing – Ranking – Time series anomaly detection – Image recognition (deep learning)
- 4. Lesson 0 • Corollary number 2 by @dx • “In theory, there is no difference between theory and practice. But, in practice, there is” Yogi Bera
- 5. Introduction • In machine learning, computers apply statistical learning techniques to automatically identify patterns in data. • A core objective of a learner is to generalize from its experience.
- 6. •WHAT?
- 7. Lesson 1 • Forget all you know about testing
- 8. Traditional software programming• In a general way, we do this: • With a specification of a function » f(x) – We implement the function to meet the specification » f(x)=y
- 9. Traditional software programming• How we do testing, basically: – Inputs a=[X1,X2,…,Xn] – Expect results b=[Y1,Y2,…,Yn] – We use assertions to validate the specification – f(Xi)==Yi
- 10. Machine learning software programming • In the most of cases • We give examples – Pairs (Xi,Yi) – Induce f() such that • y≈f(x) for given pairs, and generalizes well for unseen x
- 11. Machine learning software programming • In other cases • We give examples – Input (Xi) – No Yi are given to the learning algorithm, leaving it on its own to find structure in its input
- 12. Testing (ML) software • Traditional software is modular – We can decompose it and understand it – Each module has inputs and outputs that can be defined and isolated
- 13. Testing (ML) software • Machine learning systems appear to be monolithic – Everything depends on everything else – Changing any one thing changes everything else
- 14. Lesson 2 • Learn about machine learning – Kinds of learning – Coding
- 15. Lesson 3 • The models will learn what you teach them to learn. – Test models no requirements – Adversarial examples – Algorithms don't align with reality – Spurious correlations • Corollary number 4 “human beings are so fools to programming” by @dx
- 16. Lesson 4 • No matter what you have read on blogs, you need a good level of mathematical knowledge • Remember again corollary number 2
- 17. Lesson 5 • Build your own ecosystem – Frameworks – GPU or CPU?
- 18. Lesson 6 • Identifies how much data you really need – More data or better models?
- 19. Lesson 7 • You need a new toolset – Experimental data scientist
- 20. Lesson 8 • Be a story teller – Have a goal – Manage the expectative, this is not magic
- 21. Summary
Editor's Notes
- #3: Soy ingeniero en computacion, egresado de la universidad nacional de mexico Tengo una maestria en ciencias con especialidad en astronomia, es en este punto donde yo conoci el machine learning y el bigdata, claro que esa epoca no se llamaban asi Fui tester en google, donde floreciomi pasion por las pruebas de software Actualmente soy principal researcher at formiik and formiiklabs
- #4: Formiik es el producto lider en america latina para la optimizacion de trabajo en ca
- #5: No importa cuanto hayas leido sobre machine learning, cuantos posts hayas comentado en kdnuggets o cuantos tweets de kirk borne hayas compartido. No haces testing de machine learning hasta que lo haces en la vida real Hay mucha investigacion sobre como aplicar machine learning en testing, pero hay muy poco sobre como hacer testing sobre aplicaciones de machine learning
- #6: Esta es un definicion de machine learning, que vemos en blogs, revistas y demas lugares No dice mucho al respecto
- #8: Lo primero que debes hacer es quitarte todas las ideas que tienes sobre “testing tradicional” No porque esten mal, pero ya no son aplicables como tal en este contexto Se que esto puede sonar pretencioso pero es la verdad
- #10: Si, ya se que eso no es testing is checking y validaciones Espero que Michael bolton no este en esta sala
- #14: In the machine learning world, we can identify mistakes. We can also retrain a network from scratch to try to fix a mistake. However, we do not know how to make small local changes, fixing a small bug without changing everything at once, which is the everyday practice that defines bugfixing in traditional programming. In machine learning, fixing a bug means restarting and rebuilding the whole system.
- #15: Debes conocer, los tipos de aprendizaje, en que casos se usan, que algoritmos existen Aprender a programar, es indispensable Las pruebas, las validaciones, la generacion y limpieza de datos deben ser codificadas Recuerda terminator, enviaron a una maquina para acabar con otra maquina, es la unica manera