Bringing AI to Production - An Introduction
- 1. Bringing AI to Production Ben Fishman Best Practices benfish22 benf22.github.io
- 2. >100$ Billion Thousands of companies >20k papers Yearly investment in AI: ©Ben Fishman || benf22.github.io
- 3. >100$ Billion And still… 80% - 90% of the AI projects FAIL Thousands of companies >20k papers Yearly investment in AI: ©Ben Fishman || benf22.github.io
- 4. How to Improve? Why? ©Ben Fishman || benf22.github.io
- 5. Ben Fishman Education • Ph.D. Candidate – Computer Science • M.Sc. Electrical Engineering • B.Sc. Bio Medical Engineering Industry • Director of AI & Algo @ Microsoft • Senior Data Scientist @ Microsoft • Team Lead @ Mobileye Background • Computer Vision • Audio • Speech • ML/ DL/ GenAI ©Ben Fishman || benf22.github.io
- 6. The Main Reasons for Failure ©Ben Fishman || benf22.github.io
- 7. The Main Reasons for Failure AI doesn’t fit the Core Problem ©Ben Fishman || benf22.github.io
- 8. The Main Reasons for Failure AI doesn’t fit the Core Problem Wrong Trajectory ©Ben Fishman || benf22.github.io
- 9. The Main Reasons for Failure AI doesn’t fit the Core Problem Wrong Trajectory Inefficient Development Process ©Ben Fishman || benf22.github.io
- 10. The Main Reasons for Failure AI doesn’t fit the Core Problem Wrong Trajectory Inefficient Development Process The Gap between POC to Production ©Ben Fishman || benf22.github.io
- 11. The Main Reasons for Failure AI doesn’t fit the Core Problem Wrong Trajectory Inefficient Development Process The Gap between POC to Production Cross Domain Teams ➔Different Development Process ➔ Lack of Trust in AI Product System Backend Frontend ©Ben Fishman || benf22.github.io
- 12. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 13. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 14. Application Oriented Algo ©Ben Fishman || benf22.github.io
- 15. Application Oriented Algo Run E2E (sooner rather than later) • Better understanding the app • Simple (off-the shelf) → Complex ©Ben Fishman || benf22.github.io
- 16. Application Oriented Algo Run E2E (sooner rather than later) • Better understanding the app • Simple (off-the shelf) → Complex Be Relevant • Test (only) what you need • Fix (only) what you need App: Detecting pedestrians crossing the road ©Ben Fishman || benf22.github.io
- 17. Application Oriented Algo Run E2E (sooner rather than later) • Better understanding the app • Simple (off-the shelf) → Complex Be Relevant • Test (only) what you need • Fix (only) what you need Interesting Not Interesting App: Detecting pedestrians crossing the road ©Ben Fishman || benf22.github.io
- 18. Application Oriented Algo Run E2E (sooner rather than later) • Better understanding the app • Simple (off-the shelf) → Complex Be Relevant • Test (only) what you need • Fix (only) what you need Align with other stake holders • Product • System Not Interesting App: Detecting pedestrians crossing the road ©Ben Fishman || benf22.github.io Interesting
- 19. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 20. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 21. Data Evaluation Invest in Data & Evaluation ©Ben Fishman || benf22.github.io
- 22. Data Evaluation ❖ Companies tend to underestimate their importance Invest in Data & Evaluation ©Ben Fishman || benf22.github.io
- 23. Data Evaluation ❖ Companies tend to underestimate their importance ❖ The compass & accelerator of the development process Invest in Data & Evaluation ©Ben Fishman || benf22.github.io
- 24. Invest in Data & Evaluation Evaluation ©Ben Fishman || benf22.github.io
- 25. Invest in Data & Evaluation Evaluation Engine Accurate Reliable Easy to use Fast Focuses you Raises problems fast Provides confidence Evaluation ©Ben Fishman || benf22.github.io
- 26. Invest in Data & Evaluation Evaluation Engine Accurate Reliable Easy to use Fast Focuses you Raises problems fast Provides confidence Metrics Raw Metrics Not enough Refined Metrics Fits your application Evaluation ©Ben Fishman || benf22.github.io
- 27. Invest in Data & Evaluation Evaluation Engine Accurate Reliable Easy to use Fast Focuses you Raises problems fast Provides confidence Metrics Raw Metrics Not enough Refined Metrics Fits your application Human in The Loop Evaluation ©Ben Fishman || benf22.github.io
- 28. Invest in Data & Evaluation Data ©Ben Fishman || benf22.github.io
- 29. Data Collection Invest in Data & Evaluation Data Data Annotation Data Analysis Data Engineering Data Roles Datasets Creation Legal Privacy Responsible AI Curation Auto Labeling ©Ben Fishman || benf22.github.io
- 30. Data Collection Invest in Data & Evaluation Data Roles Data Manager Data Collectors Data Annotators Data Engineers Data Data Annotation Data Analysis Data Engineering Data Roles Datasets Creation Data Analyst Legal Privacy Responsible AI Curation Auto Labeling ©Ben Fishman || benf22.github.io
- 31. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 32. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 33. Make it a Factory ©Ben Fishman || benf22.github.io
- 34. Make it a Factory Problem: Many technical challenges ➔ Inefficient work + Focus distraction ©Ben Fishman || benf22.github.io
- 35. Make it a Factory Problem: Many technical challenges ➔ Inefficient work + Focus distraction Solution: Solve them ©Ben Fishman || benf22.github.io
- 36. Make it a Factory Problem: Many technical challenges ➔ Inefficient work + Focus distraction Solution: Solve them It’s not appealing and not deliverable – BUT it is crucial for success ©Ben Fishman || benf22.github.io
- 37. Make it a Factory Infra/ MLOps Commonality Teamwork Problem: Many technical challenges ➔ Inefficient work + Focus distraction Solution: Solve them It’s not appealing and not deliverable – BUT it is crucial for success ©Ben Fishman || benf22.github.io
- 38. Make it a Factory Infra/ MLOps Data Pipelines Parallelization Training Acceleration Commonality Teamwork Problem: Many technical challenges ➔ Inefficient work + Focus distraction Solution: Solve them It’s not appealing and not deliverable – BUT it is crucial for success ©Ben Fishman || benf22.github.io
- 39. Make it a Factory Infra/ MLOps Code Tools Environment Data Pipelines Parallelization Training Acceleration Commonality Teamwork Problem: Many technical challenges ➔ Inefficient work + Focus distraction Solution: Solve them It’s not appealing and not deliverable – BUT it is crucial for success ©Ben Fishman || benf22.github.io
- 40. Make it a Factory Infra/ MLOps Code Tools Environment Methods Terminology Conventions Data Pipelines Parallelization Training Acceleration Commonality Teamwork Problem: Many technical challenges ➔ Inefficient work + Focus distraction Solution: Solve them It’s not appealing and not deliverable – BUT it is crucial for success ©Ben Fishman || benf22.github.io
- 41. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 42. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 43. Divide & Conquer Divide large problems into smaller problems ➔ Easier to solve them ©Ben Fishman || benf22.github.io
- 44. Divide & Conquer Divide large problems into smaller problems ➔ Easier to solve them Time Modules People ©Ben Fishman || benf22.github.io
- 45. Divide & Conquer Time Modules People ©Ben Fishman || benf22.github.io
- 46. Divide & Conquer Split your timeline into smaller segments: • To improve in small incremental steps • To name & control your status • To be able to reproduce • To show progress Time Modules People ©Ben Fishman || benf22.github.io
- 47. Divide & Conquer Split your timeline into smaller segments: • To improve in small incremental steps • To name & control your status • To be able to reproduce • To show progress Time Modules People Version everything Code Version Algo Version Data Version Dataset Version Metrics Version ©Ben Fishman || benf22.github.io
- 48. Divide & Conquer Time Modules People ©Ben Fishman || benf22.github.io
- 49. Divide & Conquer Algo Modules Modularity rather than E2E models Time Modules People Dev Modules Separate module for each functionality (Data preparation, training …) ©Ben Fishman || benf22.github.io
- 50. Divide & Conquer Algo Modules Modularity rather than E2E models Time Modules People Interpretability Debuggability Reproducability Dev Modules Separate module for each functionality (Data preparation, training …) Better Control Faster Development Smaller tasks are simpler ©Ben Fishman || benf22.github.io
- 51. Divide & Conquer Algo Modules Modularity rather than E2E models Time Modules People Interpretability Debuggability Reproducability Modulate more than you already have Dev Modules Separate module for each functionality (Data preparation, training …) Better Control Faster Development Smaller tasks are simpler ©Ben Fishman || benf22.github.io
- 52. Divide & Conquer Time Modules People ©Ben Fishman || benf22.github.io
- 53. Divide & Conquer Time Modules People Split the work between team members Separate roles in the team ©Ben Fishman || benf22.github.io
- 54. Divide & Conquer Application oriented algo Make it a factory Invest in Data & Evaluation Don’t be naïve The Principals ©Ben Fishman || benf22.github.io
- 55. Be in touch Bring Them Home ©Ben Fishman || benf22.github.io