AI and Machine Learning for Testers
- 1. W6 Special Topics 5/10/17 11:30 AI and Machine Learning for Testers Presented by: Jason Arbon Appdiff, Inc. Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ info@techwell.com -‐ http://www.starwest.techwell.com/
- 2. Jason Arbon Jason Arbon is the CEO of Appdiff, which is redefining how enterprises develop, test, and ship mobile apps with zero code and zero setup required. He was formerly the director of engineering and product at Applause.com/uTest.com, where he led product strategy to deliver crowdsourced testing via more than 250,000 community members and created the app store data analytics service. Jason previously held engineering leadership roles at Google and Microsoft and coauthored How Google Tests Software and App Quality: Secrets for Agile App Teams.
- 3. AI and Machine Learning for Testers Jason Arbon, CEO @Appdiff
- 5. Ai for Test Automation 3 Relevant Context Testing Neural Net Ranker Personalized Web Search and Chrome Test Automation AI for Mobile Test Automation
- 6. Ai for Test Automation Agenda AI For Testing Testing AI Future
- 7. Appdiff Presentation5 The Real Data Scientists Joanne Tseng Data Scientist @appdiff.com Appdiff mission: Transform app development with automation & insights Francis Iannacci Lead Data Scientist @appdiff.com
- 8. AI for App Testing Features Complexity increases exponentially as new features and states interact with existing features Tests Test coverage grows linearly because tests can only be added one at a time Time Complexity/Coverage COVERAGE GAP 6 Testing Needs AI
- 10. When will AI Start Testing?
- 11. When: Today
- 12. Our Story10 AI for Testing
- 13. What We Do11 ML : Subjective or complex labels via Humans
- 14. Our Story12 AI for Testing: Input Reduce input space 150 legit actions per page 10 clicks deep 2x10^65 paths Train like a human/tester
- 15. Our Story13 AI for Testing
- 16. Our Story14 AI for Testing
- 17. Our Story15 AI Driven Test Flows
- 18. Our Story16 AI Driven Test Flows
- 19. Our Story17 AI Driven Test Flows
- 21. Coverage
- 22. Our Story20 AI bots can test almost any app
- 24. Our Story22 Automation Coverage: Bots 2/3rds Existing Tests Long Sequence of Dependant Actions and Verifications Basic Tasks (Login, Search, Create Account, Add items to Card, etc.) Specific Sequences of Events with Specific Input (search for ‘beanie babies’, etc.
- 25. Our Story23 Automation Coverage: Bots soon 100%
- 26. Our Story24 Automation Coverage: Bots... 10X Canonical Tests Learning Tests Cross-app
- 27. portfolio
- 28. Our Story26 Appdiff Represents the Evolution of Software Quality 1 AD HOC TESTING Reactively test 2 MANUAL TESTING Proactively test 3 TEST AUTOMATION Automate repetition 4 AI-DRIVEN APPROACH Accelerate coverage App QA App Automation QA App
- 29. How to test AI?
- 30. Our Story28 Testing AI: Action / Flow Graphs
- 31. Our Story29 Testing AI: App Generation and Traversal
- 32. Our Story30 Testing AI: Graph Generation and Traversal
- 33. What We Do31 Testing AI: Drift Networks are randomly initialized Same overall correctness score can mean diff performance on subsets of data
- 34. What is Next?
- 35. What We Do33 DARPA HARD
- 36. What We Do34 Humans vs Bots Humans Train Bots in Course of Work Humans Remain to Personalize and Re-train bots.
- 37. What We Do35 Testing Flow with AI
- 38. What We Do36 AI Test Description Language: AIT Focus on Intent, not “how” Human Readable App-Independant
- 39. Our Story37 AI for Testing
- 40. What We Do Features Complexity increases exponentially as new features and states interact with existing features Tests Test coverage grows linearly because tests can only be added one at a time Time Complexity/Coverage COVERAGE GAP 38 Testing Needs AI
- 43. Our Story41 Teaching AI How To Test
- 44. What We Do42 ML: Page Label Training
- 45. As long as there is software, there will be software testing Humans Code
- 46. Our Story44 AI for Testing
- 47. Appendix: How It Works45 Like a Redline for Your App Agile development leaves teams struggling to achieve adequate test coverage. Automatically identify changes to UX across versions of your app. Version X Version Y Version X Version Y 1.2s 1.8s Slower AddedRemoved Performance UX
- 48. Testing with AI?