Mobile Automation: Lessons From The Trenches
1 like579 views
This document discusses challenges and lessons related to automating mobile application testing. It notes that mobile testing faces issues like device fragmentation, difficulty simulating real-world conditions, and maintaining a device lab. The key lessons are to prioritize test cases, automate functional tests on simulators first, then automate critical cases on real devices. Non-mobile elements like data and environments should also be automated. Choosing the right tools, using cloud options, and simulating conditions can help address challenges of eating the "elephant" of full mobile automation.
Mobile Automation: Lessons From The Trenches
- 1. ©2015 InfoStretch Corporation. All rights reserved. Dr. Ashok Karania | January 21, 2015 Mobile Automation Lesson from the Trenches
- 2. ©2015 InfoStretch Corporation. All rights reserved. Agenda § Mobile Landscape & Tes1ng Challenges § Automa1on Challenges § Lessons From The Trenches § Case Study § Q & A
- 3. ©2015 InfoStretch Corporation. All rights reserved. § Apps Tested § Employees 100+ § Apps developed and deployed 3000+ Global Footprint A leading provider of next-gen mobile application lifecycle services ranging from design and development to testing and sustenance. About us Locations Corporate HQ: Silicon Valley Offices: Conshohocken (PA), Ahmedabad (India), Pune (India), London (UK) 600+ 250K § Active Mobile Customers Engagements 60+ About Infostretch § Real devices in mobile labs
- 4. ©2015 InfoStretch Corporation. All rights reserved. Mobile Landscape & Testing Challenges
- 5. ©2015 InfoStretch Corporation. All rights reserved. 25% 1.7B 95%Of adults are online with 3+ devices Forrester 2013 Number of smartphones and tablets shipped worldwide will more than double by 2017 IDC 2014 Of enterprises allow BYOD in some form FierceMobileIT2014 31.7%The mobile applications market is expected to increase 31.7% by 2017 IDC 2014
- 6. ©2015 InfoStretch Corporation. All rights reserved. THE FUTURE IS MOBILE WEARABLES GLASS FLEXIBLE SMARTPHONESVIRTUAL
- 7. ©2015 InfoStretch Corporation. All rights reserved. AND BEYOND INTERNET OF THINGS BIONICS AND ROBOTICS COGNITIVE COMPUTING 3D PRINTERS TRANSPARENT LCDs CONNECTED CARS
- 8. ©2015 InfoStretch Corporation. All rights reserved. YOU NEED A NEW VISION, NOT A NEW VERSION CONNECTED AGILE COMMUNITY NEXT GEN ENTERPRISE
- 9. ©2015 InfoStretch Corporation. All rights reserved. Mobile Automation Is Imperative § Mobile Testing Is Everything We Do On Web & More & More § Enables To Meet Compressed Release and Development Cycles § Address Fragmentation of Devices, Platforms and Carriers § Generate Real World Testing Conditions § Improve Overall Testing Quality § Overall Time & Resource Efficiencies § Systematic
- 10. ©2015 InfoStretch Corporation. All rights reserved. Automation Challenges from the Trenches
- 11. ©2015 InfoStretch Corporation. All rights reserved. 1. Device Fragmentation ~12K Android Devices
- 12. ©2015 InfoStretch Corporation. All rights reserved. 2. Why Real Devices Are A Nightmare Fragile Slow Maintenance Issues Cost Other App Inter-Operability !
- 13. ©2015 InfoStretch Corporation. All rights reserved. 3. Simulating Real Life Conditions The Real Mobile End User Experience Can Be Different § Servers Will Be Loaded § Devices Will Behave Differently § Network Will React Differently Apps Behave Differently In Special Conditions § CPU Usage § RAM § Network Conditions § Other Apps § Environment conditions § Light Conditions We Need Real Life Testing Conditions!
- 14. ©2015 InfoStretch Corporation. All rights reserved. 4. Mobile Device Lab Is A Big Challenge § Test Lab Creation § Test Lab Maintenance § Team § Buy Or Cloud § Globally Available § Management § Contracts § OS Upgrades § Procurement Policy § Performance § Security
- 15. ©2015 InfoStretch Corporation. All rights reserved. Lessons from the Trenches
- 16. ©2015 InfoStretch Corporation. All rights reserved. The best way to deal with mobile device automa4on challenge is to not automate every thing on the mobile devices ” “
- 17. ©2015 InfoStretch Corporation. All rights reserved. How To Eat An Elephant § Priori1ze Business Needs & Analyze Test Cases § Design An Automa1on Framework § Select Right Automa1on Tool License VS OS, Integra1ons, Exis1ng Investment, Mobile Friendly, Team Experience § Automate All Func1onal Test Cases On Simulators Mobile Web Can Be Simulated By Tes1ng On Real Browsers Using User Agent Strings Na1ve Can Be Tested By Using Simulators § Automate Device Centric & Cri1cal Func1onal Test Cases On Real Devices § Iden1fy & Integrate Non-‐Mobile Elements Of Automa1on Data Condi1oning, SeRngs Pre-‐Requisites, Environment SeRngs
- 18. ©2015 InfoStretch Corporation. All rights reserved. NATURE OF TEST CASES Func1onal Non-‐ Func1onal Security Performance TYPE OF APP Business App Social Networking Gaming App U1lity App VERIFICATION CRITERIA Business Rules Look & Feel Response Time Content TARGET USERS Enterprises Consumer Field Engineers Test Case Analysis 1.
- 19. ©2015 InfoStretch Corporation. All rights reserved. Meeting Device Fragmentation - Device Type of the App Games Social Business User Personas Business Teen Social Geography Asia Pac Americas etc. App Functions Streaming Screen Res GPS etc. Device Popularity Form Factor OS Output: Device/OS/ Test Matrix
- 20. ©2015 InfoStretch Corporation. All rights reserved. iOS Android HTC Aria Android HTC Droid BB Torch LG CU620 LG VX9000 Device Implementations for Sign On (10+) BB 8520 Devices Browser Open Browser Go To Sign On Make Payment Confirm Payment Sign OffText Verify Make Payment Test caseQTP/Selenium Address ATM Search Browser Open Select Acct View Schedule Payment Sync Text Quick Links Nick Name Browser Go To Sign Off ZIP ATM Search Make Payment Sync Image Confirm Payment Framework Sign On Component Library 2. Mobile Test Automation Framework
- 21. ©2015 InfoStretch Corporation. All rights reserved. Automation Tools Tool Supported Pla.orms Basis of Automa5on Scrip5ng Language Instrumenta5o n Required Repor5ng Format Cost A Object Access All Major No HTML, Junit, CSV Free P Both Na5ve, Visual Basic, Java Either HTML, PDF, Na5ve Paid K Object Access Objec1ve-‐C Yes XCode Free C Object Access Ruby, C#, JVM Yes HTML Free MR Object Access Python Yes Junit Free R Object Access Java Yes Junit Free MT Both JS, Na1ve Yes HTML/Junit Paid X Both C#, Ruby No HTML Paid T GUI N.A Yes Various Paid E GUI Na1ve No Na1ve Paid
- 22. ©2015 InfoStretch Corporation. All rights reserved. Cloud Options • Real Devices – Perfecto • Simulators – Sauce Labs In-House Mobile Labs Crowd Sourcing Meeting The Device Lab Challenge
- 23. ©2015 InfoStretch Corporation. All rights reserved. APPROACH QUALITY COST EFFECTIVENESS SECURITY PRIVACY ACCOUNTABILITY TIME TO MARKET DEVICE COVERAGE GLOBAL TESTING Simulator/Emulator 3 5 3 4 4 4 3 5 Real Mobile Devices 4 2 5 5 5 2 5 4 Remote Mobile Access 4 4 4 4 4 5 4 4 Crowd sourcing 2 3 2 1 1 3 4 4 Managed Testing Services 5 3 5 5 5 4 5 4 Guide: Least Effective 0 Most Effective 5 Comparative Evaluation Of Device Testing Options
- 24. ©2015 InfoStretch Corporation. All rights reserved. Create Real Life Conditions Manage CPU Usage Manage Memory Manage Network BandwidthCreate & Manage Interruptions Manage Location Of DeviceManage OrientationTurn WIFI On or Off
- 25. ©2015 InfoStretch Corporation. All rights reserved. Summary: How To Eat An Elephant One Bit At A Time § Prioritize Business Needs & Analyze Test Cases § Design An Automation Framework § Select Right Automation Tool § Automate All Functional Test Cases On Simulators § Automate Device Centric & Critical Functional Test Cases On Real Devices § Identify & Integrate Non-Mobile Elements Of Automation
- 26. ©2015 InfoStretch Corporation. All rights reserved. Continue The Dialogue… Email: ashok.karania@infostretch.com Linkedin: ashokkarania Blog: blog.infostretch.com Blog: www.ashokkarania.com Visit ATOS Booth Today & Tomorrow!
- 27. ©2015 InfoStretch Corporation. All rights reserved. CONNECT WITH US: Thank You Tel: 408.727.1100 Email: info@infostretch.com URL: www.infostretch.com