Velocity2013 mobile ci_intuit
1 like267 views
Tina Su discusses how Intuit implemented continuous integration and mobile test automation to speed up their mobile development cycle. They created an Intuit Virtual Device Lab (VDL) that gives developers browser-based access to real mobile devices. This allows automated tests to be run simultaneously across many device configurations. Intuit also developed a shared test library and uses Cucumber and Calabash for behavior-driven testing on Android and iOS. Continuous integration with the VDL and test automation reduced Intuit's iteration cycle from days to minutes, improving release quality and developer productivity.
Velocity2013 mobile ci_intuit
- 1. Speed to Market in Mobile Development Finding the right solution with continuous integration on real devices Tina Su, Director of Development, Intuit
- 2. About me Nearly 20 years of industry experience in R&D, quality and engineering management, with a strong passion for automation. § Engineering Director at Intuit, Developer Productivity § Sr. Engineering Director at Yahoo, Mobile Platforms § Sr. Engineering Manager at Sun, Java Standard and Mobile Editions § Compiler Developer at MIPS and Tandem Tina Su, Intuit
- 3. Agenda § Mobile trends & developers’ challenges § Speed & quality through continuous integration - Intuit Virtual Device Lab - Intuit mobile test automation § Takeaways
- 4. Aggressive mobile momentum “2013 Internet Trends,” KPCB Will surpass desktop by 2014 It should just work! Proliferation of platforms, OS versions & device types
- 5. Developer challenges § Immense configuration matrix - OS platforms (Android, iOS, Windows, phone vs. tablet, …) - Inputs (touch/gesture, real buttons, voice, camera, GPS receiver) - Outputs (portrait/landscape, screen resolution, language) § Application delivery through the markets - App signing, provision (app & device), submission § Offline experience - Network, different network bandwidth/latency, no-network - HTML5 offline app § Media challenges - Some codecs don’t work on a device - Streaming, sound § Security & privacy
- 6. Intuit mobile app portfolio today Apple Google Amazon SMS 0 5 10 15 20 490 22 15 11 3 491 mobile apps for financial institutions
- 7. Ramping up Intuit’s innovation engine Pain: Takes days per build to test combinations of device/OS configurations physically & manually, one by one … QA can’t keep up with the changes Goal: Reduce development iteration cycle from days to minutes on an ever-growing number of devices Strategy: Speed & quality through mobile continuous integration • iPhone 4 • iPhone 4S • iPhone 5 • iPad 2 • iPad 3 • iPad Mini Devices OS versions • iOS 5.0 • iOS 5.1 • iOS 6.0 • iOS 6.1.x • iOS 7.0 Builds X X Daily builds • Apple • Google • Amazon Platforms X
- 8. Continuous integration (CI) What is CI? § Automate build, test & deployment process § Integrate & build early, often, daily, for every check-in § Best practice of Agile development § Jenkins is widely adopted, flexible & extensible Benefits of CI in a mobile world § Instant feedback on quality, functionality & system impact § Less complex, no integration hell § Detect and address device configuration issues early § Involve stakeholders early § Ultimately achieve shippable release any time
- 9. Speed and quality through mobile CI Key enablers: • Mobile device lab • Test automation Test processorReal devices Jenkins SCM: source repository Developers/ IDE Automatic test run Test results Submit the app to app store if passed Check in Check-in triggers automated CI Install the test app to real devices in parallel Run tests on real devices in parallel Analyze test results Trigger testing in mobile device lab Process starts here
- 10. 3rd-party vendor solutions didn’t work for us § Couldn’t handle E2E automation of testing pre-release apps - Dependency on internal resources & infrastructure § Vendor lock-in: customized CI & test automation solutions - No flexibility to use our existing tools & frameworks § Disadvantages of vendors’ shared devices - Security concern on testing on shared devices - No access to corporate network - Cost-prohibitive to reserve devices § Private cloud (on-site setup) with local devices not practical - With sensitive customer data, we need private cloud - Extensive use of devices for daily testing of hundreds of mobile apps is expensive
- 11. Why test on real devices? Most mobile features can be tested on simulators, but simulators have limitations: § Device capabilities (GPS, camera, accelerometer, gyros), form factors & pixel density § Real-world experience over the real connection (3G, 4G, crappy Wi-Fi, etc.) § True performance, security tests § Testing on non-default Web browsers (e.g., Opera Mini) § Not all simulator/emulators have good quality (don’t reflect real rendering/behavior, bugs) § Testing SMS & app integration
- 12. Intuit Virtual Device Lab
- 13. Intuit Virtual Device Lab (VDL) Gives access to real mobile devices using a browser Intranet VDL User machine User machine User machine 3-click access Always on Integrated with dev infrastructure Globally available iPhone iPad Galaxy Captivate Optimus Nexus One Desire Incredible Manufacturers/ devices: Droid Milestone Moto
- 14. VDL architecture – Android Windows PC Tomcat Java image transferring application Browser Java applet Internet/ intranet Device interaction app Developer’s laptop VDL Image transfer USB connection
- 15. VDL architecture – iOS Mac with Xcode installed Tomcat Java VNC client Port forwarding Browser Java applet Internet/ intranet VNC server USB connection Developer’s laptop VDL
- 16. Open source tools technologies used in VDL Key capabilities Android iOS Installing mobile applications into the real device adb install APK file Fruitstrap https://github.com/ghughes/ fruitstrap Transferring device screen images, controlling device Screencast tool http://code.google.com/p/ androidscreencast Veency VNC app Fetching device local files adb pull filename sftp command (jailbroken devices only)
- 17. VDL key features functionality today Control the device through standard browser/ RESTful APIs Change orientation to landscape or portrait Copy local file to the device from your PC Get file from the device onto your local PC Execute an ADB command on the device See current running logs in your browser Download history logs to your local PC
- 18. Continuous integration – how it works Product builds Trigger file launches test VDL device inventoryResource manager Select a device Test execution engine Results Monitor for new builds
- 19. Mobile continuous integration Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously. Submit, build, test … in minutes vs. hours Select devices tests1 Tests run automatically across selected devices with each build 2 See the results!3
- 20. Next step: over-the-air replaces USB connection Fast onboarding Scalable Any device Internet/ intranet App installation Test execution Results retrieval VDL
- 22. Mobile test automation tools Instrumentation-based Non-instrumentation AndroidiOS MonkeyRunner MOET Sikuli Android SDK Robotium TestDroid (BitBar) Calabash (LessPainful) Soasta iOS SDK UIAutomation UISpec Frank MonkeyTalk DA/ PerfectoMobile eggPlant Cross-platform solution iOS Driver Appium • Assessed as of 2012
- 23. Mobile test automation trends § Abstraction with domain-specific language (DSL) - Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET) - Support behavior-driven development (BDD) § Client/server-based approach (remotely driven tests via HTTP) § Instrumentation non-instrumentation solutions complement each other § Mobile cloud testing environment - From hardware (cradle) to software approach to drive tests on real devices § Automated OTA app installation tests
- 24. Intuit automation framework of choice/recommendation § Native apps: - Cucumber + Calabash for BDD on iOS Android § Mobile Web: - Selenium/WebDriver
- 25. What is Cucumber? It lets stakeholders (customers, QE or developers) describe how software should behave in plain text. § BDD: Written in business domain-specific language (DSL) - Good abstraction over different device types § One format for multiple purposes - UI feature specification - Automated tests § Cucumber itself is NOT a test automation tool - Add-on to any test automation framework - Examples: Calabash, Sikuli Frank mobile test frameworks
- 26. What is Calabash? § Open source cross-platform mobile test framework - iOS + Android, simulator + real device, native + hybrid § Supports Cucumber for BDD § Application needs instrumentation that embeds a Calabash HTTP server - Tests run from a remote client (in a Jenkins slave) - Tests can be driven via Wi-Fi (no USB wire) § Ruby clients in API level (Java clients in experiment) § Works better in touch simulation real device testing § Mobile cloud test execution available by LessPainful
- 27. Intuit shared test library – rapid test development Internal open source repository Teams focus on creating Use shared test lib … and then contribute to common test library Tests Contribute Tests Common test library Product tests TestsTests Globalization test framework (I18N/L10N) Performance/load test framework Security test framework Native/hybrid test framework High-level test domain-specific language (Cucumber steps) Mobile back-end Mobile front-end
- 28. Code build Deploy Test Results Connect device Notify email Get devices Consolidate report Upload report From: manual 50*5 = 250 15 30 5 10 20 15 30 10 = 6 hours Code build Trigger Execute tests in parallel Consolidated report in archive Deploy on devices 0 5 10 0 = 15 mins To: automated CI TIME SAVED: 95%Release quality increased * Stunning results: reduced iteration cycle from hours to minutes Tested on 5 devices
- 29. VDL cost saving over vendors’ private clouds (today) Annual cost ($) # of devices Intuit Virtual Device Lab 150 3rd-party vendor 1 local device in cradle Assumptions: 50 native apps daily check-in Note: As vendor pricing changes, so do the savings Savings 25 3rd-party vendor 2 local device w/o cradle (Software approach) Savings Private cloud setup behind corp firewall
- 30. Key takeaways Building custom solution vs. 3rd-party vendor solution § Understand your technical/platform/global spread § Many vendor solutions provide a larger set of devices/OS versions in the global market § Quite a few offer mobile test services – might be tightly coupled with their device access solution automation framework § Connecting to your corporate network is a challenge – some provide private cloud (on-site setup within your network) but it’s cost-prohibitive Invest in automation and CI § Automate tests with robust automation framework § Enable rapid test development through shared test library § Adopt CI in mobile development for speed quality
- 31. More mobile innovations at Intuit § Mobile crash reporter § Mobile components design patterns § Mobile analytics SDK gateway Visit booth #601 – Accelerate Development § Demos conversations § Contest § Mobile device charging station § Foosball § We’re hiring ! Win a weekend with a TESLA!