A Data-Driven Approach for Mobile Testing and Automation

T2
Session - Mobile Testing
4/27/17 10:00 AM

A Data-Driven Approach for Mobile
Testing and Automation

Presented by:

Satyajit Malugu
GoDaddy

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - https://www.techwell.com/

Satyajit Malugu
GoDaddy

A senior SDET at GoDaddy, Satyajit Malugu is focused on mobile testing. His work
involves automation of native and hybrid apps, and providing test perspective to
his team that recently converted from waterfall to agile. As a testing leader in a
company that is deploying a suite of native apps, Satyajit is involved with
strategizing, executing, and mentoring other testers and stakeholders on best
practices for native mobile testing. Previously, Satyajit worked at Urbanspoon
and gained SDET knowledge at Microsoft. He frequently presents on mobile
topics. Read his blog and follow him on Twitter at @malugu.

5/2/17
1
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Utilizing data for mobile testing
and automation
by Satyajit Malugu
Sr.SDET at Godaddy
04/27/2017
About me
•  SDET for 10 years, focused on mobile apps
•  Primary tester for mobile apps at
•  Urbanspoon
•  Godaddy domain finder/ legacy apps/ Investor apps
•  Godaddy Smartline

5/2/17
2
Challenges for quality decisions
•  Multiple stakeholders and their viewpoints
•  Product
•  UX
•  Marketing
•  Developers
•  Executives (high visibility)
•  Multiple sources of bug reports
•  Internal
•  External
•  Project deadlines
These are common for any software project
3
Additional challenges for mobile apps
•  UI Automation/build system flakiness
•  Test builds distribution troubles (iOS)
•  Testflight
•  Certificates
•  App store deployment wonkiness
•  Multiple entry points into app
•  Push notifications
•  Widgets
•  We develop app for an OS, not for an app(browser) on the OS
•  Memory management
•  Battery etc.
4

5/2/17
3
What is a data point
• An artifact that provides information that is:
• Quantifiable
• Measurable
• Actionable
Why is it important
•  Data helps in making objective decisions
•  Respect for bug reports
•  As the solo critics on the team, data gives you additional support
•  Reduce the impact of biases that are ever-present
•  Availability(now) bias

5/2/17
4
Data sources
7
1 Historic
3 During development
2 After deployment
4 External
Historic
8

5/2/17
5
Historic
•  From other apps within the company
•  Previous versions
•  Device matrix
•  Bug trends
After deployment
10

5/2/17
6
Crash monitoring
Customer feedback
•  App store reviews
•  Slack bots
•  Summaries
•  Customer support
•  Other feedback channels
•  Emails
•  Within apps

5/2/17
7
Usage monitoring
•  Fabric, GA, Google play analytics
•  API access logs
•  Instrumentations
Flurry flowsFabric analytics
Adoption of versions
Force upgrade strategy

5/2/17
8
During development
15
Network proxies
•  Able to intercept ‘all’ traffic to and from the apps
•  Charles/Fiddler
•  SSL proxying
•  Look/edit network requests
•  Rewrite rules

5/2/17
9
Database/network inspection
•  Stetho (android only)
17
Loggers/Videos
•  Android - logcat
•  iOS - console
•  Vysor – Android videos/screenshare/screenshots
•  Quicktime- iOS videos
•  Great info for logging bugs

5/2/17
10
Monitors
19
•  Android Monitor
•  Xcode Instruments
CI pipeline
•  Build jobs
•  Test results
•  Identify regressions
•  Find intermittent crashes

5/2/17
11
External
21
External
•  Android official dashboards - https://developer.android.com/about/dashboards/index.html
•  iOS official stats - https://developer.apple.com/support/app-store/
•  Per country usage from perfecto -
http://info.perfectomobile.com/digital-test-coverage-index.html?utm_source=tci-blog
•  Top mobile devices
•  Most used apps across categories
•  Consolidated release calendar
•  Mixpanel - https://mixpanel.com/trends/

5/2/17
12
Usage scenarios
23
Device lab planning and budgets
24
•  Problem:
•  How to plan out device labs
•  What and when to acquire devices
•  Data sources:
•  Historic
•  Internal company device matrix
•  External
•  Android/iOS dashboards
•  Release calendar

5/2/17
13
CI/CD pipeline
Lint
checks
Unit tests
Espresso
activity
tests
Espresso
smoke
tests
Manual
code
review
Merge to
main
branch
Make decisions automatic
•  Problem: Fast feedback during
development
•  Static analyzers
•  Unit tests
•  UI Tests
Checklists for releases
•  Problem: Are we ready to release?
•  Bugs – priority, trends, open
•  Test results – automation, manual
•  Code coverage
•  Solution: Jira/confluence integration, confluence dashboard

5/2/17
14
Push notification stability
27
•  Problem: How reliable are the push notification delivery
•  Adhoc/ logging
•  Solution:
•  Collecting and analyzing in a huge excel
Automation Stabilization
28
•  Problem: Finding flaky tests
•  Build jobs
•  Solution:
•  Test results analyzer
•  Slack integration
•  Problem: Is this new test flaky
•  Build jobs
•  Test runs
•  Solution:
•  Run tests ’n’ times before promoting

5/2/17
15
Quality dashboards
•  Problem: One pager status report
of quality
•  Data souces:
•  Builds jobs
•  Errors
•  Bugs
•  Solutions:
New Relic
Effective bug reports for devs
•  Problem: Devs not paying enough attention to bug reports, multiple to-fro conversations
•  Loggers with stack traces
•  Network request/responses
•  Screenshots/videos
•  Database state, memory usage etc.
•  Solution: Along with steps to reproduce include other artifacts
30

5/2/17
16
Effective bug reports for product owners
•  Problem: Product folks not paying enough attention to bug reports deprioritizing bugs
•  Customer feedback (reviews, emails)
•  Device usage matrix
•  Crash reports
•  Solution: Along with steps to reproduce include data to help evaluate the impact and risks
31
Takeaways
•  Multiple data points and use cases given
•  Analyze your project and come up with a data plan for your project
•  Completely depends on your project, company, culture
•  What worked for me won't necessarily work for you

5/2/17
17
Thank you
•  Share your experiences
•  Comments
•  Questions
33
http://mobiletest.engineer
@malugu

A Data-Driven Approach for Mobile Testing and Automation

More Related Content

What's hot (20)

Similar to A Data-Driven Approach for Mobile Testing and Automation (20)

More from TechWell (20)

Recently uploaded (20)

A Data-Driven Approach for Mobile Testing and Automation