Are You Missing Critical Mobile Tests?

JeanAnn Harrison &
XBOSoft
Are You Missing Critical
Mobile Tests?

XBOSoft info
• Founded in 2006
• Dedicated to software quality
• Software QA consulting
• Software testing services
• Offices in San Francisco, Beijing, Oslo and
Amsterdam

Housekeeping
• Everyone except the speaker is muted
• Questions via the gotowebinar control on the right side of your
screen
• Questions can be asked throughout the webinar, we’ll try to fit
them in when appropriate
• General Q & A at the end of the webinar
• You will receive info on recording after the webinar

Reminders
Make sure your drained device is available, as well as another
device to send email to the drained device and a charger.
When you see the heart “pumping iron” slides JeanAnn will
walk us through a test on your device.
Please type questions in the Chat box.
We are looking for 1-2 volunteers to take the microphone
during their device testing to let everyone know what is
happening.

About the Speaker
JeanAnn Harrison regularly speaks at
industry conferences and is published in
software testing magazines.
She has years of experience with mobile
testing and is a well-known figure in the
QA and software testing community.
Besides software testing , JeanAnn really
likes baseball & ice skating.

WEBINAR:
EXERCISES IN EXPLORATORY
TESTING
JEAN ANN HARRISON
Software Test Consultant for
Project Realms.
FOR MOBILE AND EMBEDDED TESTERS
Jean Ann Harrison Copyright 2013

AGENDA
• Introduction
• Note taking & use of slide information
• Quick overview of Mobile/Embedded
Software
• Types of Mobile Apps
• Exploratory Testing Definition
• Defining Skill Set for the Mobile Tester
• Mindmap getting started with Mobile
Testing
• Good Exploratory Testing Skills/Mobile
ET Skills
• Keys to Mobile Exploratory Testing
• Testing Beyond the GUI
• Learning the Architecture
• Organizing Tests/Stories
• Common tools for ET
• Some Test Case Ideas
• Exercises & Test Case Examples
• Summation & Conclusion
Not covered – User Experience, Usability,
Trainability, Web Application & Security Tests

MOBILE/EMBEDDED SOFTWARE &
DEVICES
• Mobile phones, tablets
• Proprietary devices: mobile heart monitors, law enforcement
ticket generators, restaurant personnel taking customer orders
on devices, GPS devices, PDAs
• Embedded Software examples: automobile’s computer system,
air traffic control systems, airplane navigation system, drug
infusion pumps in hospital rooms, elevators, cameras, robots

TYPES OF MOBILE
APPS
• Native Applications
• Local to device
• Hybrid Applications
• Local to device but
interacts w/internet
• Web Applications
• Not local to device.
All interactions on
internet

EXPLORATORY TESTING -
DEFINITION
• Quoting James Bach: “The plainest definition of exploratory
testing is test design and test execution at the same time. This is
the opposite of scripted testing (predefined test procedures,
whether manual or automated). Exploratory tests, unlike
scripted tests, are not defined in advance and carried out
precisely according to plan.”
http://www.satisfice.com/articles/what_is_et.shtml

DEFINING SKILL SET FOR
THE MOBILE/EMBEDDED
TESTER
• Some exposure or knowledge about products from the domain in which you
are testing: aerospace, medical, automobile manufacturing, airplanes,
factory systems, robotics, regulated environments, etc.
• Some knowledge of: hard sciences: math, physics, electronics, engineering,
etc for logical thought process.
• Software sciences: psychology, philosophy, sociology, human factors (human
machine interface) for creative & conceptual thought process
• These testers use skills from many knowledge domains, patterns of errors,
and basic testing skills to create tests. They are using their intuition, critical
thinking, and their mental models.
Chapter 1 – Software Test Attacks to Break Mobile & Embedded Devices

WHAT YOU NEED TO BE A
GOOD EXPLORATORY
TESTER
• Awareness
• Discipline
• Focus

KEYS TO MOBILE
EXPLORATORY TESTING
• Look beyond the GUI
• Timing
• Domain knowledge
• Knowledge of System
(hardware/firmware/software)
• Characteristic of bugs behavior, formulate
patterns
• Use life experiences for inspiration

TESTING BEYOND THE GUI =
SYSTEM INTEGRATION TESTING
SIT for Mobile/Embedded contains elements:
Software + Hardware + Software Variants + Hardware
Variants + Timing + Operating System Drivers +
Network Protocols

WHAT DO WE TEST?

LEARNING THE
ARCHITECTURE
Work closely with
development to learn more
about how the mobile
application under test works.

EXERCISE FOR GROUP
• In the question box, enter 1 test idea of what you
want to explore. The point of this exercise is you do
NOT know what the application is, so you can’t tell me
what kind of functional test to perform.
• I’m testing YOU to see what kinds of non-functional
tests you think might be good to explore.

COMMON TOOLS FOR ORGANIZATION
• Sticky notes per conditions, per variables, per test case idea
• Keep a log of what you are doing, record conditions
• Utilize the device’s log or find an open source logging tool
• Use tables to record all variables/conditions per test
• Document situation of the test, list out your test conditions

”WE BECOME MORE EXPLORATORY WHEN WE
CAN'T TELL WHAT TESTS SHOULD BE RUN, IN
ADVANCE OF THE TEST CYCLE, OR WHEN WE
HAVEN'T YET HAD THE OPPORTUNITY TO CREATE
THOSE TESTS.” JAMES BACH
• Create your description or theory. “What happens when…?”
• … battery runs low?
• … battery is charging from a dead battery?
• … when the device gets too hot?
• … when the battery is replaced due to defective battery?
• … device memory is full?
• … I receive a notification while using the phone or another application?
• … when there is a time/date change?
• … device searches are too slow?

DOCUMENTING YOUR TEST
CONDITIONS FOR THE CHARGING OF
BATTERY TEST

TEST CASE STEPS
1. Pre-select any application to
download onto device
2. Drain your device’s battery charge
completely
3. Notate time
4. Plug in charging device
5. Immediately power up device
6. Record times for first condition
where app is not installed, device
powered up
7. Drain your device’s battery charge
completely
8. Plug in charging device
9. Immediately power up device
10. Notate time
11. Start to download pre-selected app
12. Record times for each battery level
while downloading

EXERCISE: EXPLORATION FOR MORE
TESTS
• What other kinds of tests can you apply based on
the condition of starting to charge from a dead
battery?
• Record your tester stories

ORGANIZE YOUR
THOUGHTS/TESTS
Battery
level
Time/Temp Application
Installing
Application
Installed/not in
use/started
App in use (note –
various conditions
apply here)
0%-5% Time each level
increment of
charge
Timing of
download?
Would
notifications
factor?
Notifications?
Memory usage
CPU speed
6%-15% Check the temp
at each
increment
Completed
installation?
Note CPU speed,
memory usage
Searches with
application DB, online
DB?
16%-30% Interaction with data
transfer/network
communication
31%-50%

CONTROL SYSTEM CONSIDERATIONS
DEVELOPING TESTS:
HARDWARE TO SOFTWARE SIGNAL
INTERFACE
• Know the input and output connections to the software
• Consider differences in variations between devices like calibrations, physical noise,
electron, light, cold, water, dust device, response time, wear & tear on the hardware.
• Identify input & output devices with ranges and resolutions of values (how is the
software installed onto the device?)
• Define the full range of environmental input disturbances (unexpected inputs)
• Define possible environmental output disturbances (unexpected outputs)
• Determine what is or is not possible in the test lab
• Conduct a risk analysis of what is acceptable and what is not
• Understand device limitations: CPU processing capabilities, memory, time.
• The Aerospace Industry use the term, “Test like you fly, fly like you test” to combine all
facets of how a device’s software should be tested.
Originally written by Jon D Hagar Copyright 2013 Jean Ann Harrison Copyright 2013

• Download either Twitter or Facebook onto device
• Start either downloaded app
• From another device, send an email to the device’s email account
• Immediately send a tweet or post a status
• Continue to do engage Twitter or Facebook app for at least 1 minute
• Record email notification and time when sent and received.
• What other observations occurred?
EXERCISE: NOTIFICATION TEST

EXERCISE: EXPLORING MORE
FOR THE INTERRUPT TEST
• What other kinds of interrupts could be tested?
• What other tests would you consider if something
goes wrong?

TEST CASE EXAMPLE: INTERRUPTS ON MOBILE
PHONES
• Go to your App store and choose an application to download
• While the downloading is occurring, call the mobile phone
• Record observations with the download
• You may need to rely on observing a log file while implementing these tests
• If it fails, what kind of error recovery occurs?

LET’S TALK DATA AND STORAGE
• Mobile devices contain limited amount of storage space.
• Testing for mobile apps storing data locally on a mobile device means testing the
boundaries of the app’s design. How will developers handle the limitations?
• Work with your developers to understand those boundaries.
• If your app doesn’t have a boundary set, then the app can fill up your device’s
storage capabilities.
• Testing the limits might mean testing how the software polices that boundary by
having a specific limit before automatically downloading only so much information.
• Per usage, email programs only download so much data and most recent data to
the device.

TEST CASE EXAMPLE: TESTING
STORAGE LIMITS
• Download several large applications (ex: Angry Birds & Facebook are large apps)
• Go to your Settings and then to “Manage applications”.
• Review the amount of total storage free.
• Keep downloading apps until you reach below 5KB
• Then start an application where you would save data on the device.
• Check your free storage amount.
• Keep saving data until you reach a point where you receive an error message or a
“can’t save” message.
• Check the remaining free storage in the Settings again to see what the threshold is
for free space is maxed out.
• Delete the last data which was just added.
• Send data though the internet, but use the app under test (using the application
where you first started saving data) and not just email.
• What other tests can you do where storage is maxed out?

SUMMARIZATION OF TECHNIQUES
• Combining tests can help shorten project schedules
but what also happens is you can witness results.
Automation results do not give you all you need to
witness as it happens.
• Ask yourself “what if I … “ and consider tests not
related to functional behavior or conditions.
• Using tools, especially log files, can reveal memory
usage, getting statistics while conducting tests.
• Documenting includes use of video,
writing/development of tests in story formats, user
action formats

CONCLUSION
• Remember: In mobile and embedded just poking at the GUI will leave bugs
“hiding”
• Different tests by a variety of testers (developer and titled testers) is
commonly needed
• Context matters: there is no one set of tests, approaches, testers, or
attacks which will work all the time
• Limited resources on device can make testing challenging and requires an
entire team effort

JEAN ANN HARRISON’S
CONTACT
INFORMATION
• EMAIL: jaharrison@projectrealms.com
• Twitter: https://twitter.com/JA_Harrison
Or @JA_Harrison
• Project Realms Inc website:
www.projectrealms.com
• 8/10/2013, I will be facilitating a Weekend
Testing Session on Risk Based Testing for Mobile
Devices… details can be found through my
Twitter updates.

REFEREN
CES
• Software Test Attacks to Break Mobile & Embedded Devices by Jon D Hagar to be
published and released fall 2013.
• 2013 STPCon Presentation slides from “Testing Beyond the GUI” by Jon D Hagar &
Jean Ann Harrison
• http://www.ministryoftesting.com/2012/06/getting-started-with-mobile-testing-a-
mindmap/ Getting started with Mobile Testing Mindmap
• http://karennicolejohnson.com/ Karen Johnson’s website
• http://www.satisfice.com/articles/what_is_et.shtml by James Bach, Satisfice, Inc
• http://www.softwaretestpro.com/Item/5567/ A Different Take on Mobile Testing –
Test Beyond the GUI by Jean Ann Harrison

IMAGES
REFERENCES
• Slide 3 - Embedded Systems examples
• http://www.jijesoft.com/en/?option=com_content&view=article&id=18&Itemid=45
• http://www.mseedsystems.com/products/view/268/aldelo-for-restaurants-pos-wireless-edition
• http://blog.laptopmag.com/wpress/wp-content/uploads/2013/02/Pandora-Limit-Free-Listening.jpg
• slide 4 - types of Apps
• http://myeventapps.com/whats-the-difference-native-vs-web-apps/2012/794/
• Slide 5 – Exploratory Testing Definition
• http://www.testandtry.com/wp-content/uploads/explore.jpg
• slide 6 - Defining skill set
• http://cdn.cutestpaw.com/wp-content/uploads/2012/11/l-The-Thinker.jpg

IMAGES REFERENCES
• Slide 7 - mindmap used with permission from originators Karen Johnson &
James Bach
• www.ministryoftesting.com
• slide 8 - What you need to be a good Exploratory
• Testerhttp://www.wallcoo.net/paint/kagaya_celestial_exploring/Kagaya_art_Ce
lestial_Exploring_INSPIRATION1.html
• Slide 9 - puzzling together pieces
• http://www.aspire2develop.co.uk/images/People%20development%20page%2
0&%20HOME%20page%204%20piece%20jigsaw%20iStock_000002077956Smal
l.jpg
• slide 10 - Testing beyond the GUI
• http://courses.cs.vt.edu/csonline/OS/Lessons/Introduction/onion-skin-
diagram.gif

• Slide 12 – Leaning the architecture
• http://2.bp.blogspot.com/-
gO3DIydtaQA/UBoUj4afwsI/AAAAAAAAAvk/cwPAtXkcgJ4/s1600/12152009_muppets7.jpg
• Slides 13, 18, 21 22
• http://www.duvalschools.org/aprtech/student_life/BPA_Team_2/Exercise.html
• Slide 14 – Common organizational tools
• http://www.dislive.com/wp-content/uploads/2013/07/architect-interior-designer-or-
carpenter-workplace-with-desk-design-my-room-interior-design-tool-936x624.jpg
• Slide 15 – James Bach quote of Exploratory Testing
• https://admissionblog.usc.edu/files/2013/05/Question-Mark.jpg
IMAGES REFERENCES

• Slide 24 – Let’s Talk Data Storage
• https://encrypted-
tbn0.gstatic.com/images?q=tbn:ANd9GcR5pxcFFYJILFm1pg20ttHxc-
wPTm_05O4Yr7uhEP--PUZhy-bX
• Slide 27 – Conclusion
• http://www.mobileapptesting.com/wp-content/uploads/2011/03/Mobile-
Apps-300x188.jpg
IMAGES REFERENCES

Follow us: @XBOSoft
Questions at: services@xbosoft.com

Are You Missing Critical Mobile Tests?

More Related Content

Viewers also liked (6)

Similar to Are You Missing Critical Mobile Tests? (20)

More from XBOSoft (20)

Recently uploaded (20)

Are You Missing Critical Mobile Tests?

Editor's Notes