12 considerations for mobile testing (march 2017)

Brochure
12 considerations for
mobile testing
Make sure your solution includes these
essential capabilities
Get started

When it comes to mobile apps, there’s a lot at stake—from business revenue to customer loyalty and
brand image. This fact of life in the age of smart phones, tablets, and other mobile devices makes mobile
application testing extremely important and extremely daunting.
And this is where things get tricky. Today’s mobile application testing solutions vary greatly in their
capabilities. Many address only part of the mobile testing challenge, leaving you with critical unanswered
questions about the performance of your mobile app in the real world—for example, what security holes
might your mobile app be opening into your organization?
So how do you choose the right mobile testing solution? Begin with this list of 12 vendor considerations
for mobile testing.
Brochure

Mobility brings new testing challenges
Before you roll out a new app or a software upgrade, you want to be sure the app meets the expectations
of your users, addressing the three layers of functional, performance, and security considerations. It’s not
enough to test the functionality of the app in a lab setting. To have confidence in your app, you need to
conduct mobile performance and functional testing under realistic user, load, and network conditions.
So what does it take to test mobile applications? At its core, mobile application testing has the same basic
requirements of traditional application testing:
• Test environments must accurately reflect real-world conditions.
• Testing must be done under precisely repeatable conditions.
• Testing should include “what if” and edge scenarios.
• Testing must account for performance impacts on both users and infrastructure.
“It can be easy to assume that there is little difference between
managing the performance of a mobile web or native app and
managing the performance of traditional web and desktop
applications. But these are completely different beasts.”
— Aberdeen Group
Brochure

While that may sound simple enough, the reality is something else. Today’s testing requirements are
difficult to meet due to the dynamic nature of mobility and the variability of the mobile environment. For
example, the testing process for mobile apps must consider the wide range of end-user devices, many
versions of operating systems, the characteristics of different networks, and the variability of network
conditions, which change based on time of day, network utilization, and distance from the cell tower.
So how do you move forward? All of this points to the need for an end-to-end mobile testing solution that
has all the right stuff for the complete testing of your mobile apps under real-world conditions. To find your
ideal solution, begin with the following key considerations for your mobile app testing solution.
In the next few pages, we will review the 12 considerations for mobile testing.
Brochure
Common interruptions
and functionality
Object ID recognition
Testing under
real-world conditions
Manual and
automated testing
Remote access
to devices
Real devices
and emulators
Reuse of test scripts
Functional, performance,
and security testing
Native, web, and
hybrid apps
Development tools
and integration
ALM integration
Expertise
My test
solution

Key considerations for your
mobile testing solution
1. Does the solution support testing on real devices and emulators?
When it comes to mobile testing, one immediate consideration is whether to test your apps on real devices
or emulators that simulate test conditions. The pros and cons of each of these approaches should be fully
considered when formulating your mobile testing strategy.
Mobile emulators come in the form of client software packages that are free to download and run on your
local system. They provide a simple, low or often no-cost means of testing apps against different mobile
operating systems. This makes them a popular choice for testing code early in the development cycle.
But there are downsides to emulators. In particular, they don’t emulate all of the features and functions
of real devices, and they don’t run over a mobile network. So even if your app performs well, you can’t be
sure that it will work the same way on real devices. Testing on real devices is the only way to be sure you
understand the actual user experience, taking into account variables such as CPU, memory, and the specific
characteristics of the varied device matrix.
The bottom line: Look for a mobile testing solution that gives you the option to use emulators for some
tests and real devices for other tests—particularly those tests that require the highest levels of accuracy.
Brochure

2. Can test devices be accessed remotely?
Whether you’re in a small business or a global enterprise, you need a mobile testing solution that enables
remote access to a wide range of devices. Remote access gives you the flexibility to have your test devices
in one place and your developers and testers in another.
As for those test devices, they might be based in an in-house lab that you operate or hosted in a cloud by a
service provider. With a testing solution that supports remote access, you can use either approach—or both
approaches; this will give you greater flexibility and will help you scale your testing efforts.
Brochure

3. Does the solution support both manual and automated testing?
With manual approaches, your quality-assurance (QA) team conducts each step in the testing process
essentially by hand. With automated testing, the team leverages a script that runs through all of the same steps.
There will always be cases where manual testing makes sense, such as running checks on small changes to
code or doing other ad hoc testing, or simply because you haven’t built test automation skills within your QA
team yet. In fact, manual methods can be a good way to begin a mobile testing program if your organization
is not yet ready to invest in an automated testing solution. Ultimately, most organizations will want to move
to automated mobile testing to realize the associated time and labor savings, especially as releases become
more frequent and change becomes more fast paced.
Brochure
Consider this example of the number of staff hours consumed by manual testing processes over the course of a single year:
That’s the equivalent of 10 full-time staff positions. When you reflect on numbers
like those, it’s clear that you need to automate many of your tests. Automation can
greatly reduce the time and costs of testing, while giving you greater scalability
and broader test coverage, ultimately providing a better user experience. With
automation, your tests can cover more devices and more versions of the leading
mobile operating systems with much smaller requirements for staff time.
1 app
x 130 test cases
x 4 releases per year
x 20 supported devices
x 1.5 hours per test
15,600 hours of staff time

4. Can you test for real-world network conditions without setting up an entire environment?
Mobile apps are subject to many network-driven variables that aren’t an issue with web-based apps. With
mobile apps, network conditions such as jitter, latency, limited bandwidth, and packet loss must be taken into
account when testing an app and validating the end-user experience.
For example, streaming media is particularly susceptible to jitter. To avoid choppy and inconsistent playback,
and a poor end-user experience, data packets must arrive at the user’s device in the correct order and in a
timely manner. These network-related variables must be considered in the QA and development process.
While you could set up a lab to test your apps under different network conditions, that’s a costly and
time-consuming proposition. A better choice is to look for a mobile testing solution that supports network
virtualization to give you a clear view of the impacts of network conditions on the performance of your
mobile app without the need to set up a full production instance.
5. Does the solution test for common end-user interruptions and device functionality?
There are all kinds of things happening on mobile devices that could affect the performance and
functionality of an app. While users are interacting with your app, they might receive text messages, email,
and phone calls. Their batteries might run low. They might open a new URL from within your app.
These and other common interruptions can affect the performance and functionality of your app, so you
have to test for them. Your mobile testing solution should give you the ability to simulate common user
scenarios so you have a complete view of the impact on the end-user experience.
Brochure

6. Does the solution support object ID recognition?
In mobile app testing, each element on a screen is treated as an object that the test tool recognizes as part
of the recording and playback. Testing tools use different methods to recognize these objects. It’s important
to understand the differences in these methods when choosing a test solution. Some methods, while easier
to implement, have accuracy implications, and other solutions have better accuracy but require a higher-
level skill set.
In some cases, you may want to support both methods. However, if you have to support one approach due
to the reusability factor and the accuracy, your mobile test solution should support object ID recognition.
Brochure
Two of the most common methods are optical character recognition (OCR) and object ID recognition:
OCR relies on related text displayed in the application.
If the related text changes, moves, or is removed
entirely, the object is very difficult (or impossible)
to identify consistently. For this and other reasons,
OCR can be very slow and can be highly inaccurate.
Furthermore, the lack of repeatability means there
are issues with the reusability of scripts.
Object ID recognition works on the principal that every user-interface (UI)
object has a set of properties that can be used to identify, define, or validate
the object. When you use object ID recognition, you are using a unique ID to
locate and tag the object as part of the app. This means that if the object
moves, changes, or is removed, it is easier to map and your test is more
accurate (things such as position become irrelevant). In comparison to OCR,
object ID recognition enables high reusability and low-cost script
development and maintenance.

7. Does the solution support functional, performance, and security testing?
There are three layers in mobile app testing: functional, performance, and security:
• Mobile functional testing validates the functionality of the app. Does it work as it should? Does it do what
the user expects it to do?
• Mobile performance testing evaluates how well the app performs under different user, device, load, and
network conditions—e.g., does the app crash when more than 1,000 users log onto it.
• Mobile security testing identifies vulnerabilities in the software, such as access management issues, cross-
site scripting, or data storage leaks.
Each of these areas of focus is important to the end-user experience, so you want to look for a test solution
suite that supports all three. If you’re just getting started with mobile app testing, you might use a phased
approach, beginning with mobile functional testing, moving into mobile performance testing, and finally
incorporating security testing into your processes. Either way, regardless of the path you choose, you want
to work with a vendor who can help you achieve all three layers of testing.
8. Can you reuse test scripts?
To increase the return on your mobile app testing investments, you want a solution that makes it easy to
reuse test scripts. The ability to reuse test scripts across devices with the same OS will help you save time,
save money, and accelerate time to market for your apps, while increasing your test coverage.
The goal should be to reuse as many tests or portions of tests as possible, so you are not continually writing
the same scripts over and over for every variation of a device platform.
Brochure

9. Does the solution consider the specific characteristics of native apps, web, and hybrid apps?
Native apps are written in the specific language of the mobile operating system (OS). Web apps are written
in common web languages, such as HTML5 or Java. Hybrid apps are written in a combination of languages
and leverage a software shell that allows users to tap into the native capabilities of the device, such as the
camera or GPS, which pure web apps cannot do.
Your mobile testing solution should be designed to handle all three types of apps. Moreover, it should
understand the nuances and richer functionality of native apps, so you can test accordingly. For example,
compared to web apps, native apps have greater access to all of the functionality on the mobile device, and
your solution for iPhone® and Android testing needs to take this into account.
10. Do the testing tools integrate with development tools?
The modern software development lifecycle encompasses an ecosystem of tools used by different
players in the application delivery game. Especially in agile teams, which mobile teams tend to be, roles
often overlap—developers run certain types of test and QA specialists sometimes will get involved in
development activities.
While roles may blur at times, people always want to use the tools they are most comfortable with.
Developers don’t want to leave their integrated development environments (IDEs) to run tests. The same
line of thinking holds for QA specialists, who want to work within their familiar toolsets.
This reality of the age of agile processes and continuous software delivery creates the requirement for the
integration of development and testing tools. This integration enables higher levels of productivity and
helps you accelerate the delivery of high-quality software while still leaving the different personas in their
tools of choice.
Brochure

11. Do the testing tools integrate with ALM tools?
In addition to integration with development tools, your mobile testing solution should have the capability to
be integrated with your application lifecycle management (ALM) or agile project management tools. This
broader integration enables all stakeholders in the application delivery process to share information, to see
each other’s work, to assign tasks, to view test results, and to understand the status of the overall project.
Creating this traceability between assets and visibility into the end-to-end project is essential as it helps
stakeholders to see where additional efforts may need to be applied or whether the project will meet the
demanding timelines that mobile often imposes.
Furthermore, an additional area of integration to extend the lifecycle even further is between the mobile
testing and monitoring tools. This type of integration can help your test and development teams to prioritize
their efforts based on data from the real-world environment. For example, if the monitoring data shows that
a particular version of the OS is constantly crashing and a large number of users are leveraging that OS then
the development and test teams know that in the next iteration this should be an area of focus. Similarly, if
the development and test teams know that a particular device had issues in testing they can monitor the
extent to which the issues are impacting users by checking the monitoring data.
Brochure

12. Does the vendor provide best practices and service packages to help you understand your
testing needs?
To choose the right mobile testing solution for your organization, you need more than technology. You
need the expertise of mobile testing specialists who can help you explore your needs and identify the
components of a testing solution that meets your specific requirements.
What should you test? When should you test? What devices should you test on? When will you use
automation? How can you get the broadest range of coverage on a limited budget? To fully explore
questions like these, you need the help of seasoned testing specialists who have been down the same path
many times before and who can provide guidance based on previous experiences with other organizations
and knowledge of your specific industry or vertical.
Access to this expertise will help you increase your return on investment (ROI) and give you the confidence
that you can deliver the right user experience at the right time to the market, helping you stay ahead of the
competition.
Brochure

Meet all your mobile application
testing requirements
Unlike the many companies that address only part of the mobile testing challenge, HPE covers the full range of
today’s mobile testing requirements. HPE meets all 12 of the key vendor considerations for mobile testing.
Brochure
VENDOR CONSIDERATION HPE HPE MOBILE TESTING SOLUTIONS
1. Real devices and emulators HPE supports mobile app testing on both real devices and device emulators.
2. Remote access to devices HPE mobile testing solutions are designed to run tests on local and remote devices, including mobile clouds offered by partners.
3. Manual and automated testing HPE supports both manual and automated mobile testing.
4. Testing under real-world conditions HPE mobile testing solutions incorporate comprehensive tools for network virtualization.
5. Common interruptions and functionality HPE mobile testing solutions simulate common interruptions and functionality associated with end-user devices.
6. Object ID recognition HPE mobile testing solutions support object ID recognition and image-based recognition.
7. Functional, performance, and security testing HPE offers functional and performance testing via a single license and security testing via an additional license.
8. Reuse of test scripts HPE enables the easy reuse of scripts across devices of the same operating platform.
9. Native, web, and hybrid apps HPE mobile testing solutions are designed to consider the specific characteristics of native, web, and hybrid apps.
10. Development tools integration HPE mobile testing solutions are designed for integration with software development tools.
11. ALM integration HPE mobile testing solutions are tightly integrated with the HPE Application Lifecycle Management (ALM) platform and can
be integrated into agile project management solutions and HPE Application Performance Management and Monitoring stacks.
12. Expertise HPE mobile testing specialists offer expert insights into best practices for the design and use of mobile testing environments
and can offer testing as a service for a completely outsourced model.
In short, HPE has everything you need to create a fully tested, high-quality mobile experience that exceeds
the expectations of your users, allowing you to deliver amazing apps with confidence now.

Be sure with an end-to-end solution
Before you roll out a new or upgraded mobile app, you want to be absolutely sure you understand how your
software will perform for your users under real-world conditions. With end-to-end coverage of your testing
needs, HPE mobile app lifecycle solutions give you this confidence.
So don’t gamble with your business and its reputation. Be sure with HPE mobile testing.
Learn more at
https://saas.hpe.com/en-us/software/mobile-testing
https://saas.hpe.com/en-us/software/apppulse-mobile
Brochure
© Copyright 2014, 2017 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only
warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services.
Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or
omissions contained herein.
iPhone is a trademark of Apple Computer, Inc. registered in the U.S. and other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates.
All other third-party trademark(s) is/are property of their respective owner(s).
4AA5-4771ENW, March 2017, Rev. 1
Sign up for updates
“While every bit of
performance and
uptime data is useful
to understanding
mobile application
performance, in the
end the only metric
that matters is
end-user satisfaction.”
— Aberdeen Group

12 considerations for mobile testing (march 2017)

More Related Content

What's hot (20)

Similar to 12 considerations for mobile testing (march 2017) (20)

More from Antoine Aymer (6)

Recently uploaded (20)

12 considerations for mobile testing (march 2017)