Automating the Gaps of Unit Testing Mobile Apps

Automating the Gaps of
Unit Testing Mobile Apps
Effectively Utilizing Xcode’s Instruments

@ggeoffreggeoffre.com
20+ years of experience in the development, marketing, sales,
and leadership of computer software development
A C T S

Preparing your QA team for mobile testing
http://www.slideshare.net/ggeoffre

Where did the term "bug"
come from?
66 years ago on
September 9th
1947, operators of
the Mark II Aiken
Relay Computer
being tested at
Harvard University,
found something
curious trapped
between points at
Relay #70, Panel
F.

Bugs not Defects!
Outside the scope of most acceptance or unit testing
Tend to design around them not to them
Defensive programming techniques employed
More likely hardware based than software based
Environment plays a major role discovering them
Magnified greatly with mobile based development

Today’s Topics
Reference Counting
Battery Usage
Network Monitoring
Communicating Build and Version Numbers
Xcode Server Initial Setup and Bot Creation
Test Deployment Options to Devices
Crash and User Analytics Reporting
DevOps
UITesting
Instruments
Automation Instrument
Calabash Appium
UIAutomation XCTest

Automating the Gaps of Unit Testing Mobile Apps

Xcode ▶︎Open Developer Tool ▶︎Instruments

[Objective-C]
alloc/init
retain
copy
release
autorelease

Properties and @IBOutlets
atomic
nonatomic
strong
weak
retain
assign
unsafe_unretained
copy
readonly
readwrite

Automatic Reference Counting
Edit => Convert => To Objective-C ARC
Project => Build Settings

Swift
When to use ? and when to use !
When to use “if let”?
When to use as? and when to use as

WWDC 2015
Optimizing Swift Performance
https://developer.apple.com/videos/play/wwdc2015-409/

WWDC 2016
Understanding Swift Performance
https://developer.apple.com/videos/play/wwdc2016/416/

Leaks
The Leaks profiling template uses the Allocations
and Leaks instruments to measure general
memory usage in your app and check for leaks—
memory that has been allocated to objects that
are no longer referenced and reachable.
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/FindingLeakedMemory.html
iOS Developer Library - Instruments User Guide

Leaks
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/FindingLeakedMemory.html

Allocations
The Allocations profiling template uses the
Allocations and VM Tracker instruments to
measure general and virtual memory usage in
your app. However, to track down abandoned
memory that’s been allocated but isn’t needed
again, you’ll focus strictly on the Allocations
instrument. This instrument measures heap
memory usage and tracks allocations, including
specific object allocations by class.
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/FindingAbandonedMemory.html

Allocations
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/FindingAbandonedMemory.html

Zombies
The Zombies profiling template uses the
Allocations instrument to measure general
memory usage in your app, with a focus on the
detection of overreleased “zombie” objects—that
is, objects that are called after they’ve been
released and no longer exist.
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/EradicatingZombies.html

Zombies
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/EradicatingZombies.html

Energy Savvy Users
Battery Life is very important to
users
iOS Now Displays Apps in
order of Battery Usage
Be sure to test in Low Power
Mode

Energy Diagnostics
Settings > Developer >
Instruments
Enable Energy Recording
Start Recording
Xcode > Instruments
New Energy Log
File > Import Legged Data
from Device

WWDC 2015
Debugging Energy Issues

Data Savvy Users
Data caps are real
iOS displays cellular data
usage and allows users to turn
on/off apps
Be sure to test with Network
Conditioner

Network Conditioner
Settings > Developer >
Network Link Conditioner
Enable
Choose a Profile

Network Monitoring
Use iTunes to find your UDID
UDID

Network Monitoring
Use Xcode Devices Window to find your UDID
UDID

Network Monitoring
rvictl -s <UDID>
ifconfig rvi0
tcpdump -s 0 -A 'tcp dst port 80 and
(tcp[((tcp[12:1] & 0xf0) >> 2):4] =
0x504f5354)'
...
rvictl -x <UDID>

HTTPS Server Trust Evaluation
Technical Note TN2232
App Transport Security Exceptions
<key>NSAppTransportSecurity</key>
<dict>
<key>NSExceptionDomains</key>
<dict>
<key>yourdomain.com</key>
<dict>

<key>NSIncludesSubdomains</key>
<true/>

<key>NSTemporaryExceptionAllowsInsecureHTTPLoads</key>
<true/>

<key>NSTemporaryExceptionMinimumTLSVersion</key>
<string>TLSv1.1</string>
</dict>
</dict>
</dict>

macOS Sierra Apps
WireShark
tastycocoabytes.comwireshark.org
Cocoa Packet Analyzer

WWDC 2015
Your App and Next Generation Networks

Asynchronous Programming
Concurrent code execution on multicore
hardware
Grand Centeral Dispatch
Blocks
Closures
UIView Animations
NSNotifications
Local and Remote Push Notifications

Blocks [Objective-C]
dispatch_async(/* a queue /*, ^{
/* your code here */
});
dispatch_get_main_queue()
dispatch_get_global_queue(
DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)

dispatch_async(/* a queue /*, {
})
Closures Swift
dispatch_get_main_queue()
DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)

func perform_on_main(block: dispatch_block_t?)
{
dispatch_async(
dispatch_get_main_queue(), block)
}
func perform_in_background(block: dispatch_block_t?)
{
dispatch_async(
DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), block)
}
Closures Swift

perform_on_main {
}
perform_in_background {
}
Closures Swift

[UIView animateWithDuration:1.0
delay: 1.0
options: UIViewAnimationOptionCurveEaseIn
animations:^{
} completion:^(BOOL finished){
}];
[Objective-C]UIView Animation

UIView Animation
UIView.animateWithDuration(0.7,
delay: 1.0,
options: .CurveEaseIn,
animations: {
}, completion: { finished in
})
Swift

NSNotifications
[[NSNotificationCenter defaultCenter]
addObserver:self
selector:@selector(someMethod:)
name:@"SOME_NOTIFICATION"
object:nil
];
postNotificationName:@"SOME_NOTIFICATION"
object:self
];
removeObserver:self
];
[Objective-C]

NSNotifications
NSNotificationCenter.defaultCenter()
.addObserver(self,
selector:"someMethod",
name:"SOME_NOTIFICATION",
object:nil)
.postNotificationName("SOME_NOTIFICATION",
object: self)
.removeObserver(self)
Swift

WWDC 2015
Building Responsive and Efficient Apps with GCD

WWDC 2015
Designing with Animation

WWDC 2011
Using Local And Push Notifications

WWDC 2015
What's New in Notifications

WWDC 2016
Advances in UIKit Animations and Transitions

WWDC 2016
Thread Sanitizer and Static Analysis

iOS Human Interface Guidelines
https://developer.apple.com/ios/human-interface-
guidelines/overview/design-principles/

UIKit Catalog (iOS)
https://developer.apple.com/library/content/samplecode/UICatalo
g

Accessibility for Developers
https://developer.apple.com/accessibility/

Accessibility Audit
Xcode => Open Developer Tool => Accessibility Inspector

Accessibility Attributes
name Derived from the
accessibility label.
value The current value of the
control, for example, the text in a
text field.
elements Any child elements
contained within the current
element, for example, the cells in
a table view.
parent The element that contains
the current element.
The name property is one of four properties of these
elements that can be very useful in your test scripts.
Select and Set

Accessibility Inspector
Xcode => Open Developer Tool => Accessibility Inspector
Accessibility Label

WWDC 2015
iOS Accessibility

Auditing Your Apps for Accessibility
WWDC 2016

AUTOMATION
Behavior Driven Development
Acceptance Tests
JavaScript
Tool Based (Command Line Execution)
Instruments

AUTOMATION
Install Xcode from the App Store
https://itunes.apple.com/us/app/xcode/id497799835?mt=12
Xcode => Open Developer Tool => Instruments
Choose “Automation”
Select the target app to test with
Record/Run

AUTOMATION
Accessibility Label

TuneupJS.org
Tune-up is a collection of JavaScript utilities that
builds upon and improves the UIAutomation library
provided by Apple for testing iOS applications via
Instruments (get it? "tune-up"? Instruments? get
it?).
http://www.tuneupjs.org

T XCUIApplication()
Acceptance Tests
Swift or Objective-C (XCTest)
Tool Based, Project Test Target, (Command Line
Execution)
Xcode, Xcode Server Bot

T XCUIApplication()
Install Xcode from the App Store
https://itunes.apple.com/us/app/xcode/id497799835?mt=12
Create a new Project
Product => Test
Record

T XCUIApplication()
let app = XCUIApplication()
let someTextField = app.textFields["MyCustomLabel"]
Find Things
someTextField.tap()
someTextField.typeText("qwerty")
Do Some Stuff
let expectaion = self.expectationWithDescription("Something")
expectaion.fulfill()
self.waitForExpectationsWithTimeout(5.0, handler: nil)
Set Expectations

T XCUIApplication()
xcodebuild test
-project SampleProject.xcodeproj
-scheme SampleProject
-destination
'platform=iOS Simulator,
name=iPhone 6s'
Command Line

Acceptance Tests
Ruby/Cucumber/Bash
Command Line Scripting
RubyMine (optional)

Clone the iso-smoke-test-app repo…
https://github.com/calabash/ios-smoke-test-app
Install the calabash sandbox… (setup Ruby)
https://github.com/calabash/install
Build a .app for the Simulator…
Run your first cucumber test:

Scenario: Touch OK button on alert
When I touch the show alert button
Then I see an alert
Then I see the "Smoke Test!" alert
And I wait for all animations to stop
And I can dismiss the alert with the OK button
When(/^I touch the show alert button$/) do
wait_for_element_exists("view marked:'show alert'")
touch("view marked:'show alert'")
end
Features
Steps
Ruby
Gherkin

Acceptance Tests
Earl Grey

WWDC 2015
UI Testing in Xcode

Managing Simulators
Visible and Meaningful Version
Numbers
Xcode Server Bots for CI
iTunes Connect Crash Analytics
Flurry User Analytics and Reports
Fabric Crash Analytics Support
HockeyApp QA and Beta Testing
Some DevOps Tips

Manage Simulators
Window => Devices

Add a new Run
Script Phase
Reset All Simulators
osascript -e 'tell application "iOS Simulator" to quit'
osascript -e 'tell application "Simulator" to quit'
xcrun simctl erase all
Project => Target => Build Phases

Run Script Git Version Info
Build Version Number
buildVersion=$(/usr/libexec/PlistBuddy -c "Print CFBundleShortVersionString”
“${PROJECT_DIR}/${INFOPLIST_FILE}")
buildNumber=$(git rev-parse --verify HEAD | cut -c1-5)
buildString=$buildVersion' ('$buildNumber')'
/usr/libexec/PlistBuddy -c "Set PreferenceSpecifiers:1:DefaultValue $buildString”
"${SRCROOT}/Settings.bundle/Root.plist"
/usr/libexec/PlistBuddy -c "Set :CFBundleVersion $buildNumber"
"${PROJECT_DIR}/${INFOPLIST_FILE}"

File => New => Settings Bundle

Settings Bundle Root.strings
/* A single strings file, whose title is specified in your preferences schema. The strings
files provide the localized content to display to the user for each of your preferences. */
"Group" = "SampleProject";
"Name" = "Version";

Settings Bundle Root.plist
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict>
<key>PreferenceSpecifiers</key>
<array>
<dict>
<key>Title</key>
<string>About</string>
<key>Type</key>
<string>PSGroupSpecifier</string>
</dict>
<dict>
<key>DefaultValue</key>
<string>1.0</string>
<key>Key</key>
<string>version_preference</string>
<key>Title</key>
<string>Version</string>
<key>Type</key>
<string>PSTitleValueSpecifier</string>
</dict>
</array>
<key>StringsTable</key>
<string>Root</string>
</dict></plist>

Install OS X Server and Xcode on the target Mac
Enable the Xcode service (administrator account)
Add the remote repository for the project
Create a “BOT” to build and test the project
View the results in the browser
Setup Xcode Server

WebView Screen Saver
GitHub / liquidx / webviewscreensaver

WWDC 2016
Advanced Testing and Continuous Integration

Crash Analytics
Technical Note TN2151
Understanding and Analyzing Application Crash Reports
https://developer.apple.com/library/content/technotes/tn2151/_index.html
App Distribution Guide - Analyzing Crash Reports
https://developer.apple.com/library/content/documentation/IDEs/Conceptual/
AppDistributionGuide/AnalyzingCrashReports/AnalyzingCrashReports.html

WWDC 2015
Getting the Most out of App Analytics

Flurry
Measure, track and analyze app performance, user
acquisition and activity with Flurry Analytics. Get a deep
understanding of app performance metrics and everything
your users are doing.
Create a Yahoo account and get your API Keys
Add the Flurry framework to your project
Integrate Flurry Analytics into your App
https://developer.yahoo.com/analytics/

Fabric hooks deeply into your existing build process and
automatically upload your dSYMs so you don’t have to. Now
you can sleep well, knowing you’ll get the crash reports you
need to keep your app stable and your users happy.
Create and confirming your Fabric account
Select the project you want to integrate Fabric into
Add the Fabric framework to your project
Fabric
https://developer.apple.com/testflight/

Distribute beta versions, collect live crash reports, get
feedback from real users and analyze test coverage. Bring
DevOps to your apps by integrating with your existing build
and work item management.
Create an Account to Manage Teams and Apps
Uploading a Build for an App
Set Up and Invite Testers
HockeyApp
http://hockeyapp.net/features/

TestFlight Beta Testing makes it easy to invite users to test
your iOS, watchOS, and tvOS apps before you release them
on the App Store. You can invite up to 2,000 testers using
just their email address.
Create an App Store iTunes Connect Record
Uploading a Build for an App
Set Up and Invite Testers
TestFlight
https://developer.apple.com/testflight/

ggeoffre
slideshare.net
http://www.slideshare.net/ggeoffre

Automating the Gaps of Unit Testing Mobile Apps

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to Automating the Gaps of Unit Testing Mobile Apps (20)

Recently uploaded (6)

Automating the Gaps of Unit Testing Mobile Apps