Oh so you test? - A guide to testing on Android from Unit to Mutation

Paul Blundell
@blundell_apps
blog.blundellapps.com github.com/blundell
AutoTrader

- Define different types of testing
- Clarify testing & naming
- Benefits of each type of testing
- Codez available to help you
- Some architecture hints & tips
Abstract

Test Types
- Unit Tests
- Integration Tests
- Functional Tests
- Acceptance Tests
- Mutation Tests
- Vendor Tests
- Fuzz Tests

- Test a single block of code / function
- Test independently from collaborators
- Only test against interface definitions
- New behaviour : new unit test
Unit Tests - What

@Test
public void testGetLastPathSegmentReturnsFilename() {
SecureUrl secureUrl = new SecureUrl(
"http://test.com/some/random/url/filename.mp4");
String lastPathSegment = secureUrl.getLastPathSegment();
assertThat(lastPathSegment).isEqualTo("filename.mp4");
}
Unit Tests - Example

- Daily development work
- Allow for continuous refactoring
- Test Driven Development
- Documentation for legacy developers
Unit Tests - When

Unit Tests - Where
- All building blocks
- All levels of an application architecture
- Help pinpoint a place of failure

- Highlight UI issues
- Guarantee combined system integrity
- Prove integration of components
Unit Tests - Won’t

- Interaction between two or more classes
- Use real objects
- Can use threads
- Can access database
Integration Tests - What

@Test(expected = IllegalStateException.class)
public void testFindFragmentWithUnknownTagResourceIdThrowsError() {
Activity activity = Robolectric.buildActivity(Activity.class).get();
FragmentManager fM = activity.getFragmentManager();
Resources resources = activity.getResources();
TaskFragmentFinder finder = new TaskFragmentFinder(resources);
finder.findTaskFragment(fM, UNKNOWN_TASK_ID);
}
Integration Tests - Example

- Ensure parity of collaboration between objects
- Testing expected changes - db schema
- Testing environment integration
Integration Tests - When

- All building blocks
- Interactions between layers
- Help pinpoint integration issues
Integration Tests - Where

- Highlight UI issues
- Show specific code block of failure
- Give instant feedback
- Hard to diagnose some failures
- Full system confidence
Integration Tests - Won’t

- Whole system, nearly end to end
- Don’t care about intermediary steps
- Slower to run
- Mostly controller driven
Functional Tests - What

public void testShowsFilmInformationAfterApiCallFinished() {
String expectedTitle = "Swimming Pool";
startFilmActivity();
onData(is(instanceOf(Film.class)))
.atPosition(FIRST)
.onChildView(withId(R.id.film_text_view_title))
.check(matches(allOf(isDisplayed(),
withText(equalToIgnoringCase(expectedTitle)))));
}
Functional Tests - Example

- Evaluate expectation of sum of the parts
- Confirming feature completion
- TDD Keep you focused
- Confidence in features of system
Functional Tests - When

- Span whole architecture
- Touching live systems
- Above integrated components
Functional Tests - Where

- Show specific broken units of code
- Lack of knowledge of the details
- Give quick feedback for specific problems
Functional Tests - Won’t

- Black box tests
- From user perspective
- Specialised form of functional tests
- Model the final complete system
Acceptance Tests - What

public void testRotationInBrowseScreenMaintainsActionBar() {
swipeToBrowseScreen();
Activity activity = rotate(this);
assertActionBarOpen(activity);
}
private void assertActionBarOpen(Activity activity) {
assertTrue("Expected ActionBar open but was closed.",
activity.getActionBar().isShowing());
}
Acceptance Tests - Example

- User interface is integrated
- TDD feedback loop
- Capturing differences across devices
- Screenshots for greater feedback
Acceptance Tests - When

- Span whole architecture
- Touching live systems
- Above integrated components
Acceptance Tests - Where

- Run very fast
- No feedback upon cause (just symptoms)
- Help day-to-day incremental improvements
Acceptance Tests - Won’t

- Mutate the state of your code
- Insert faults in your software
- Unit tests are ran
- Tests fail “mutant is killed” thumbs up
- Tests pass “mutant survived” thumbs down
Mutation Tests - What

Mutation Tests - Example
if (a == b) {
// do something
}
will be mutated to
if (a != b) {
// do something
}
Different
Mutators:
if (a == b) {
// do something
}
will be mutated to
if (true) {
// do something
}
public int method(int i) {
i++;
return i;
}
will be mutated to
public int method(int i) {
i--;
return i;
}

- Find holes in your test suite
- Depends on % code coverage
- Confidence in your unit tests
- Complexity of your problem is high
Mutation Tests - When

- Outside of your code
- Around your test suite
- Not written yourself but configured
- Configuration is key
Mutation Tests - Where

- Give any confidence in working software
- Show collaboration between objects
- Prove if the application actually works
- Help at all without unit tests
Mutation Tests - Won’t

- can be unit, integration, functional
- learn how 3rd
party libraries work
- confirming understanding
- safety net for outside changes
Vendor Tests - What

@Test
public void setAndGetActiveSessionAreTrustworthy() {
Session.setActiveSession(mockFacebookSession);
FacebookSession session = Session.getActiveSession();
assertThat(mockFacebookSession).isEqualTo(session);
}
Vendor Tests - Example

- Incorporating 3rd party library
- Not beneficial to add retrospectively
- Replacing libraries with confidence
- Wanting to ‘in house’ a library feature
Vendor Tests - When

- Around your 3rd party libraries
- One off test
- Suite inside unit tests
Vendor Tests - Where

- Confirm your domain code
- Replace unit, integration or acceptance tests
- Adds overhead to development
Vendor Tests - Won’t

- Feeding your software random data
- Wait to see what breaks
- It is not logical
- Android = Application Exerciser Monkey
- Combinatory with other tools
Fuzz Tests - What

adb shell monkey -p com.your.package.name -v 50000
Fuzz Tests - Example
// Allowing start of Intent { act=mubi.intent.action.ON_BOARD cmp=com.mubi/.onboard.OnboardActivity } in package com.
mubi
:Sending Touch (ACTION_DOWN): 0:(1091.0,659.0)
:Sending Touch (ACTION_UP): 0:(1085.1356,667.17145)
:Sending Touch (ACTION_DOWN): 0:(467.0,404.0)
:Sending Touch (ACTION_UP): 0:(472.5769,398.45746)
// CRASH: com.mubi (pid 754)
// Short Msg: java.lang.IllegalStateException
// Long Msg: java.lang.IllegalStateException: Fragment WatchFragment{413d0d98} is not currently in the FragmentManager
// Build Label: google/nakasi/grouper:4.1.1/JRO03H/405518:user/release-keys
// Build Changelist: 405518
// Build Time: 1364293068000
// java.lang.IllegalStateException: Fragment WatchFragment{413d0d98} is not currently in the FragmentManager
// at android.app.FragmentManagerImpl.saveFragmentInstanceState(FragmentManager.java:586)
// at android.support.v13.app.FragmentStatePagerAdapter.destroyItem(FragmentStatePagerAdapter.java:140)
// at android.support.v4.view.ViewPager.setAdapter(ViewPager.java:413)
// at com.mubi.onboard.OnboardActivity.updateViewPager(OnboardActivity.java:139)
// at com.mubi.onboard.OnboardActivity.onRetrieved(OnboardActivity.java:132)
// at com.mubi.onboard.OnboardTaskFragment$3.run(OnboardTaskFragment.java:77)
// at android.os.Handler.handleCallback(Handler.java:725)
// at android.os.Handler.dispatchMessage(Handler.java:92)
// at android.os.Looper.loop(Looper.java:137)
// at android.app.ActivityThread.main(ActivityThread.java:5195)
// at java.lang.reflect.Method.invokeNative(Native Method)
// at java.lang.reflect.Method.invoke(Method.java:511)
// at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:795)
// at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:562)
// at com.android.internal.os.ZygoteInit.main(Native Method)
// at dalvik.system.NativeStart.main(Native Method)
//
** Monkey aborted due to error.
Events injected: 4191

- From the beginning
- Thin slice (UI)
- Continuous integration, faster feedback
- Identifying differences across Android versions
- Highlight refactoring effort
Fuzz Tests - When

- Black box testing
- On top of your application (Android)
- Custom fuzz testing - input level
- Continuous integration
Fuzz Tests - Where

- Prove application is running correctly
- Only reports crashes
- Highlights quality rather than bugs
- Not a replacement for unit, integration etc
Fuzz Tests - Won’t

Helpers / Libraries
- Brief overview
- What test types it benefits
- Any positives
- Any negatives

Mockito
//mock creation
List mockedList = mock(List.class);
//using mock object - doesn’t throw exceptions
mockedList.add(“one”);
//selective & explicit verification
verify(mockList).add(“one”);
Mockito is a mocking framework that tastes really good. It lets you write beautiful tests with clean
& simple API. Mockito doesn't give you a hangover because the tests are very readable and they
produce clean verification errors.
- Unit Tests : mocking dependencies
- Integration Tests : questionable use

Fest
FEST is a collection of libraries, whose mission is to simplify software testing.
No more confusion about the order of the "expected" and "actual" values. Our assertions are very
readable as well: they read very close to plain English, making it easier for non-technical people
to read test code.
Regular JUnit:
assertEquals(View.GONE, view.getVisibility());
Regular FEST:
assertThat(view.getVisibility()).isEqualTo(View.GONE);
FEST Android:
assertThat(view).isGone(); - Unit Tests
- Integration Tests
- Functional Tests
- Acceptance Tests : anywhere you do assertions
improving readability

“Robolectric is a unit test framework that de-fangs the Android SDK jar so you can test-drive the
development of your Android app. Tests run inside the JVM on your workstation in seconds. “
Robolectric
@RunWith(RobolectricTestRunner.class)
public class ViewingQueryTaskTest {
@Test
public void testAlwaysClosesCursor() {
viewingQueryTask.run();
verify(mockCursor).close();
}
}
- Unit tests : for day to day activity including TDD
- Integration tests : to handle Android
- Positive : speed vs on device tests
- Negative : the rabbit hole

“Robotium is an Android test automation framework. Robotium makes it easy to write powerful
and robust automatic black-box UI tests. With the support of Robotium, test case developers can
write functional and user acceptance test scenarios, spanning multiple Android activities.“
Robotium
- Functional tests : check when you click a button the
shared preferences are updated
- Acceptance tests : ensure features have been
completed and take screenshots for feedback
public class EditorTest extends ActivityInstrumentationTestCase2<EditorActivity> {
private Solo solo;
public void setUp() throws Exception {
solo = new Solo(getInstrumentation(), getActivity());
}
public void testFileExtensionIsInMenu() {
solo.sendKey(Solo.MENU);
solo.clickOnText("More");
solo.clickOnText("Preferences");
solo.clickOnText("Edit File Extensions");
Assert.assertTrue(solo.searchText("rtf"));
}
}

Use Espresso to write concise, beautiful, and reliable Android UI tests
Espresso tests state expectations, interactions, and assertions clearly without the distraction of boilerplate
content, custom infrastructure, or messy implementation details getting in the way.
Espresso
public void testSayHello() {
onView(withId(R.id.name_field).perform(typeText("Dave"));
onView(withId(R.id.greet_button)).perform(click());
onView(withText("Hello Dave!")).check(matches(isDisplayed()));
}
- Functional tests : check when you click a button the shared
preferences are updated
- Acceptance tests : ensure features have been completed, doesn’t
support screenshots as an API but you can still do it
http://tiny.cc/disableAnim
http://tiny.cc/espressoScreenshot

The Monkey is a program that runs on your emulator or device and generates pseudo-random
streams of user events such as clicks, touches, or gestures, as well as a number of system-level
events. You can use the Monkey to stress-test applications that you are developing, in a random
yet repeatable manner.
UI Exerciser Monkey
adb shell monkey -p com.your.package.name -v
50000
- Fuzz tests : as already discussed allows you to test input events
Don’t get the Application Exerciser Monkey & the Monkey Runner mixed
up.
App Ex Monkey – Fuzz Testing
Monkey Runner – Android device control using Python scripts

The uiautomator testing framework lets you test your user interface (UI) efficiently by
creating automated functional UI testcases that can be run against your app on one or more
devices.
UI Automator
extend UIAutomatorTestCase
UIObject & UISelector
Min SDK 16 Jelly Bean
adb shell uiautomator runtest MyTests.jar -c com.example.MyApp
Tests are in the jar, and the package is the package of the app under test.
- Acceptance tests : acts just like a user.

Architecture Empower / Encumber

- Allows for separation of concerns
- Java only module
- Directed testing
- Encapsulating change
- Reuse
Modules / Modularity Empowers

- Readable & flexible CI builds
- Flavors, build types
- Swapping out whole components
- Greater range of testing flexibility
Gradle Empowers

- Test reliability
- Test speed
- Data integrity
Mock Services Empower

- Hexagonal architecture
- Insulates system from outside dependencies
- Empowers testing
- Disconnect from platform requirements
Ports & Adapters Empower

- Quality of your code reflects quality of your
tests
- Object Oriented programming lends itself to
quality testing
- TDD
- Static methods are evil
Low Code Quality Encumbers

- Framework lock in
- Google Play services
- Android code examples
- Activity life cycle
Android (frameworks) Encumber

- Believe in testing
- Believe in clean code
- Software craftsmanship
- TDD
- Be energetic
- Enjoy it
YOU Empower || Encumber

Monkey Runner
Spoon
Jmock
Hexagonal Architecture
Contract Tests
Others
Groovy Tests
Static analysis tools
Charles
cURL
Postman
Google sizes (small medium large)

Online Blogs
Martin Fowler http://martinfowler.com/tags/testing.html
JB Rainsberger http://blog.thecodewhisperer.com/
Kevin Rutherford http://silkandspinach.net/
Robert Martin https://sites.google.com/site/unclebobconsultingllc/blogs-by-robert-martin
References
More Info
Android Fuzz Tests http://developer.android.com/tools/help/monkey.html
Hardcore Fuzz Tests http://www.ibm.com/developerworks/library/j-fuzztest/index.html
Vendor Tests https://www.youtube.com/watch?v=47nuBTRB51c#t=23m34s
Mutation Tests http://pitest.org/
Google Testing Opinion http://googletesting.blogspot.co.uk/2010/12/test-sizes.html
Mockito https://code.google.com/p/mockito/
Mocking Frameworks http://blogs.telerik.com/skimedic/posts/13-07-23/top-5-reasons-to-use-a-mocking-framework
Android Fest https://github.com/square/fest-android
Robolectric http://robolectric.org/
Robotium https://code.google.com/p/robotium/
Espresso https://code.google.com/p/android-test-kit/wiki/Espresso
UI Automator http://developer.android.com/tools/help/uiautomator/index.html
SOLID Principles http://www.its-on-the-internet-so-it-must-be-true.com/2012/06/solid-principles.html

Paul Blundell
Questions?
@blundell_apps

Oh so you test? - A guide to testing on Android from Unit to Mutation

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Oh so you test? - A guide to testing on Android from Unit to Mutation (20)

More from Paul Blundell (12)

Recently uploaded (20)

Oh so you test? - A guide to testing on Android from Unit to Mutation