![Preface
The app ecosystem is enormous. We have grown dependent on smartphone apps for
almost every aspect of our lives. When the apps don’t perform as expected, the
consequences can range from midly irritating to life-threatening. For this reason, testing
smartphone apps is very important—particularly the GUI that is the main window into the
app’s functionality. Unfortuanately, GUI testing can be a time-consuming process, which
leads to fewer tests being run, further exacerbating the app’s quality problems.
This book focuses on the specific problem of GUI testing for iOS apps found on
Apple’s products such as the iPhone and the iPad. Apple provides developers with a
testing framework called UI Automation, but its capabilities are limited in terms of
speeding up the testing process. The result is that GUI testing of complex iOS apps can
take many hours.
The solution proposed here is to leverage the parallelism inherent in the Hadoop
distributed platform to provide an environment for concurrent test execution. The
approach builds upon an existing system, called HadoopUnit, that was previously used to
reduce regression testing time of large JUnit test suites. HadoopUnit has been customized
to drive UI Automation test cases in a manner that is easy for developers and testers to
adopt, yet provides measurable improvement in test case execition times.
WHAT IS UNIQUE ABOUT THIS BOOK?
This book represents the continuation of research that began in 2009 on addressing the
problem of execution times for large regression test suites. The first of this work were the
HadoopUnit distributed exection environment and the SMART-T migration decision
framework [60]. The work also led to the creation of a new community of researchers and
practicioners interesting in software testing in the cloud (STITC) [59][62].
The STITC project evolved to examine the applicability of testing as a service
(TaaS) to hard problems in software testing (HPST) [61][58]. TaaS is a promising new
development that offers a service-oriented interface to the testing capabilities provided by
an environement like HadoopUnit. The number one problem found in the HPST project
was education & training, and it’s currently an open question where TaaS may help
alleviate this timeless challenge.
GUI testing of iOS apps is a timely update and specific instance of the classic
regression testing problem, and one that HadoopUnit is well suited to address. The
research reported in this book is unique is its application of an advanced environment such
as HadoopUnit for concurrent testing in manner that is accessible to almost all developers
and testers—even if they are of modest means. A simple two-node cluster is all that is
needed to realize significant testing benefits.
WHO SHOULD READ THIS BOOK?
Anyone who is involved in GUI testing of iOS apps will find the material presented in this
book valuable. This is particularly true for testers, but developers, managers, and even](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-17-320.jpg)
![CHAPTER 1
Introduction
Mobile applications have become an important part of the software development industry
and a driver of the overall technology economy. In June 2014, Apple announced that 75
billion apps had been downloaded from its App Store [53]. As shown in Figure 1.1, Apple
is on track for 85 billion downloads by the end of October. Gartner predicts the total
number of mobile apps downloaded across all platforms to double by 2016 [22].
Figure 1.1: App store downloads [53].
One of the main factors behind these impressive numbers is the growing use of
smartphones and tablets. Smartphones will soon pass personal computers to become the
dominant computing platform worldwide [49]. With so many apps in use, quality becomes
an important issue. A game crashing is one thing; an app interacting with essential systems
is something else altogether.
In July 2014, Apple and IBM entered into a strategic partnership to develop
business-focused apps for the enterprise [45]. In October 2014, IBM and SAP entered into
a strategic partnership wherein IBM will provide the cloud infrastructure for SAP’s
enterprise applications, including its growing number of mobile apps [24]. These
agreements signal a clear shift in the app ecosystem, away from a purely consumer-
oriented world to one that embraces corporate users as well. This shift will only increase
the need for high-quality mobile apps.
One way that helps ensure quality of mobile applications is through testing.
Software testing is one of the main activities in the software engineering life cycle](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-21-320.jpg)
![(requirements, design, construction, testing, and maintenance). Testing is a very important
process that helps assess the functionality and ensure the quality of software before being
released by identifying defects and problems [8]. System-level GUI testing, which is a
primary focus of this work, is especially important in mobile application testing because it
tests an application after all components has been integrated to ensure that the application
behaves as intended [47].
On the iOS platform, GUI testing can be done through the use of Apple’s UI
Automation framework [4]. It was first introduced with iOS 4 as part of Instruments, a
suite of profiling tools for tracing application’s behaviors and performance related issues
at runtime. UI Automation allow users to write scripts to exercise the user interface of an
iOS application by simulating users’ gestures, such as tapping, pinching, swiping, or
flicking. With UI Automation, testers do not have to repeatedly perform the same set of
actions on the app GUI every time code is integrated into the baseline.
1.1 GUI TESTING OF IOS APPS
At the 2010 WWDC conference Apple made a very interesting statement when they first
introduced UI Automation: “Automating User Interface Testing with Instruments or How
to Find Bugs While You Sleep” [2]. This statement implied that execution of a GUI test
suite could take a considerable amount of time to finish. There are two main reasons for
slow test execution: the number of test cases to run and the execution time of each test
case.
A good GUI test suite covers all functionality of an application accessible through
its user interface and exercises every user interface element presented to users on screen.
The user interface of a mobile application as well as the variations through which users
can interact with the application can grow large in number and the amount of test cases to
cover in a test suite increases accordingly [13]. Moreover, the test suite continues to grow
over the course of software development process, especially when agile methods are used:
more test cases are added each iteration. The testing also needs to account for different
versions of operating systems, device orientations, and form factors of mobile devices.
The execution speed of GUI testing also can be very slow and time-consuming. This
is because testing through the GUI proceeds in real time, at end-user speeds, not processor
speeds [39]. This type of testing asserts the behaviors of an application from the outside,
just as the users would actually see. This means that the testing tool needs to constantly
wait for the rendering of user interface elements and animations, or wait for network
delays throughout the test execution. Combining the slow test execution speed with the
total number of test cases results in the test process being tediously slow.
GUI test execution of a realistic test suite can take an hour or more to execute, and
if the user interface of the application is complicated, the time required for a test execution
to complete might go beyond the time and resources available for a project. This limits
how many times testing can be performed, which in turn affects app quality.
To solve the problems presented by slow-running large test suites is more
complicated than simply having multiple separated processes executing more test scripts](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-22-320.jpg)
![on a single machine. One reason is that with UI Automation, only one test script could be
run at a time. Moreover, only one instance of iOS simulator could be run at a time [4]. A
different solution is needed.
1.2 RAPID TESTING WITH HadoopUnit
There are many solutions proposed to solve the test execution time problem. One way is to
reduce the number of test cases to run. There are many researchers [36, 23] who propose
various approaches to reduce the number of test cases to run. Instead of executing the
whole regression test suite, selective re-test techniques attempt to execute only a subset of
test cases in a test suite. There are several such techniques, such as test case selection [51],
test case prioritization [66], and test suite reduction [25]. However, these kinds of
techniques are complicated and can be computationally expensive, as decisions have to be
made on which test cases not to run. With such decisions, some of the regression bugs
might have been missed with the dropped test cases if the analysis is not flawless. The re-
test all approach is more straightforward in that it avoids such errors [23]. Further
discussion on regression test selection is provided in Section 2.1.1.
Another approach to mitigate the problem is to move to a faster test execution
environment. A pre-existing framework, called HadoopUnit [46], was designed to solve
this problem. In previous studies, HadoopUnit provided an infrastructure for executing
JUnit test cases concurrently in a cluster or in the cloud to speed up the test execution
process. The result from the previous studies have shown that using a medium-scale
cluster of 150 nodes, the test execution runtime of Hadoop’s JUnit test suite with 230 test
cases could be impressively improved for up to approximately 55 times faster [60] by
executing a single test per node. However, because not everyone has access to large
amount of IT resources, even with a medium cluster as shown from the previous studies,
there is also a study of HadoopUnit on a smaller-scale of cluster of 4 nodes and the result
have shown that such impressive performance improvement is not always guaranteed.
Some modifications had been made to HadoopUnit to further fine-tune its performance by
letting each node to execute more than one test concurrently [9].
The structure of the HadoopUnit platform and the problems that it solved by
reducing the execution time of JUnit test suite make it a suitable candidate as a platform
for system-level GUI testing of iOS applications. In order to migrate testing to a
distributed execution environment, several artifacts involved in the testing process of the
current test execution environment must be identified. The important artifacts in system-
level GUI testing of iOS applications include a set of test cases that composes a test suite,
the UI Automation testing tool, a trace document capturing the test results of each test
execution, and the operational test environment. These artifacts are studied in this book for
feasibility in migrating to a new environment, with HadoopUnit customized accordingly
as needed.
1.3 RELATED WORK
There are several other GUI testing tools and distributed testing platforms related to the
work presented in this book. The GUI testing tools described below offer alternatives to](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-23-320.jpg)
![Apple’s UI Automation, but come with their own caveats. There are fewer choices when it
comes to distributed testing platforms currently available, but that may change if Testing
as a Service becomes more prevalent.
1.3.1 GUI TESTING TOOLS
The primary reason that UI Automation was selected as a testing tool for GUI testing tool
in this book was because it works without requiring software engineers to install any
additional software or customize their Xcode projects. However, there are several third-
party testing tools available that could be used GUI testing as well. The main reason to
consider a third-party testing tool is when the team was already familiar with the tools or
language used by the tools. Although they rely on different technologies and
implementations, they share similar concepts on how these testing tools work together.
As shown in Figure 1.2, there are three common components in these testing tools:
an agent, a script or feature, and an IDE.
1. An Agent: A framework or a small server that will be installed to run inside the
system under test. The installation needs to be performed on an iOS project
manually through the Xcode IDE.
2. A Script or a Feature: The main set of test scripts for the system under test.
Programming languages used for each tool will be different based on the
underlying technologies used by the testing tool. For example, Objective-C [5],
JavaScript [42], CoffeeScript [6], or Cucumber [26].
3. An IDE: This component is separate piece of software running on a tester
machine or remote server that helps record the interactions, execute the scripts,
and issue commands that interact with the system under test through the agent
component installed inside the application.
Figure 1.2: Overall architecture of third-party testing tools.
GUITAR [43] is a framework for performing automated GUI testing. The GUITAR
framework has several notable features including automated test case generation and
execution. It consists of four main components: (1) Ripper for generating a structure](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-24-320.jpg)
![model of the GUI of the system under test; (2) Graph Converter for converting the
structure model into a graph; (3) the Test Case Generator for generating test cases based
on the graph; and (4) Replayer for automated executing the generated test cases. GUITAR
has a plugin-based architecture meaning that a specific plugins are needed to support
different platforms. GUITAR supports multiple platforms, such as Java, Web, and
Android. iPhone GUITAR [19] is still under-development to support GUI testing of iOS
application. One weakness of GUITAR is that it does not support any manual test cases
development (both scripted and captured) and leads instead towards automating the whole
process.
MonkeyTalk [11] is a free, open-source, object-based recording tool from Gollila
Logic that examines the app’s code to understand the underlying interactions. This
approach is supposed to be more accurate that image-based recording that detects pixel on
the screen. It also results in a more readable and maintainable script when compared to
scripts that rely on image recognition technology. MonkeyTalk has its own IDE to
construct or record a new test script and playback existing scripts. There are a lot of
actions and gestures to select from, including wait time specified in milliseconds.
MonkeyTalk uses its own proprietary MonkeyTalk Scripts for constructing test cases
while still supporting JavaScript. MonkeyTalk also support command-line test execution
through either Ant Runner or Java Runner.
Frank [57] is a tool for automated iOS testing by Pete Hodgson. It is described by
the tool’s developers as “Selenium for native iOS apps.” Frank uses a combination of
Cucumber and JSON commands that are sent to a server running inside the native
application. The benefit of using natural language like Cucumber is its readability for non-
technical stakeholders in the project. Frank includes a tool called Symbiote that let users
explore the system under test via a Web browser. Its focus, however, is specifically on
running a simulator, making it hard to run the test suite on the actual device.
KIF [52] is a tool for automated iOS testing created by Square, the company behind
the Square Credit Card Reader for iOS and Android devices. Like MonkeyTalk and Frank,
KIF allows users to automate iOS GUI testing. However, the tool is based on Objective-C
and is built on top of OCUnit, instead of scripting languages like JavaScript, Cucumber, or
MonkeyTalk Script. This does lead to a better integration of the test suite; it can access
almost anything inside the system under test. However, KIF is more favorable for teams
that consist mainly of developers (as opposed to testers) because programming knowledge
is required to construct the test cases.
1.3.2 DISTRIBUTED TESTING PLATFORMS
There are several projects that implement a distributed testing platform or framework for
software testing and share a similar focus of reducing test execution times for a large
regression test suite. There are a few commercial offerings beginning to appear, but they
are not suitable for use with UI Automation.
CloudTesting [16] is a framework that provides a solution to parallelize the
execution of a test suite over a distributed infrastructure. It has a similar goal to
HadoopUnit, which is to improve the performance of the test execution process by](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-25-320.jpg)
![distributing the execution of a test suite over a networked infrastructure such as cloud or a
private cluster. However, CloudTesting takes a different approach by integrating the
framework with development tools used by software developers or testers with the goal to
encapsulate all of the underlying work from them. The framework was divided into
several different components.
• Configuration: This component takes care of the underlying configuration for
the project, such as the paths, hosts in the cluster, and load balancing
parameters between them, as well as other settings necessary to set up the test
environment.
• Reflection: This component takes care of the test case extraction process that
extracts all the test cases to be executed under the distributed infrastructure.
The result from this component is used by the next component, Distribution.
• Distribution: This component sits in the middle of GridUnit, between the
framework and the distributed infrastructure, and takes care of distributing the
execution of test suites and moving test results back and forth between them.
• Connection: This component provides communications with the distributed
infrastructure. It is used by the Distribution component to manage each test and
its test result back to the client.
• Log: This component records every event that occurred during the test process.
• Main: This component encapsulates other components to make the distributed
testing process transparent to the users.
All of these components are packaged as a separated plugin for the IDE used by
software developers and testers. The current implementation only supports the Eclipse
IDE and the Amazon Web Services infrastructure. Most of the responsibility that these
components provide was already available by Hadoop. To support the other development
IDE such as Xcode, another plugin for the IDE needs to be separately developed and so
could not be used in the context of this work.
GridUnit [17, 18] is a grid-based test execution tool for distributing the execution
of JUnit test cases to a computational grid environment. GridUnit was developed on top of
OurGrid [44], a peer-to-peer computational grid that provide access to an execution
environment to run parallel applications. GridUnit provides functionality similar to
HadoopUnit in that it provides automatic test case distribution, test load distribution, and
test execution control. However, GridUnit require users to develop a test runner to
schedule jobs; with HadoopUnit, the underlying Hadoop platform handles this
automatically. Moreover, in its current form, GridUnit supports test case executions of
JUnit written in Java only, while HadoopUnit was designed to be able to execute test cases
independent of programming language.](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-26-320.jpg)
![CHAPTER 2
Background
This chapter provides background information of three key topics related to this work:
software testing, UI Automation, and Hadoop and HadoopUnit.
2.1 SOFTWARE TESTING
Software testing is a very important activity in the software development lifecycle. It
involves a technical investigation and evaluation of a software product to find information
about its quality [34]. It is one of the main activities in the classic software engineering
life cycle like the waterfall model [54] (Figure 2.1).
Figure 2.1: Software engineering life cycle—waterfall model.
Software testing may occur at any different level throughout a software
development life cycle. According to the ISTQB, there are 4 levels of testing: component
testing (or unit testing), integration testing, system testing, and acceptance testing [8]
(Figure 2.2).](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-27-320.jpg)
![Figure 2.2: Levels of Testing.
Component testing (also known as unit testing) is a level of testing that runs at the
lowest level down to individual function or method of a particular software product. There
are many different types of tools that help testing at this level, for example, a unit test
framework like JUnit for testing Java applications, CppUnit for testing C++ applications,
and OCUnit for testing Objective-C applications. Even simple debugging tools can be
used for component testing [21].
Integration testing studies how two or more components work together by
inspecting the interfaces between different components or between different parts of a
system under test. System testing tests the behavior of a system or a software product by
considering them as a complete system, after integrating every component together, in an
attempt to demonstrate whether or not the system meets its objectives. Acceptance testing
is a level of testing that tests a system or a software product with respect to the users’
needs to determine whether or not it should be accepted.
Software testing raises the level of confidence about the quality of a software
product before it’s released to the users or customers. It can also be used to ensure that
software works correctly after being migrated to a new operating environment. Alessandro
and Filippo illustrated the importance of software testing in this context, where testing can
used to find potential divergences between the system under migration and the newly
migrated system by performing regression testing [40]. They emphasized the importance
of creating a system-level test suite used to test the newly migrated system on every
iteration of migration to confirm the correctness of the system and the equivalence with
the previous version of the system. System-level testing helps reassure that changes in
detailed implementation of the underlying components, such as code refactoring or even
internal architectural changes, will not affect the overall functionality of the system.
2.1.1 REGRESSION TESTING
IEEE defines regression testing as a process of rerunning the test of a system to ensure](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-28-320.jpg)
![that any modification of the system does not introduce any unintended side effects [28,
37]. It involves executing those test cases that have previously been executed to confirm
that any modifications of the software or the environment have not broken any part of the
application. Regression testing can be performed at any of the four levels of testing.
Regression testing is generally performed whenever a new version of an application
is produced or a new version of environment (e.g., an operating system upgrade) is
released. Performing regression testing helps ensure that a new release of an application
did not introduce any form of regression errors. For instance, a fix or modification to one
component might cause another component to break, or cause a previous bug fix to fail
[32].
Regression testing is particularly important in iterative software development,
where a software project is broken down into several short phases [8](Figure 2.3). In
iterative life cycles, newly introduced functionality will need testing and all existing
functionality will need regression testing, meaning more testing is required on each
subsequent iteration. This leads to larger regression test suites due to the accumulation of
test cases over time. With large test suites, performing regression testing can become
costly and it might not always feasible to run everything due to the excessively long
execution times incurred.
Figure 2.3: Iterative development model.
2.1.2 GUI TESTING
According to Ayman et al., GUI testing is a type of testing that validates the visual
properties of GUI elements and the functionalities accessible through them [30]. GUI
testing is performed on an application to verify that it functions and behaves correctly,
whether or not it does what it was supposed to do, and if it provides good user experience
to users [41].
The GUI is a very important part of an application because it acts as a bridge that
connects the application and its users. Unlike unit testing, which focuses on the
component level (an individual unit inside of an application), GUI testing focuses on the
system level—the user interface from the outside of an application, just as the users would
see.](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-29-320.jpg)
![GUI testing involves walking through every user interface element of an application
and following all workflows through the interface to accomplish a particular task. This
means there can be a lot of test cases to cover in order to test every possible variation that
users could experience. This type of testing can be time-consuming because every
interaction has to pass through the GUI of the application, for the entire test suite, in real
(not simulated or accelerated) time [39].
2.2 UI AUTOMATION
Manual GUI testing of an application involves launching and walking through every
feature that an application provides through its GUI, one by one, until all possible
variations are covered. This can be cumbersome and error-prone if the application is
complicated enough. There will be mistakes and some specific paths or variations that
users could take would be left untested. Nothing can replace manual testing activities by
skilled testers; however, such repetitive tasks can be automated using available automated
testing tools [47].
Apple provides the Xcode IDE for free. It is a powerful suite of tools for developing
iOS and OS X application. On the iOS platform, GUI testing can be done through the use
of the UI Automation framework. UI Automation was built on top of JavaScript to
perform automated user interface testing of an iOS application. It was first introduced in
iOS 4 as one of many instruments, a suite of profiling tools for tracing application’s
behavior and performance related issue at runtime.
By writing test scripts, UI Automation can used to exercise an iOS application
through its GUI as well as simulating user gestures, such as tapping, pinching, swiping, or
flicking. It can also handle unexpected alerts, simulating multi-tasking, handling
orientation changes, and setting the location of the device to specific coordinates.
Instruments creates a report called the trace document that records all the behaviors of the
system under test when performing these interactions, which can be inspected by testers
for misbehaviors [4].
2.2.1 UI AUTOMATION SCRIPT
Apple represents all user interface elements of an application through the Accessibility
framework. UI Automation communicates with a system under test using assistive
technologies such as VoiceOver that reads aloud all the user interface elements and gives
visually impaired users a clue on how to navigate an application. Anything that the
Accessibility framework can see is accessible by UI Automation [47].
User interface elements displayed on a screen of an iOS application are accessible
through the JavaScript API provided by UI Automation. Although the language and the
syntax are the same, there are some differences between the JavaScript library used by the
UI Automation framework and the JavaScript library used by typical web browsers. For
example, there is no concept of document object model (DOM) in the UI Automation
framework and there are some additional APIs provided to facilitate creating a test suite,
such as the “#import directive” and methods to filter array of elements.
Instruments also provides tools that help recording and play back interaction with an](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-30-320.jpg)
![inspect the document for misbehaviors.
Although there is no built-in support for assertions in UI Automation, UI
Automation scripts can be extended by importing an external library like TuneUp JS that
provides a collection of assertion functions to be used to provide extra functionality [63].
The written test script can be used to test both iPhone and iPad application on a simulator
and also can be used to test on an actual device that connected to Xcode [4].
2.2.2 COMMAND-LINE WORKFLOW WITH UI AUTOMATION
Taken together, the Xcode IDE and Instruments provide an integrated user interface for all
development activities from coding to testing. However, in order to automate the testing
process, the test has to be executed through the command-line interface. There are two
important commands related to GUI testing of iOS applications and are used throughout
this work: xcodebuild and instruments. xcodebuild is a command to build an Xcode
project through the command-line terminal. A typical structure of this command is shown
in Figure 2.5. (The backslash at the end of each line is just to break the command into
multiple lines for the readability.)
Figure 2.5: Sample Xcodebuild command.
• The -project option is to specify the project file of the Xcode project.
• The -scheme option is to specify the scheme for Xcode to build a project. A
scheme is a collection of configurations, build setting, and targets that Xcode
uses when building or testing an application. There can be multiple schemes in
an Xcode project but only one can be active at a time (it can be specified with
this option).
• The -configuration option instructs the compiler which build configuration to
use. Xcode’s Profile action builds an app for the Release configuration, so for
testing, Release is used.
• The CONFIGURATION_BUILD_DIR is an Xcode configuration setting to
specify which directory the compiler should put all the build-related files into.
This directory will also be used by the instruments command to perform
testing.
• The TARGETED_DEVICE_FAMILY is another Xcode configuration setting
to specify which device family to target. The value of 1 means the targeted
device family is iPhone or iPod touch while the value of 2 means the targeted
device family is iPad. The value of 1,2 means the app is universal.
• The last build option is just to instruct Xcode to build this project.](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-32-320.jpg)
![After the Xcode project has been built with xcodebuild command, the application is
ready for testing. instruments is a command to start the Instruments tool through the
command-line terminal. A typical structure of the instruments command is shown in
Figure 2.6.
Figure 2.6: Sample of an instruments command.
• The -t option is to tell Instruments which trace document template to use for
this test execution. A trace document tells Instruments to start a profiling tool
for the task, which, in this case, is the UI Automation tool. In the example, the
trace-document template is TEMPLATE.tracetemplate and is stored in the
automation directory.
• The -D option tells Instruments the directory in which it can store all the trace
documents. That directory needs to be created beforehand, otherwise the tool
won’t start. In the example, the directory is automation_results and the trace
document is named TRACE. The trace document is very important as
Instruments stores all information about the testing into the file, including all
log messages and screenshots of the application under test that will give clues
to testers what’s wrong with the application when it fails. When there is a
problem with test execution, this directory is where testers can get all the test-
related information.
• The following line is the app-bundle of the system under test that was
previously built using the xcodebuild command. In the example, the app-
bundle is named SUT.app and is stored in /tmp/PROJECT_DIRECTORY.
• The -e UIARESULTSPATH is to specify an environment variable to tell UI
Automation the location to write an XML copy of the automation trace log
into.
• The -e UIASCRIPT is to specify an environment variable to tell UI
Automation the path to the test script to be used for testing.
The basic structure of these commands indicates what information testers need for
each test execution and that HadoopUnit needs to track.
2.2.3 RAKE
Rake is a Ruby build tool with capabilities similar to Make [64]. Rake works on Mac OS
X without requiring any additional programs to be installed. With Rake, all complexity of
building an Xcode project with xcodebuild and running the test with Instruments
commands can be hidden in a very simple Rake task. There also are some useful features
of Rake that can be added into the process. Those features include:](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-33-320.jpg)
![specified by using —rakefile option as seen in Figure 2.9.
Figure 2.9: Sample of a command to invoke a Rake task with -–rakefile.
Rake is used throughout this work for many testing scenarios. For instance, to
sequentially execute a set of test cases on a single machine or to test a particular test case
given a path to a UI Automation script in a form of a Rake task.
2.2.4 VIRTUALIZATION
Virtualization is the process of emulating real IT resources with virtual IT resources.
Virtual machines created from the process act exactly like actual IT resources with their
own guest operating systems, which are independent of the host operating system that
created them. Virtual machines run through the virtualization (hypervisor) software, which
in turn run on a physical host, virtual machines can be easily copies and moved to another
virtualization host as needed [20].
There are some limitations on using the normal execution environment to execute
UI Automation test cases. First of all, only one test script can be run at a time. Therefore,
all test cases in a test suite have to be run sequentially in this environment. It is also not
possible to have multiple instances of the simulators running on a single machine, or to
have multiple instances of the debuggers to run the tests on multiple devices attached to
the machine. This means that this execution process could not be sped up simply by
having multiple separated processes to execute test cases in parallel on a single machine.
Many practitioners suggested the use of multiple virtual machines running
simultaneously on a single machine to solve the problem that only one instance of UI
Automation debugger and simulator can be run at a time. Having multiple virtual
machines run on a single machine means that there are multiple debuggers to run the tests.
With limited resources, virtualization technologies can increase the efficiency and better
utilization of the available resources in a cost effective manner. However, due to a strict
licensing policies stipulated in Apple’s software license agreement under the Section
2B(iii) [3], only two additional virtual machines of Mac OS X could be run on a single
Mac machine. This restrictive license limits how far hardware resources could be utilized
under this environment.
Although using virtualization technologies increase the number of debuggers and
simulators to run the test, this solution still requires manual test distribution, meaning test
cases needs to be manually selected and distributed to those virtual machines and
manually executed. A good system should not require its users to be responsible for test
case distribution. This also could slow down the process if the size of the cluster becomes
large enough [36]. Nevertheless, this work-around suggests a distributed execution
environment where there can be multiple instances of debuggers running in a cluster of
machines, while test cases also get automated distribution. Virtualization technologies
were used to set up such a cluster in various configurations for the experiments described
in Chapter 4.](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-35-320.jpg)
![2.3 HADOOP AND HadoopUnit
This section provides a brief overview of Hadoop, including the Hadoop Distributed File
System (HDFS) and the Map Reduce (MR) programming model, and the HadoopUnit
distributed test execution environment that is built upon the Hadoop platform.
2.3.1 HADOOP
Hadoop is a collection of open source tools, libraries, and methodologies designed to run
on commodity hardware or in the cloud that collectively serve as a scalable platform for
big data analysis [56]. Hadoop consists of two core components: the Hadoop Distributed
File System and MapReduce. A set of machines running these two components is known
as a Hadoop cluster. An individual machine in a Hadoop cluster is known as a node and a
cluster can consist of just one node or as many as a thousand of these nodes.
HDFS
Data in a Hadoop cluster is stored in the Hadoop Distributed File System (HDFS). In
HDFS, data files are split into blocks, usually of 64 MB or 128 MB each, which are
significantly larger than a conventional file system that is usually has a block size of 64
KB. These blocks are distributed across many different machines in the Hadoop cluster.
Hadoop was designed with an assumption that hardware will fail, so some of the blocks
are replicated on other machines, meaning that there are more than one machine storing
the same copy of these blocks. In case of hardware failure, the same block of data could
still be extracted from the other nodes in the cluster.
The node in the cluster that stores these blocks of data files is called DataNode.
There is a single master node in the cluster called the NameNode that controls and keeps
track of the locations of all these data blocks stored in different DataNode in the cluster as
well as which blocks compose a file stored in the HDFS. There is also one or more
Secondary NameNodes that act as checkpoints for the NameNode. In the case where the
NameNode fails, the NameNode can be restarted using a backup snapshot stored in the
Secondary NameNode [65] (Figure 2.10).](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-36-320.jpg)
![Figure 2.10: HDFS architecture.
MapReduce
Hadoop provides big data processing capability through the MapReduce (MR)
programming model. The MapReduce programming model was built upon the concept
that a computation is applied (“mapped”) over a large number of records distributed all
over the cluster to generate partial results that, in turn, are aggregated (“reduced”) to
produce the final solution [14]. The MapReduce programming model hides underlying
execution details from users and provides automatic parallelization and distribution where
developers can concentrate on writing data processing functions in a form of MapReduce
jobs.
A MapReduce job consists of two functions: a Map function and a Reduce function.
A Map function processes a given split of data derived from a block of data in a form of
key/value pairs and produces intermediary output, also as a set of key/value pairs. The
shuffle and sort works in the background, based on the key from the intermediary output,
and feeds the output as an input to a Reduce function. A Reduce function aggregates the
values of the processed intermediary output from the Map functions based on the key part
of the given key/value pairs and provides the ultimate results of the MapReduce job.
Figure 2.11 illustrates how MapReduce works.](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-37-320.jpg)
![Figure 2.11: Overview of how MapReduce works.
MapReduce jobs are controlled by a software process running on the master
NameNode called the JobTracker. Clients submit a MapReduce job through the
JobTracker, which the Job-Tracker then assigns Map and Reduce tasks to, using the other
DataNodes that stored a block of data to be processed in the cluster. These DataNodes
each run a software process called the Task-Tracker. The TaskTracker receives Map or
Reduce tasks from the JobTracker and instantiates the given task on the node and report
progress back to the JobTracker. In case of task failures or in case when a particular task
take unusually long, the tasks are restarted on other available nodes in the cluster
automatically in the background by the JobTracker. Under speculative execution, if there
is a free node available in the cluster, a redundant task will be assigned to it and the result
will be collected from the node that finish executing the task first.
2.3.2 HadoopUnit
HadoopUnit is a distributed execution environment proposed by Parveen and Tilley [46]
that is tailored specifically for executing JUnit test cases concurrently in the cloud.
HadoopUnit was originally developed to address the need of mitigating the lengthy test
execution of the Hadoop production code by using the Hadoop platform itself, because the
method of test execution at that time took very long times to run and could not provide
timely feedbacks to the developers. Hadoop was created using Java and its test cases are in
the form of JUnit, hence HadoopUnit was originally created specifically for executing
JUnit test cases [60].
HadoopUnit Architecture
HadoopUnit was built upon Hadoop with an adapted concept that a test suite is composed
of a large set of test cases. Test cases are distributed all over machines in a cluster and the
test execution is considered computing or processing upon them. The execution of test
cases generates partial test results, which are part of the test suite. The final result is](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-38-320.jpg)
![Figure 2.12: Overall architecture of HadoopUnit.
Testing with HadoopUnit
Testing with HadoopUnit begins with testers creating a HadoopUnit project and uploading
the test suite to the file system where all the production code (a.k.a. the system under test),
test cases for testing the production code, and dependent libraries were already in place in
the HadoopUnit cluster. The test suite is then extracted and split into multiple sets of test
cases that are distributed to machines in the Hadoop cluster. Each node executes the given
set of test cases and returns back the test result. All the test results of all the execution of
test cases are then being merged and reported back to the tester.
In keeping with HadoopUnit’s goal of reducing the cost of running large regression
test suites, it adopted the retest-all approach to software testing. While selective re-test
techniques such as test case selection (test only what have been changed), test case
prioritization (test the one with higher priority earlier), and test suite reduction (eliminate
test cases from the test suite) can reduce the cost of regression testing, and indeed they can
also be used in conjunction with HadoopUnit, they also reduce the chances that regression
bugs can be revealed. Some set of test cases that could potentially expose those bugs
might be eliminated in the reduction process [23, 50]. The retest-all approach is the
simplest approach that avoids such errors and was selected by HadoopUnit; HadoopUnit
shifts the focus to making the testing faster.
To be able to be executed concurrently under a distributed environment such as
HadoopUnit, a set of test cases must be independent from one another. They cannot be
executed at the same time without violating temporal dependencies if these test cases are
co-dependent. The tests should not affect each other and the order of execution should also
not affect the final test result. This means that the test should be self-contained, isolated,](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-40-320.jpg)
![and fully functional. Although having this requirement in a set of test cases means that it
can take longer to execute each test case, such isolated tests are valuable and provide
higher quality feedback to the testers. This is because the failed test cases are not caused
by any other dependent test cases in the test suite, which would require testers to further
interpret the meaning of each failed test case [60, 7].
Requiring manual test case distribution not only slows down the whole testing
process but also demands a substantial number of tasks be done by the testers themselves.
After finishing the execution, testers also have to gather the results from the different
machines to which they manually distributed the tests. This can be overwhelming,
especially if the cluster size becomes large. HadoopUnit provides automated and
transparent test case distribution, which is an ability that HadoopUnit inherited from
Hadoop: it can push input data into nodes in the cluster automatically as well as to collect
results and report back when finished. Test load distribution also is handled automatically
by the underlying Hadoop platform for load balancing of the cluster. This is because if
some of test cases take longer than another, they are distributed to other available nodes in
the cluster automatically [9]. This approach of HadoopUnit better utilizes the available
resources and prevents bottlenecks for rapid test execution.](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-41-320.jpg)
![CHAPTER 3
Using UI Automation with HadoopUnit
In migrating from a traditional execution environment to a distributed execution
environment like HadoopUnit, careful, up-front planning needs to be taken, especially
when planning to automate the testing process. Simply using a good set of tools does not
guarantee the success of the project. This chapter provides information on UI Automation
test suites, HadoopUnit customization, and using HadoopUnit as a platform for
performing GUI testing of iOS applications.
3.1 UI AUTOMATION TEST SUITES
In order to migrate from a traditional GUI testing environment to the HadoopUnit
distributed execution environment, the System Under Test (SUT) and its test suite must be
prepared so that an execution of different test cases can be performed independently in
parallel in the new environment. It is recommended practice that a test suite be developed
in conjunction with the application so that the test could be run frequently as the
development processes [48]. It is worth noting that an automated GUI testing project
should be treated as a genuine programming project [33].
Once an application is released, it may be used or serviced for years. During the
course of time, an application may inevitably need to be fixed (patches or bug fixes),
changed (improvement or enhancements, or environmental changes such as operating
system upgrade), or extended (adding new functionality to the application) [8]. The tests,
once written, must also be maintained along with the application. Using capture/replay
tools helps testers getting up and run to build a test suite quickly. However, for the test
suite to be maintainable, some considerations on the test case design are needed so that the
test cases will have such qualities. The test suite needs to evolve along with the software
project [12].
The UI of an application will change throughout the life cycle of software
development, especially in agile methodologies where changes in requirements are
welcome, as long as the changes satisfy the users. Test cases need to be designed to be
able to cope with this flexibility so the costs of such modification to the test case are kept
to the minimum. The goal is to reduce the amount of work required to update the test
scripts [39, 12].
3.1.1 TEST CASE DESIGN
Every feature of an application should be defined and written as a function within a test
suite where test cases that need to test these features will need to access these functions. In
other words, instead of having test cases to contain scripts for tapping a particular button
to achieve some particular tasks, a function for such tasks should be created to wrap the
underlying interactions with the UI around inside the function. This practice hides the
low-level implementation of how the feature could be accessed. When the UI of the
feature changes, the function that accesses that feature is the only place that needs to be](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-42-320.jpg)
![updated, instead of every test case that tests the feature. It’s also a good idea to pull out
common scripts and create utility functions that other test cases can use, so when more
functionality is added to the function, such as additional logging capabilities, then all test
cases that use this function gain the additional functionality automatically [31, 35].
Jonathan Penn suggested a novel technique for designing UI Automation scripts for
better maintenance by representing each screen of an application as a screen object with
necessary methods to access GUI elements displayed on the screen [47]. The test cases
that need to have access to the screens can import the script containing the screen object
and perform testing on GUI elements as necessary with the provided methods. This way, if
the user interface of the screen was changed, only the script representing the screen needs
to be updated, instead of every place that needs to access that changed user interface, thus
reducing the ripple effect from UI changes to test scripts.
For HadoopUnit, while the dependency within the same test case is acceptable,
different test cases in a test suite needs to be designed in such a way that they can be
executed independently—meaning the order of execution of test cases should not affect
the test result. Because the test case execution will be executed on different nodes on a
cluster, such dependency can cause problems in test execution.
This characteristic is also called test case contamination: the test case tA should not
contaminate the test case tB no matter which one got executed first [60, 36]. A test case
should set up the necessary data and other information required for testing without
requiring other test cases (self-contained).
There is also another benefit of keeping test cases independent: a test case could be
updated, improved, or refactored without affecting the other test cases [27]. Since each test
case is independent of each other, when a test case fails, testers know exactly which test
case to look for rather than requiring the tester to analyze which other test cases also
depend on the failed test case.
3.1.2 TEST CASE ANALYSIS
Test case analysis is necessary for an application with existing UI Automation test suites
in order to migrate to HadoopUnit environment. For example, the existing test suites
might contain all test cases in one UI Automation script file. Distributing test cases in such
a test suite having only one master UI Automation script file is not possible because it
requires a mechanism to selectively execute test cases inside the test suite. This is due to
the limitation of UI Automation that a parameter cannot be sent to the UI Automation
engine through the command line terminal to selectively execute a specific test case within
a test suite. Such types of test suites require test case analysis to break the test suite down
into several files of test cases in order to prepare the test suites to run on HadoopUnit
distributed environment.
As mentioned in the previous section, the UI Automation test script file representing
each test case should be self-contained and must be independent of one another. It can
contain many tests but it should not require the other test file to work. If there are such
dependencies between them, the tester should consider grouping them together into one](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-43-320.jpg)
![script file.
The result of test case analysis is a set of UI Automation test script files, each
representing a single test case. Each file should be executable with Instruments.
3.2 HadoopUnit CUSTOMIZATION
The original version of HadoopUnit was designed specifically to execute JUnit test cases
for Java applications, hence, it operates under a different environment from that of iOS
applications. Some customization had to be made to set up HadoopUnit to prepare it for
GUI testing of iOS applications. This section describes changes to the operational
environment, gathering test results, and the resultant HadoopUnit architecture.
3.2.1 OPERATIONAL ENVIRONMENT
In order to migrate GUI testing of iOS application from a traditional environment towards
a distributed execution environment like HadoopUnit, a Hadoop cluster needs to be set up
in a way that each node is be able to compile, build, and run iOS applications. This
requires each node in the cluster to have the Xcode, Instruments, iOS simulator, iOS SDK,
as well as the Command line tool for OS X installed on the node. These tools operate
under the Mac OS X operating system—Apple’s proprietary operating system based on
UNIX. This environmental requirement leads to several limitations and considerations that
HadoopUnit has to accommodate.
Only one test case can be run at a time during test execution. This limitation means
that regardless of the number of test cases in a test suite, they have to be executed
sequentially [4]. This is not a problem for a distributed execution environment like
HadoopUnit since there are many different nodes of machines available in the cluster to
execute the test case concurrently. However, HadoopUnit needs to be configured to
execute only one Map function at a time to accommodate this limitation, rather than the
default value of two.
Only one instance of the iOS simulator can be run at a time. This is a limitation of
the iOS simulator, a tool for running the application under test itself. This limitation is
similar to the test case execution limitation, which means that another instance of iOS
simulator cannot be started to run another set of tests on a single machine. In this case, the
number of nodes available in the cluster directly correlates to the number of simulators
available for test case execution.
As described in Section 2.2.4, due to licensing restrictions, only two additional
instances of an OS X virtual machines can be run and they have to run on Apple’s
hardware only [3]. This requirement limits the size of the Hadoop cluster that could be
constructed given available hardware resources. This restrictive requirement also makes it
difficult to move this test execution environment to the cloud. Mac OS X isn’t available as
an instance type on major cloud providers such as Amazon’s EC2 or Microsoft’s Azure.
This means that a Hadoop cluster of Mac OS X machines for testing iOS applications
currently needs to be constructed and used internally only.
HadoopUnit as a platform for distributed test execution needs to account for these](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-44-320.jpg)
![Offensive as are all these forms of speech without meaning, they
are not more so than the hollow language of—strange to say,—some
moral philosophers. Many persons have been so impressed by the
ethical essays of Seneca, in which he sings the praises of poverty,
and denounces in burning language the corruption of Rome and the
extortion in the provinces, that they could account for the excellence
of these writings only on the theory of a Christian influence; and a
report gained credit that the Roman philosopher had met and
conversed with the Apostle Paul. But what are these brilliant moral
discourses? Reading them by the light of the author’s life and
character, we find they are only words. A late German historian tells
us that the same Seneca who could discourse so finely upon the
abstemiousness and contentment of the philosopher, and who, on all
occasions, paraded his contempt for earthly things as nothingness
and vanity, amassed, during the four years of his greatest prosperity
and power, a fortune of three hundred millions of sesterces,—over
fifteen millions of dollars. While writing his treatise on “Poverty,” he
had in his house five hundred citrus tables, tables of veined wood
brought from Mount Atlas, which sometimes cost as much as twenty-
five, and even seventy thousand dollars. The same Seneca, who
denounced extortion with so virtuous anger, built his famous
museum gardens with the gold and the tears of Numidia. The same
Seneca, who preached so much about purity of morals, was openly
accused of adultery with Julia and Agrippina, and led his pupil Nero
into still more shameful practices. He wrote a work upon “Clemency,”
yet had, beyond question, a large part of Nero’s atrocities upon his
conscience. It was he who composed the letter in which Nero
justified before the Senate the murder of his own mother.[13]
Common, however, as are meaningless phrases on the stump and
platform, and even in moral treatises, it is to be feared that they are
hardly less so in the meeting-house, and there they are doubly
offensive, if not unpardonable. It is a striking remark of Coleridge,](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-48-320.jpg)
![that truths, of all others the most awful and interesting, are too often
considered so true that they lose all the power of truth, and lie
bedridden in the dormitory of the soul, side by side with the most
despised and exploded errors. Continual handling wears off the
beauty and significance of words, and it is only by a distinct effort of
the mind that we can restore their full meaning. Gradually the terms
most vital to belief cease to mean what they meant when first used;
the electric life goes out of them; and, for all practical purposes, they
are dead. Hence it is that “the traditional maxims of old experience,
though seldom questioned, have often so little effect on the conduct
of life, because their meaning is never, by most persons, really felt,
until personal experience has brought it home. And thus, also, it is
that so many doctrines of religion, ethics, and even politics, so full of
meaning and reality to first converts, have manifested a tendency to
degenerate rapidly into lifeless dogmas, which tendency all the
efforts of an education expressly and skilfully directed to keeping the
meaning alive are barely found sufficient to counteract.”[14]
There can be little doubt that many a man whose life is thoroughly
selfish cheats himself into the belief that he is pious, because he
parrots with ease the phrases of piety and orthodoxy. Who is not
familiar with scores of such pet phrases and cant terms, which are
repeated at this day apparently without a thought of their meaning?
Who ever attended a missionary meeting without hearing “the
Macedonian cry,” and an account of some “little interest,” and “fields
white for the harvest”? Who is not weary of the ding-dong of “our
Zion” and the solecism of “in our midst”; and who does not long for a
verbal millennium when Christians shall no longer “feel to take” and
“grant to give”? “How much I regret,” says Coleridge, “that so many
religious persons of the present day think it necessary to adopt a
certain cant of manner and phraseology as a token to each other!
They must ‘improve’ this and that text, and they must do so and so
in a ‘prayerful’ way; and so on. A young lady urged upon me, the](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-49-320.jpg)
![better kept till it is wanted. Loud utterances of virtuous indignation
against evil from the platform, or in the drawing-room, do not
characterize the spiritual giant; so much indignation as is expressed
has found vent; it is wasted; is taken away from the work of coping
with evil; the man has so much less left. And hence he who restrains
that love of talk lays up a fund of spiritual strength.”[15]
“Prune thou thy words, the thoughts control
That o’er thee swell and throng;
They will condense within thy soul,
And change to purpose strong.
But he who lets his feelings run
In soft luxurious flow,
Shrinks when hard service must be done,
And faints at every woe.
Faith’s meanest deed more favor bears,
Where hearts and wills are weigh’d.
Than brightest transports, choicest prayers,
Which bloom their hour and fade.”[16]
It is said that Pambos, an illiterate saint of the middle ages, being
unable to read, came to some one to be taught a psalm. Having
learned the simple verse, “I said, I will take heed to my ways, that I
offend not with my tongue,” he went away, saying that was enough if
it was practically acquired. When asked six months, and again many
years after, why he did not come to learn another verse, he
answered that he had never been able truly to master this. A man
may have a heart overflowing with love and sympathy, even though
he is not in the habit of exhibiting on his cards “J. Good Soul,
Philanthropist,” and was never known to unfold his cambric
handkerchief, with the words, “Let us weep.” On the other hand,
nothing is easier than to use a set phraseology without attaching to it
any clear and definite meaning,—to cheat one’s self with the
semblance of thought or feeling, when no thought or feeling exists.
It has been truly said that when good men who have no deep](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-52-320.jpg)
![religious fervor use fervent language, which they have caught from
others, or which was the natural expression of what they felt in other
and better years,—above all, when they employ on mean and trivial
occasions expressions which have been forged in the fires of
affliction and hammered out in the shock of conflict,—they cannot
easily imagine what a disastrous impression they produce on keen
and discriminating minds. The cheat is at once detected, and the
hasty inference is drawn that all expressions of religious earnestness
are affected and artificial. The honest and irrepressible utterance of
strong conviction and deep emotion commands respect; but intense
words should never be used when the religious life is not intense.
“Costing little, words are given prodigally, and sacrificial acts must
toil for years to cover the space which a single fervid promise has
stretched itself over. No wonder that the slow acts are superseded by
the available words, the weighty bullion by the current paper money.
If I have conveyed all I feel by language, I am tempted to fancy, by
the relief experienced, that feeling has attained its end and realized
itself. Farewell, then, to the toil of the ‘daily sacrifice!’ Devotion has
found for itself a vent in words.”[17]
Art, as well as literature, politics, and religion, has its cant, which is
as offensive as any of its other forms. When Rossini was asked why
he had ceased attending the opera in Paris, he replied, “I am
embarrassed at listening to music with Frenchmen. In Italy or
Germany, I am sitting quietly in the pit, and on each side of me is a
man shabbily dressed, but who feels the music as I do; in Paris I
have on each side of me a fine gentleman in straw-colored gloves,
who explains to me all I feel, but who feels nothing. All he says is
very clever, indeed, and it is often very true; but it takes the gloss off
my own impression,—if I have any.”](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-53-320.jpg)
![FOOTNOTES:
[13] Ulhorn’s “Conflict of Christianity with Heathenism;” pp. 93,
94.
[14] Mill’s “Logic.”
[15] Sermons, by Rev. F. W. Robertson.
[16] Professor J. H. Newman.
[17] “Life and Letters of F. W. Robertson.”](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-54-320.jpg)
![the heads of the jury unimportant matter, I drive out matter more
important that I had previously lodged there.’”
Joubert, a French author, cultivated verbal economy to such an
extreme that he tried almost to do without words. “If there is a man
on earth,” said he, “tormented by the cursed desire to get a whole
book into a page, a whole page into a phrase, and this phrase into
one word,—that man is myself.” The ambition of many American
speakers, and not a few writers, is apparently the reverse of this. We
do not seem to know that in many cases, as Hesiod says, a half is
more than the whole; and that a speech or a treatise hammered out
painfully in every part is often of less value than a few bright links,
suggestive of the entire chain of thought. Who wants to swallow a
whole ox, in order to get at the tenderloin?
Prolixity, it has been well said, is more offensive now than it once
was, because men think more rapidly. They are not more thoughtful
than their ancestors, but they are more vivid, direct, and animated in
their thinking. They are more impatient, therefore, of long-
windedness, of a loose arrangement, and of a heavy, dragging
movement in the presentation of truth. “A century ago men would
listen to speeches and sermons,—to divisions and subdivisions,—that
now would be regarded as utterly intolerable. As the human body is
whisked through space at the rate of a mile a minute, so the human
mind travels with an equally accelerated pace. Mental operations are
on straight lines, and are far more rapid than they once were. The
public audience now craves a short method, a distinct, sharp
statement, and a rapid and accelerating movement, upon the part of
its teachers.”[18] It is, in short, an age of steam and electricity that
we live in, not of slow coaches; an age of locomotives, electric
telegraphs, and phonography; and hence it is the cream of a
speaker’s thoughts that men want,—the wheat, and not the chaff,—
the kernel, and not the shell,—the strong, pungent essence, and not
the thin, diluted mixture. The model discourse to-day is that which](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-61-320.jpg)
![gives, not all that can be said, even well said, on a subject, but the
very apices rerum, the tops and sums of things reduced to their
simplest expression,—the drop of oil extracted from thousands of
roses, and condensing all their odors,—the healing power of a
hundred weight of bark in a few grains of quinine.
“Certainly the greatest and wisest conceptions that ever issued
from the mind of man,” says South, “have been couched under, and
delivered in, a few close, home, and significant words.... Was not the
work of all the six days [of creation] transacted in so many words?...
Heaven, and earth, and all the host of both, as it were, dropped from
God’s mouth, and nature itself was but the product of a word.... The
seven wise men of Greece, so famous for their wisdom all the world
over, acquired all that fame, each of them by a single sentence
consisting of two or three words. And γνῶθι σεαυτὸν still lives and
flourishes in the mouths of all, while many vast volumes are extinct,
and sunk into dust and utter oblivion.”
Akin to the prolixity of style which weakens so many speeches, is
the habitual exaggeration of language which deforms both our public
and our private discourse. The most unmanageable of all parts of
speech, with many persons, is the adjective. Voltaire has justly said
that the adjectives are often the greatest enemies of the
substantives, though they may agree in gender, number, and case.
Generally the weakness of a composition is just in proportion to the
frequency with which this class of words is introduced. As in gunnery
the force of the discharge is proportioned, not to the amount of
powder that can be used, but to the amount that can be thoroughly
ignited, so it is not the multitude of words, but the exact number
fired by the thought, that gives energy to expression. There are
some writers and speakers who seem to have forgotten that there
are three degrees of comparison. The only adjectives they ever use
are the superlative, and even these are raised to the third power.
With them there is no gradation, no lights and shadows. Every hill is](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-62-320.jpg)
![adjectives, that he “drove a substantive-and-six” whenever he spoke
in public, and that he would be as pathetic as the grand lamentations
in “Samson Agonistes” on the obstruction of fish-ways, and rise to
the cathedral music of the universe on the right to manufacture
India-rubber suspenders. When Chief-Justice Shaw, before whom he
had often pleaded, heard that there was a new edition of
“Worcester’s Dictionary,” containing two thousand five hundred new
words, he exclaimed, “For heaven’s sake, don’t let Choate get hold of
it!”[19]
Even scientific writers, who might be expected to aim at some
exactness, often caricature truth with equal grossness, describing
microscopic things by colossal metaphors. Thus a French naturalist
represents the blood of a louse as “rushing through his veins like a
torrent!” Even in treating on this very subject of exaggeration, a
writer in an English periodical, after rebuking sharply this American
fault, himself outrages truth by declaring that “he would walk fifty
miles on foot to see the man that never caricatures the subject on
which he speaks!” To a critic who thus fails to reck his own rede, one
may say with Sir Thomas Browne: “Thou who so hotly disclaimest
the devil, be not thyself guilty of diabolism.”
Seriously, when shall we have done with this habit of amplification
and exaggeration,—of blowing up molehills into Himalayas and
Chimborazos? Can anything be more obvious than the dangers of
such a practice? Is it not evident that by applying super-superlatives
to things petty or commonplace, we must exhaust our vocabulary, so
that, when a really great thing is to be described, we shall be
bankrupt of adjectives? It is true there is no more unpardonable sin
than dulness; but, to avoid being drowsy, it is not necessary that our
“good Homers” should be always electrifying us with a savage
intensity of expression. There is nothing of which a reader tires so
soon as of a continual blaze of brilliant periods,—a style in which a
“qu’il mourut” and a “let there be light” are crowded into every line.](https://image.slidesharecdn.com/137822-250318182100-d6a2b6fa/85/Testing-iOS-Apps-with-HadoopUnit-3rd-Edition-Scott-Tilley-Krissada-Dechokul-65-320.jpg)
Testing iOS Apps with HadoopUnit 3rd Edition Scott Tilley Krissada Dechokul
- 1. Read Anytime Anywhere Easy Ebook Downloads at ebookmeta.com Testing iOS Apps with HadoopUnit 3rd Edition Scott Tilley Krissada Dechokul https://ebookmeta.com/product/testing-ios-apps-with- hadoopunit-3rd-edition-scott-tilley-krissada-dechokul/ OR CLICK HERE DOWLOAD EBOOK Visit and Get More Ebook Downloads Instantly at https://ebookmeta.com
- 2. Recommended digital products (PDF, EPUB, MOBI) that you can download immediately if you are interested. iOS 10 SDK Development Creating iPhone and iPad Apps with Swift 1st Edition Chris Adamson Janie Clayton https://ebookmeta.com/product/ios-10-sdk-development-creating-iphone- and-ipad-apps-with-swift-1st-edition-chris-adamson-janie-clayton/ ebookmeta.com Web Development with Clojure Build Bulletproof Web Apps with Less Code 3rd Edition Dmitri Sotnikov Scot Brown https://ebookmeta.com/product/web-development-with-clojure-build- bulletproof-web-apps-with-less-code-3rd-edition-dmitri-sotnikov-scot- brown/ ebookmeta.com iOS 15 Tricks and Tips 3rd Edition 2022 https://ebookmeta.com/product/ios-15-tricks-and-tips-3rd-edition-2022/ ebookmeta.com Love That Story 1st Edition Jonathan Van Ness https://ebookmeta.com/product/love-that-story-1st-edition-jonathan- van-ness/ ebookmeta.com
- 3. The Value of Labor: The Science of Commodification in Hungary, 1920-1956 1st Edition Martha Lampland https://ebookmeta.com/product/the-value-of-labor-the-science-of- commodification-in-hungary-1920-1956-1st-edition-martha-lampland/ ebookmeta.com 81st Conference on Glass Problems First Edition. Edition The) Acers (American Ceramics Society https://ebookmeta.com/product/81st-conference-on-glass-problems-first- edition-edition-the-acers-american-ceramics-society/ ebookmeta.com A Manager's Guide for Better Decision-Making: Easy to Apply Tools and Techniques 1st Edition Masud https://ebookmeta.com/product/a-managers-guide-for-better-decision- making-easy-to-apply-tools-and-techniques-1st-edition-masud/ ebookmeta.com The Possibility of the Sublime Aesthetic Exchanges 1st Edition Lars Aagaard-Mogensen https://ebookmeta.com/product/the-possibility-of-the-sublime- aesthetic-exchanges-1st-edition-lars-aagaard-mogensen/ ebookmeta.com Mile High Salvation Carolyn Delaney https://ebookmeta.com/product/mile-high-salvation-carolyn-delaney/ ebookmeta.com
- 4. Marina Abramovi■ 2nd Edition Mary Richards https://ebookmeta.com/product/marina-abramovic-2nd-edition-mary- richards/ ebookmeta.com
- 6. Testing iOS Apps with HadoopUnit Rapid Distributed GUI Testing
- 7. Synthesis Lectures on Software Engineering The Synthesis Lectures on Software Engineering publishes 75-150 page publications on all aspects of software design, engineering, and process management. Testing iOS Apps with HadoopUnit: Rapid Distributed GUI Testing Scott Tilley and Krissada Dechokul December 2014 Hard Problems in Software Testing: Solutions Using Testing as a Service (TaaS) Scott Tilley and Brianna Floss August 2014 Model-Driven Software Engineering in Practice Marco Brambilla, Jordi Cabot, Manuel Wimmer September 2012
- 8. Copyright © 2015 by Morgan & Claypool All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quotations in printed reviews, without the prior permission of the publisher. Testing iOS Apps with HadoopUnit: Rapid Distributed GUI Testing Scott Tilley and Krissada Dechokul www.morganclaypool.com ISBN: 9781627056519 print ISBN: 9781627056526 ebook DOI 10.2200/S00614ED1V01Y201411SWE003 A Publication in the Morgan & Claypool Publishers series SYNTHESIS LECTURES ON SOFTWARE ENGINEERING #3 Series ISSN 2328-3319 Print 2328-3327 Electronic
- 9. Testing iOS Apps with HadoopUnit Rapid Distributed GUI Testing Scott Tilley Florida Institute of Technology Krissada Dechokul Suwat Dechokul Part., Ltd. SYNTHESIS LECTURES SOFTWARE ENGINEERING #3
- 10. ABSTRACT Smartphone users have come to expect high-quality apps. This has increased the importance of software testing in mobile software development. Unfortunately, testing apps—particularly the GUI—can be very time-consuming. Exercising every user interface element and verifying transitions between different views of the app under test quickly becomes problematic. For example, execution of iOS GUI test suites using Apple’s UI Automation framework can take an hour or more if the app’s interface is complicated. The longer it takes to run a test, the less frequently the test can be run, which in turn reduces software quality. This book describes how to accelerate the testing process for iOS apps using HadoopUnit, a distributed test execution environment that leverages the parallelism inherent in the Hadoop platform. HadoopUnit was previously used to run unit and system tests in the cloud. It has been modified to perform GUI testing of iOS apps on a small- scale cluster—a modest computing infrastructure available to almost every developer. Experimental results have shown that distributed test execution with HadoopUnit can significantly outperform the test execution on a single machine, even if the size of the cluster used for the execution is as small as two nodes. This means that the approach described in this book could be adopted without a huge investment in IT resources. HadoopUnit is a cost-effective solution for reducing lengthy test execution times of system-level GUI testing of iOS apps. KEYWORDS software testing, iOS, apps, Hadoop, HadoopUnit, cloud computing, cluster
- 11. Contents Foreword Preface Acknowledgments Dedication 1 Introduction 1.1 GUI Testing of iOS Apps 1.2 Rapid Testing with HadoopUnit 1.3 Related Work 1.3.1 GUI Testing Tools 1.3.2 Distributed Testing Platforms 2 Background 2.1 Software Testing 2.1.1 Regression Testing 2.1.2 GUI Testing 2.2 UI Automation 2.2.1 UI Automation Script 2.2.2 Command-Line Workflow with UI Automation 2.2.3 Rake 2.2.4 Virtualization 2.3 Hadoop and HadoopUnit 2.3.1 Hadoop 2.3.2 HadoopUnit 3 Using UI Automation with HadoopUnit 3.1 UI Automation Test Suites 3.1.1 Test Case Design 3.1.2 Test Case Analysis 3.2 HadoopUnit Customization 3.2.1 Operational Environment 3.2.2 Test Results
- 12. 3.2.3 Revised Architecture 3.3 Using HadoopUnit 3.3.1 Test Case List 3.3.2 Rake 3.3.3 Test Execution 4 Rapid GUI Testing of iOS Apps 4.1 Experiments 4.1.1 Experiment I 4.1.2 Experiment II 4.1.3 Experiment III 4.2 Discussion of Results 4.3 Threats to Validity 4.3.1 Test Suites 4.3.2 Hadoop Optimization 4.3.3 Network Issues 5 Summary 5.1 Summary of Results 5.1.1 Research Objectives 5.1.2 Research Contributions 5.2 Future Work 5.3 Concluding Remarks Appendix A Setting up a HadoopUnit Cluster on Mac OS X Appendix B HadoopUnit Source Code for iOS GUI Testing References About the Authors
- 13. Figures Figure 1.1: App store downloads Figure 1.2: Overall architecture of third-party testing tools Figure 2.2: Levels of Testing Figure 2.3: Iterative development model Figure 2.4: Sample UI automation test case Figure 2.5: Sample Xcodebuild command Figure 2.6: Sample of an instruments command Figure 2.7: Sample of a Rake task for executing an instruments test Figure 2.8: Sample of a command to invoke a defined Rake task Figure 2.9: Sample of a command to invoke a Rake task with-–rakefile Figure 3.1: Architecture of HadoopUnit for GUI testing of iOS applications Figure 3.2: Sample of a test case list Figure 3.3: Sample of command to transfer a file to the HDFS Figure 3.4: The Hadoop command to initiate test execution with HadoopUnit Figure 3.5: The Hadoop command to download files from the HDFS Figure 4.1: A sample screenshot of the system under test Figure 4.2: Sample of a test case Figure 4.3: Code for executing test cases sequentially with a Rake task Figure 4.4: Sequential execution time on a single machine Figure 4.5: Concurrent execution time on a 2-nodes cluster Figure 4.6: Concurrent execution time on a 4-node cluster Figure 4.7: Execution time comparison of the three experiments Figure 4.8: Total test execution time approximation equation Figure 4.9: Ideal case for test execution time approximation equation
- 14. Tables Table 4.1: Sequential execution time on a single machine (in seconds) Table 4.2: Concurrent execution time on a 2-node cluster (in seconds) Table 4.3: Concurrent execution time on a 4-node cluster (in seconds) Table 4.4: Performance comparisons of the three experiments (in seconds) Table 4.5: Performance factors over sequential execution
- 15. Foreword Software quality has never been as important as it is today. A tidal wave of new software development is seen in mobile applications, which are quickly becoming ubiquitous. They are used for many purposes, from enjoyable entertainment to safety critical applications used by first responders. At the same time, the bar to entering the mobile application marketplace is very low. Anyone with the desire to create a mobile application can do so with freely available tools in a rather short amount of time. A quick browse through the Apple App Store reveals that many applications fall short of user expectations: there are many apps in the store with three stars or fewer. Creating a mobile application is one thing. Testing it to ensure that it works correctly, is secure, and achieves high usability factors is another thing altogether. First, there is the plethora of mobile device platforms and appliances—each of which has its own configuration and behavior differences. Second, the time required to test a moderately featured application just one time on one platform is substantial. Manual testing can be tedious and automated testing simulating user actions can relieve some of the tedium, but it still requires time to design and perform functional tests. Third, mobile applications undergo constant change. The constant march of change fixes bugs and introduces new features—and new defects. Therefore, testing should include not only testing bug fixes and new features, but also regression testing of the unchanged features. However, this all takes precious time. Finally, consider that many mobile application developers consist of small teams, perhaps just an individual, with limited funds and resources for testing. So, the customers get to be the “testers,” except they use the applications for important real-world tasks like arranging travel, managing finances, tracking severe weather, and navigating roads. The customers don’t consider themselves as testers. They are customers and even though they might not have paid any money for an app, they don’t like to waste their time on defective ones. Also, when mobile applications fail during important tasks, defects are more than a mere inconvenience—they impact lives in a negative way. While developers may perform bug fixes, customers often abandon apps quickly due to a bad experience. It is very easy to delete an app from an iOS device, so developers who wish to have a successful app in the App Store need to understand the detrimental impact of defects on their success. That’s why this book is important for mobile application developers and testers. HadoopUnit offers a solution for testing mobile apps on the iOS platform that is not only free, but reduces test time dramatically. There will still be a need for usability and compatibility testing, but the more troubling defects are those that impact mobile application reliability.
- 16. Scott Tilley and Krissada Dechokul have done an excellent job with this book in describing in detail how to design and implement tests in HadoopUnit. My hope is that everyone involved in developing iOS applications will read this book and create more reliable and robust applications that get five-star reviews. Randall W. Rice CSTE, CSQA, CTAL Founder, Principal Consultant, and Vice-President of Research and Development Rice Consulting Services, Inc.
- 17. Preface The app ecosystem is enormous. We have grown dependent on smartphone apps for almost every aspect of our lives. When the apps don’t perform as expected, the consequences can range from midly irritating to life-threatening. For this reason, testing smartphone apps is very important—particularly the GUI that is the main window into the app’s functionality. Unfortuanately, GUI testing can be a time-consuming process, which leads to fewer tests being run, further exacerbating the app’s quality problems. This book focuses on the specific problem of GUI testing for iOS apps found on Apple’s products such as the iPhone and the iPad. Apple provides developers with a testing framework called UI Automation, but its capabilities are limited in terms of speeding up the testing process. The result is that GUI testing of complex iOS apps can take many hours. The solution proposed here is to leverage the parallelism inherent in the Hadoop distributed platform to provide an environment for concurrent test execution. The approach builds upon an existing system, called HadoopUnit, that was previously used to reduce regression testing time of large JUnit test suites. HadoopUnit has been customized to drive UI Automation test cases in a manner that is easy for developers and testers to adopt, yet provides measurable improvement in test case execition times. WHAT IS UNIQUE ABOUT THIS BOOK? This book represents the continuation of research that began in 2009 on addressing the problem of execution times for large regression test suites. The first of this work were the HadoopUnit distributed exection environment and the SMART-T migration decision framework [60]. The work also led to the creation of a new community of researchers and practicioners interesting in software testing in the cloud (STITC) [59][62]. The STITC project evolved to examine the applicability of testing as a service (TaaS) to hard problems in software testing (HPST) [61][58]. TaaS is a promising new development that offers a service-oriented interface to the testing capabilities provided by an environement like HadoopUnit. The number one problem found in the HPST project was education & training, and it’s currently an open question where TaaS may help alleviate this timeless challenge. GUI testing of iOS apps is a timely update and specific instance of the classic regression testing problem, and one that HadoopUnit is well suited to address. The research reported in this book is unique is its application of an advanced environment such as HadoopUnit for concurrent testing in manner that is accessible to almost all developers and testers—even if they are of modest means. A simple two-node cluster is all that is needed to realize significant testing benefits. WHO SHOULD READ THIS BOOK? Anyone who is involved in GUI testing of iOS apps will find the material presented in this book valuable. This is particularly true for testers, but developers, managers, and even
- 18. end-users can benefit from understanding the challenges faced when using the UI Automation framework and the possible benefits of using the customized HadoopUnit to address these challenges. Modern software engineering—and app development in particular—involves the use of sophisticated IDEs and integrated coding platforms. Many of these tools are moving to the cloud. An understanding of how Hadoop can be used in the domain of GUI testing provides valuable insight into the power of the MapReduce programmign paradigm. OUTLINE OF THE BOOK Chapter 1 discusses the challenges of GUI testing with UI Automation, outlines prior results for rapid testing using HadoopUnit, and outlines related work in the areas of GUI testing tools and distributed testing platforms. Chapter 2 provides background information on the challenges of software testing in general, and GUI testing of iOS apps with UI Automation in particular, and summarizes the Hadoop platform and the HadoopUnit distributed test execition environment. Chapter 3 outines how HadoopUnit can be used to drive the UI Automation framework to faciliate parallel test execution. Chapter 4 details three experiments in rapid GUI testing of iOS apps using the customized HadoopUnit. Lastly, Chapter 5 summarizes the main results, objectives, and contributions of this work and outlines possile avenues of further investiation. The book also contains two appendices. Appendix A describes how to set up a HadoopUnit cluster on Mac OS X. Appendix B provides source code samples for HadoopUnit, suitably modified for iOS GUI testing with UI Automation. Scott Tilley Krissada Dechokul Melbourne, FL Bangkok, Thailand October 2014
- 19. Acknowledgments We are indebted to everyone who helped develop HadoopUnit—the platform upon which this research is built: Tauhida Parveen, Eric Bower, colleagues at Yahoo!, collaborators at SAP, and members of the global STITC community. Our thanks to Apple for making the excellent UI Automation framework available for free as part of the Xcode development environment. We appreciate the the invaluable comments privided by the book’s reviewers. Their suggestions helped improve the text. Any remaining errors or omissions are ours alone. We are grateful to the Florida Institute of Technology for supporting this research. Finally, our gratitude to Morgan & Claypool for their guidance and patience in helping us publish the results of our work.
- 20. Dedication To Miel— Scott Tilley To my parents— Krissada Dechokul
- 21. CHAPTER 1 Introduction Mobile applications have become an important part of the software development industry and a driver of the overall technology economy. In June 2014, Apple announced that 75 billion apps had been downloaded from its App Store [53]. As shown in Figure 1.1, Apple is on track for 85 billion downloads by the end of October. Gartner predicts the total number of mobile apps downloaded across all platforms to double by 2016 [22]. Figure 1.1: App store downloads [53]. One of the main factors behind these impressive numbers is the growing use of smartphones and tablets. Smartphones will soon pass personal computers to become the dominant computing platform worldwide [49]. With so many apps in use, quality becomes an important issue. A game crashing is one thing; an app interacting with essential systems is something else altogether. In July 2014, Apple and IBM entered into a strategic partnership to develop business-focused apps for the enterprise [45]. In October 2014, IBM and SAP entered into a strategic partnership wherein IBM will provide the cloud infrastructure for SAP’s enterprise applications, including its growing number of mobile apps [24]. These agreements signal a clear shift in the app ecosystem, away from a purely consumer- oriented world to one that embraces corporate users as well. This shift will only increase the need for high-quality mobile apps. One way that helps ensure quality of mobile applications is through testing. Software testing is one of the main activities in the software engineering life cycle
- 22. (requirements, design, construction, testing, and maintenance). Testing is a very important process that helps assess the functionality and ensure the quality of software before being released by identifying defects and problems [8]. System-level GUI testing, which is a primary focus of this work, is especially important in mobile application testing because it tests an application after all components has been integrated to ensure that the application behaves as intended [47]. On the iOS platform, GUI testing can be done through the use of Apple’s UI Automation framework [4]. It was first introduced with iOS 4 as part of Instruments, a suite of profiling tools for tracing application’s behaviors and performance related issues at runtime. UI Automation allow users to write scripts to exercise the user interface of an iOS application by simulating users’ gestures, such as tapping, pinching, swiping, or flicking. With UI Automation, testers do not have to repeatedly perform the same set of actions on the app GUI every time code is integrated into the baseline. 1.1 GUI TESTING OF IOS APPS At the 2010 WWDC conference Apple made a very interesting statement when they first introduced UI Automation: “Automating User Interface Testing with Instruments or How to Find Bugs While You Sleep” [2]. This statement implied that execution of a GUI test suite could take a considerable amount of time to finish. There are two main reasons for slow test execution: the number of test cases to run and the execution time of each test case. A good GUI test suite covers all functionality of an application accessible through its user interface and exercises every user interface element presented to users on screen. The user interface of a mobile application as well as the variations through which users can interact with the application can grow large in number and the amount of test cases to cover in a test suite increases accordingly [13]. Moreover, the test suite continues to grow over the course of software development process, especially when agile methods are used: more test cases are added each iteration. The testing also needs to account for different versions of operating systems, device orientations, and form factors of mobile devices. The execution speed of GUI testing also can be very slow and time-consuming. This is because testing through the GUI proceeds in real time, at end-user speeds, not processor speeds [39]. This type of testing asserts the behaviors of an application from the outside, just as the users would actually see. This means that the testing tool needs to constantly wait for the rendering of user interface elements and animations, or wait for network delays throughout the test execution. Combining the slow test execution speed with the total number of test cases results in the test process being tediously slow. GUI test execution of a realistic test suite can take an hour or more to execute, and if the user interface of the application is complicated, the time required for a test execution to complete might go beyond the time and resources available for a project. This limits how many times testing can be performed, which in turn affects app quality. To solve the problems presented by slow-running large test suites is more complicated than simply having multiple separated processes executing more test scripts
- 23. on a single machine. One reason is that with UI Automation, only one test script could be run at a time. Moreover, only one instance of iOS simulator could be run at a time [4]. A different solution is needed. 1.2 RAPID TESTING WITH HadoopUnit There are many solutions proposed to solve the test execution time problem. One way is to reduce the number of test cases to run. There are many researchers [36, 23] who propose various approaches to reduce the number of test cases to run. Instead of executing the whole regression test suite, selective re-test techniques attempt to execute only a subset of test cases in a test suite. There are several such techniques, such as test case selection [51], test case prioritization [66], and test suite reduction [25]. However, these kinds of techniques are complicated and can be computationally expensive, as decisions have to be made on which test cases not to run. With such decisions, some of the regression bugs might have been missed with the dropped test cases if the analysis is not flawless. The re- test all approach is more straightforward in that it avoids such errors [23]. Further discussion on regression test selection is provided in Section 2.1.1. Another approach to mitigate the problem is to move to a faster test execution environment. A pre-existing framework, called HadoopUnit [46], was designed to solve this problem. In previous studies, HadoopUnit provided an infrastructure for executing JUnit test cases concurrently in a cluster or in the cloud to speed up the test execution process. The result from the previous studies have shown that using a medium-scale cluster of 150 nodes, the test execution runtime of Hadoop’s JUnit test suite with 230 test cases could be impressively improved for up to approximately 55 times faster [60] by executing a single test per node. However, because not everyone has access to large amount of IT resources, even with a medium cluster as shown from the previous studies, there is also a study of HadoopUnit on a smaller-scale of cluster of 4 nodes and the result have shown that such impressive performance improvement is not always guaranteed. Some modifications had been made to HadoopUnit to further fine-tune its performance by letting each node to execute more than one test concurrently [9]. The structure of the HadoopUnit platform and the problems that it solved by reducing the execution time of JUnit test suite make it a suitable candidate as a platform for system-level GUI testing of iOS applications. In order to migrate testing to a distributed execution environment, several artifacts involved in the testing process of the current test execution environment must be identified. The important artifacts in system- level GUI testing of iOS applications include a set of test cases that composes a test suite, the UI Automation testing tool, a trace document capturing the test results of each test execution, and the operational test environment. These artifacts are studied in this book for feasibility in migrating to a new environment, with HadoopUnit customized accordingly as needed. 1.3 RELATED WORK There are several other GUI testing tools and distributed testing platforms related to the work presented in this book. The GUI testing tools described below offer alternatives to
- 24. Apple’s UI Automation, but come with their own caveats. There are fewer choices when it comes to distributed testing platforms currently available, but that may change if Testing as a Service becomes more prevalent. 1.3.1 GUI TESTING TOOLS The primary reason that UI Automation was selected as a testing tool for GUI testing tool in this book was because it works without requiring software engineers to install any additional software or customize their Xcode projects. However, there are several third- party testing tools available that could be used GUI testing as well. The main reason to consider a third-party testing tool is when the team was already familiar with the tools or language used by the tools. Although they rely on different technologies and implementations, they share similar concepts on how these testing tools work together. As shown in Figure 1.2, there are three common components in these testing tools: an agent, a script or feature, and an IDE. 1. An Agent: A framework or a small server that will be installed to run inside the system under test. The installation needs to be performed on an iOS project manually through the Xcode IDE. 2. A Script or a Feature: The main set of test scripts for the system under test. Programming languages used for each tool will be different based on the underlying technologies used by the testing tool. For example, Objective-C [5], JavaScript [42], CoffeeScript [6], or Cucumber [26]. 3. An IDE: This component is separate piece of software running on a tester machine or remote server that helps record the interactions, execute the scripts, and issue commands that interact with the system under test through the agent component installed inside the application. Figure 1.2: Overall architecture of third-party testing tools. GUITAR [43] is a framework for performing automated GUI testing. The GUITAR framework has several notable features including automated test case generation and execution. It consists of four main components: (1) Ripper for generating a structure
- 25. model of the GUI of the system under test; (2) Graph Converter for converting the structure model into a graph; (3) the Test Case Generator for generating test cases based on the graph; and (4) Replayer for automated executing the generated test cases. GUITAR has a plugin-based architecture meaning that a specific plugins are needed to support different platforms. GUITAR supports multiple platforms, such as Java, Web, and Android. iPhone GUITAR [19] is still under-development to support GUI testing of iOS application. One weakness of GUITAR is that it does not support any manual test cases development (both scripted and captured) and leads instead towards automating the whole process. MonkeyTalk [11] is a free, open-source, object-based recording tool from Gollila Logic that examines the app’s code to understand the underlying interactions. This approach is supposed to be more accurate that image-based recording that detects pixel on the screen. It also results in a more readable and maintainable script when compared to scripts that rely on image recognition technology. MonkeyTalk has its own IDE to construct or record a new test script and playback existing scripts. There are a lot of actions and gestures to select from, including wait time specified in milliseconds. MonkeyTalk uses its own proprietary MonkeyTalk Scripts for constructing test cases while still supporting JavaScript. MonkeyTalk also support command-line test execution through either Ant Runner or Java Runner. Frank [57] is a tool for automated iOS testing by Pete Hodgson. It is described by the tool’s developers as “Selenium for native iOS apps.” Frank uses a combination of Cucumber and JSON commands that are sent to a server running inside the native application. The benefit of using natural language like Cucumber is its readability for non- technical stakeholders in the project. Frank includes a tool called Symbiote that let users explore the system under test via a Web browser. Its focus, however, is specifically on running a simulator, making it hard to run the test suite on the actual device. KIF [52] is a tool for automated iOS testing created by Square, the company behind the Square Credit Card Reader for iOS and Android devices. Like MonkeyTalk and Frank, KIF allows users to automate iOS GUI testing. However, the tool is based on Objective-C and is built on top of OCUnit, instead of scripting languages like JavaScript, Cucumber, or MonkeyTalk Script. This does lead to a better integration of the test suite; it can access almost anything inside the system under test. However, KIF is more favorable for teams that consist mainly of developers (as opposed to testers) because programming knowledge is required to construct the test cases. 1.3.2 DISTRIBUTED TESTING PLATFORMS There are several projects that implement a distributed testing platform or framework for software testing and share a similar focus of reducing test execution times for a large regression test suite. There are a few commercial offerings beginning to appear, but they are not suitable for use with UI Automation. CloudTesting [16] is a framework that provides a solution to parallelize the execution of a test suite over a distributed infrastructure. It has a similar goal to HadoopUnit, which is to improve the performance of the test execution process by
- 26. distributing the execution of a test suite over a networked infrastructure such as cloud or a private cluster. However, CloudTesting takes a different approach by integrating the framework with development tools used by software developers or testers with the goal to encapsulate all of the underlying work from them. The framework was divided into several different components. • Configuration: This component takes care of the underlying configuration for the project, such as the paths, hosts in the cluster, and load balancing parameters between them, as well as other settings necessary to set up the test environment. • Reflection: This component takes care of the test case extraction process that extracts all the test cases to be executed under the distributed infrastructure. The result from this component is used by the next component, Distribution. • Distribution: This component sits in the middle of GridUnit, between the framework and the distributed infrastructure, and takes care of distributing the execution of test suites and moving test results back and forth between them. • Connection: This component provides communications with the distributed infrastructure. It is used by the Distribution component to manage each test and its test result back to the client. • Log: This component records every event that occurred during the test process. • Main: This component encapsulates other components to make the distributed testing process transparent to the users. All of these components are packaged as a separated plugin for the IDE used by software developers and testers. The current implementation only supports the Eclipse IDE and the Amazon Web Services infrastructure. Most of the responsibility that these components provide was already available by Hadoop. To support the other development IDE such as Xcode, another plugin for the IDE needs to be separately developed and so could not be used in the context of this work. GridUnit [17, 18] is a grid-based test execution tool for distributing the execution of JUnit test cases to a computational grid environment. GridUnit was developed on top of OurGrid [44], a peer-to-peer computational grid that provide access to an execution environment to run parallel applications. GridUnit provides functionality similar to HadoopUnit in that it provides automatic test case distribution, test load distribution, and test execution control. However, GridUnit require users to develop a test runner to schedule jobs; with HadoopUnit, the underlying Hadoop platform handles this automatically. Moreover, in its current form, GridUnit supports test case executions of JUnit written in Java only, while HadoopUnit was designed to be able to execute test cases independent of programming language.
- 27. CHAPTER 2 Background This chapter provides background information of three key topics related to this work: software testing, UI Automation, and Hadoop and HadoopUnit. 2.1 SOFTWARE TESTING Software testing is a very important activity in the software development lifecycle. It involves a technical investigation and evaluation of a software product to find information about its quality [34]. It is one of the main activities in the classic software engineering life cycle like the waterfall model [54] (Figure 2.1). Figure 2.1: Software engineering life cycle—waterfall model. Software testing may occur at any different level throughout a software development life cycle. According to the ISTQB, there are 4 levels of testing: component testing (or unit testing), integration testing, system testing, and acceptance testing [8] (Figure 2.2).
- 28. Figure 2.2: Levels of Testing. Component testing (also known as unit testing) is a level of testing that runs at the lowest level down to individual function or method of a particular software product. There are many different types of tools that help testing at this level, for example, a unit test framework like JUnit for testing Java applications, CppUnit for testing C++ applications, and OCUnit for testing Objective-C applications. Even simple debugging tools can be used for component testing [21]. Integration testing studies how two or more components work together by inspecting the interfaces between different components or between different parts of a system under test. System testing tests the behavior of a system or a software product by considering them as a complete system, after integrating every component together, in an attempt to demonstrate whether or not the system meets its objectives. Acceptance testing is a level of testing that tests a system or a software product with respect to the users’ needs to determine whether or not it should be accepted. Software testing raises the level of confidence about the quality of a software product before it’s released to the users or customers. It can also be used to ensure that software works correctly after being migrated to a new operating environment. Alessandro and Filippo illustrated the importance of software testing in this context, where testing can used to find potential divergences between the system under migration and the newly migrated system by performing regression testing [40]. They emphasized the importance of creating a system-level test suite used to test the newly migrated system on every iteration of migration to confirm the correctness of the system and the equivalence with the previous version of the system. System-level testing helps reassure that changes in detailed implementation of the underlying components, such as code refactoring or even internal architectural changes, will not affect the overall functionality of the system. 2.1.1 REGRESSION TESTING IEEE defines regression testing as a process of rerunning the test of a system to ensure
- 29. that any modification of the system does not introduce any unintended side effects [28, 37]. It involves executing those test cases that have previously been executed to confirm that any modifications of the software or the environment have not broken any part of the application. Regression testing can be performed at any of the four levels of testing. Regression testing is generally performed whenever a new version of an application is produced or a new version of environment (e.g., an operating system upgrade) is released. Performing regression testing helps ensure that a new release of an application did not introduce any form of regression errors. For instance, a fix or modification to one component might cause another component to break, or cause a previous bug fix to fail [32]. Regression testing is particularly important in iterative software development, where a software project is broken down into several short phases [8](Figure 2.3). In iterative life cycles, newly introduced functionality will need testing and all existing functionality will need regression testing, meaning more testing is required on each subsequent iteration. This leads to larger regression test suites due to the accumulation of test cases over time. With large test suites, performing regression testing can become costly and it might not always feasible to run everything due to the excessively long execution times incurred. Figure 2.3: Iterative development model. 2.1.2 GUI TESTING According to Ayman et al., GUI testing is a type of testing that validates the visual properties of GUI elements and the functionalities accessible through them [30]. GUI testing is performed on an application to verify that it functions and behaves correctly, whether or not it does what it was supposed to do, and if it provides good user experience to users [41]. The GUI is a very important part of an application because it acts as a bridge that connects the application and its users. Unlike unit testing, which focuses on the component level (an individual unit inside of an application), GUI testing focuses on the system level—the user interface from the outside of an application, just as the users would see.
- 30. GUI testing involves walking through every user interface element of an application and following all workflows through the interface to accomplish a particular task. This means there can be a lot of test cases to cover in order to test every possible variation that users could experience. This type of testing can be time-consuming because every interaction has to pass through the GUI of the application, for the entire test suite, in real (not simulated or accelerated) time [39]. 2.2 UI AUTOMATION Manual GUI testing of an application involves launching and walking through every feature that an application provides through its GUI, one by one, until all possible variations are covered. This can be cumbersome and error-prone if the application is complicated enough. There will be mistakes and some specific paths or variations that users could take would be left untested. Nothing can replace manual testing activities by skilled testers; however, such repetitive tasks can be automated using available automated testing tools [47]. Apple provides the Xcode IDE for free. It is a powerful suite of tools for developing iOS and OS X application. On the iOS platform, GUI testing can be done through the use of the UI Automation framework. UI Automation was built on top of JavaScript to perform automated user interface testing of an iOS application. It was first introduced in iOS 4 as one of many instruments, a suite of profiling tools for tracing application’s behavior and performance related issue at runtime. By writing test scripts, UI Automation can used to exercise an iOS application through its GUI as well as simulating user gestures, such as tapping, pinching, swiping, or flicking. It can also handle unexpected alerts, simulating multi-tasking, handling orientation changes, and setting the location of the device to specific coordinates. Instruments creates a report called the trace document that records all the behaviors of the system under test when performing these interactions, which can be inspected by testers for misbehaviors [4]. 2.2.1 UI AUTOMATION SCRIPT Apple represents all user interface elements of an application through the Accessibility framework. UI Automation communicates with a system under test using assistive technologies such as VoiceOver that reads aloud all the user interface elements and gives visually impaired users a clue on how to navigate an application. Anything that the Accessibility framework can see is accessible by UI Automation [47]. User interface elements displayed on a screen of an iOS application are accessible through the JavaScript API provided by UI Automation. Although the language and the syntax are the same, there are some differences between the JavaScript library used by the UI Automation framework and the JavaScript library used by typical web browsers. For example, there is no concept of document object model (DOM) in the UI Automation framework and there are some additional APIs provided to facilitate creating a test suite, such as the “#import directive” and methods to filter array of elements. Instruments also provides tools that help recording and play back interaction with an
- 31. application, while also generating a line of automation script that perform the recorded actions automatically. This feature greatly helps testers building their test cases faster. However, it doesn’t record timing of the recorded actions, although delays could be added manually if timing is desired. Since it is built on top of JavaScript, UI Automation gives testers flexibility in writing test cases to test their applications. The test script can be composed of a collection of statements to assert the behavior of an application under test or it can be grouped together into a simple JavaScript function and get invoked like a normal JavaScript function. An example of a simple test case can be seen in Figure 2.4. The test case asserts that a table cell was actually removed from the table after simulating various actions and gestures. Figure 2.4: Sample UI automation test case. UI Automation provides various means to report whether a test case passed or failed by using simple logging functions, as shown in Figure 2.4. There are two main types of logging functions: logging with Test Status and logging with Severity Levels. Logging with Test Status functions are those functions that indicate that a test has completed successfully, unsuccessfully, or terminated abnormally. Examples of the functions of this type are logStart, logPass, logFail, or logIssue. Logging with Severity Levels function allows the tester to write debugging message with a specific level of severity. Examples of the function include logMessage, logError, logWarning, and logDebug. Instruments also automatically grabs a screenshot of the current state of the system under test whenever a test case fails, so testers could see what the screen looked like to help determine what went wrong when the test case failed. A function to capture the screenshot can be invoked manually at any point in the test case. UI Automation records log messages and stores these screenshots in a trace document so that the testers can later
- 32. inspect the document for misbehaviors. Although there is no built-in support for assertions in UI Automation, UI Automation scripts can be extended by importing an external library like TuneUp JS that provides a collection of assertion functions to be used to provide extra functionality [63]. The written test script can be used to test both iPhone and iPad application on a simulator and also can be used to test on an actual device that connected to Xcode [4]. 2.2.2 COMMAND-LINE WORKFLOW WITH UI AUTOMATION Taken together, the Xcode IDE and Instruments provide an integrated user interface for all development activities from coding to testing. However, in order to automate the testing process, the test has to be executed through the command-line interface. There are two important commands related to GUI testing of iOS applications and are used throughout this work: xcodebuild and instruments. xcodebuild is a command to build an Xcode project through the command-line terminal. A typical structure of this command is shown in Figure 2.5. (The backslash at the end of each line is just to break the command into multiple lines for the readability.) Figure 2.5: Sample Xcodebuild command. • The -project option is to specify the project file of the Xcode project. • The -scheme option is to specify the scheme for Xcode to build a project. A scheme is a collection of configurations, build setting, and targets that Xcode uses when building or testing an application. There can be multiple schemes in an Xcode project but only one can be active at a time (it can be specified with this option). • The -configuration option instructs the compiler which build configuration to use. Xcode’s Profile action builds an app for the Release configuration, so for testing, Release is used. • The CONFIGURATION_BUILD_DIR is an Xcode configuration setting to specify which directory the compiler should put all the build-related files into. This directory will also be used by the instruments command to perform testing. • The TARGETED_DEVICE_FAMILY is another Xcode configuration setting to specify which device family to target. The value of 1 means the targeted device family is iPhone or iPod touch while the value of 2 means the targeted device family is iPad. The value of 1,2 means the app is universal. • The last build option is just to instruct Xcode to build this project.
- 33. After the Xcode project has been built with xcodebuild command, the application is ready for testing. instruments is a command to start the Instruments tool through the command-line terminal. A typical structure of the instruments command is shown in Figure 2.6. Figure 2.6: Sample of an instruments command. • The -t option is to tell Instruments which trace document template to use for this test execution. A trace document tells Instruments to start a profiling tool for the task, which, in this case, is the UI Automation tool. In the example, the trace-document template is TEMPLATE.tracetemplate and is stored in the automation directory. • The -D option tells Instruments the directory in which it can store all the trace documents. That directory needs to be created beforehand, otherwise the tool won’t start. In the example, the directory is automation_results and the trace document is named TRACE. The trace document is very important as Instruments stores all information about the testing into the file, including all log messages and screenshots of the application under test that will give clues to testers what’s wrong with the application when it fails. When there is a problem with test execution, this directory is where testers can get all the test- related information. • The following line is the app-bundle of the system under test that was previously built using the xcodebuild command. In the example, the app- bundle is named SUT.app and is stored in /tmp/PROJECT_DIRECTORY. • The -e UIARESULTSPATH is to specify an environment variable to tell UI Automation the location to write an XML copy of the automation trace log into. • The -e UIASCRIPT is to specify an environment variable to tell UI Automation the path to the test script to be used for testing. The basic structure of these commands indicates what information testers need for each test execution and that HadoopUnit needs to track. 2.2.3 RAKE Rake is a Ruby build tool with capabilities similar to Make [64]. Rake works on Mac OS X without requiring any additional programs to be installed. With Rake, all complexity of building an Xcode project with xcodebuild and running the test with Instruments commands can be hidden in a very simple Rake task. There also are some useful features of Rake that can be added into the process. Those features include:
- 34. • creating or removing a directory; • storing common paths into variables; • processing or constructing string; • sequentially executing a command after another; • conditionally executing commands based on given parameters; and • killing process after finish executing a task. The fragment of code shown in Figure 2.7 is a sample of a Rake task defined for executing a test with Instruments. The code starts by defining a number of common paths of an Xcode project. Then the code defines a Rake task named test, which accepts a UI Automation script’s file name before starts building and performing a test, respectively. After testing is finished, the simulator process is terminated. Figure 2.7: Sample of a Rake task for executing an instruments test. A Rake task defined as in the example above could also be invoked by executing the command shown in Figure 2.8. Figure 2.8: Sample of a command to invoke a defined Rake task. A specific Rake file with a name different from the default one could also be
- 35. specified by using —rakefile option as seen in Figure 2.9. Figure 2.9: Sample of a command to invoke a Rake task with -–rakefile. Rake is used throughout this work for many testing scenarios. For instance, to sequentially execute a set of test cases on a single machine or to test a particular test case given a path to a UI Automation script in a form of a Rake task. 2.2.4 VIRTUALIZATION Virtualization is the process of emulating real IT resources with virtual IT resources. Virtual machines created from the process act exactly like actual IT resources with their own guest operating systems, which are independent of the host operating system that created them. Virtual machines run through the virtualization (hypervisor) software, which in turn run on a physical host, virtual machines can be easily copies and moved to another virtualization host as needed [20]. There are some limitations on using the normal execution environment to execute UI Automation test cases. First of all, only one test script can be run at a time. Therefore, all test cases in a test suite have to be run sequentially in this environment. It is also not possible to have multiple instances of the simulators running on a single machine, or to have multiple instances of the debuggers to run the tests on multiple devices attached to the machine. This means that this execution process could not be sped up simply by having multiple separated processes to execute test cases in parallel on a single machine. Many practitioners suggested the use of multiple virtual machines running simultaneously on a single machine to solve the problem that only one instance of UI Automation debugger and simulator can be run at a time. Having multiple virtual machines run on a single machine means that there are multiple debuggers to run the tests. With limited resources, virtualization technologies can increase the efficiency and better utilization of the available resources in a cost effective manner. However, due to a strict licensing policies stipulated in Apple’s software license agreement under the Section 2B(iii) [3], only two additional virtual machines of Mac OS X could be run on a single Mac machine. This restrictive license limits how far hardware resources could be utilized under this environment. Although using virtualization technologies increase the number of debuggers and simulators to run the test, this solution still requires manual test distribution, meaning test cases needs to be manually selected and distributed to those virtual machines and manually executed. A good system should not require its users to be responsible for test case distribution. This also could slow down the process if the size of the cluster becomes large enough [36]. Nevertheless, this work-around suggests a distributed execution environment where there can be multiple instances of debuggers running in a cluster of machines, while test cases also get automated distribution. Virtualization technologies were used to set up such a cluster in various configurations for the experiments described in Chapter 4.
- 36. 2.3 HADOOP AND HadoopUnit This section provides a brief overview of Hadoop, including the Hadoop Distributed File System (HDFS) and the Map Reduce (MR) programming model, and the HadoopUnit distributed test execution environment that is built upon the Hadoop platform. 2.3.1 HADOOP Hadoop is a collection of open source tools, libraries, and methodologies designed to run on commodity hardware or in the cloud that collectively serve as a scalable platform for big data analysis [56]. Hadoop consists of two core components: the Hadoop Distributed File System and MapReduce. A set of machines running these two components is known as a Hadoop cluster. An individual machine in a Hadoop cluster is known as a node and a cluster can consist of just one node or as many as a thousand of these nodes. HDFS Data in a Hadoop cluster is stored in the Hadoop Distributed File System (HDFS). In HDFS, data files are split into blocks, usually of 64 MB or 128 MB each, which are significantly larger than a conventional file system that is usually has a block size of 64 KB. These blocks are distributed across many different machines in the Hadoop cluster. Hadoop was designed with an assumption that hardware will fail, so some of the blocks are replicated on other machines, meaning that there are more than one machine storing the same copy of these blocks. In case of hardware failure, the same block of data could still be extracted from the other nodes in the cluster. The node in the cluster that stores these blocks of data files is called DataNode. There is a single master node in the cluster called the NameNode that controls and keeps track of the locations of all these data blocks stored in different DataNode in the cluster as well as which blocks compose a file stored in the HDFS. There is also one or more Secondary NameNodes that act as checkpoints for the NameNode. In the case where the NameNode fails, the NameNode can be restarted using a backup snapshot stored in the Secondary NameNode [65] (Figure 2.10).
- 37. Figure 2.10: HDFS architecture. MapReduce Hadoop provides big data processing capability through the MapReduce (MR) programming model. The MapReduce programming model was built upon the concept that a computation is applied (“mapped”) over a large number of records distributed all over the cluster to generate partial results that, in turn, are aggregated (“reduced”) to produce the final solution [14]. The MapReduce programming model hides underlying execution details from users and provides automatic parallelization and distribution where developers can concentrate on writing data processing functions in a form of MapReduce jobs. A MapReduce job consists of two functions: a Map function and a Reduce function. A Map function processes a given split of data derived from a block of data in a form of key/value pairs and produces intermediary output, also as a set of key/value pairs. The shuffle and sort works in the background, based on the key from the intermediary output, and feeds the output as an input to a Reduce function. A Reduce function aggregates the values of the processed intermediary output from the Map functions based on the key part of the given key/value pairs and provides the ultimate results of the MapReduce job. Figure 2.11 illustrates how MapReduce works.
- 38. Figure 2.11: Overview of how MapReduce works. MapReduce jobs are controlled by a software process running on the master NameNode called the JobTracker. Clients submit a MapReduce job through the JobTracker, which the Job-Tracker then assigns Map and Reduce tasks to, using the other DataNodes that stored a block of data to be processed in the cluster. These DataNodes each run a software process called the Task-Tracker. The TaskTracker receives Map or Reduce tasks from the JobTracker and instantiates the given task on the node and report progress back to the JobTracker. In case of task failures or in case when a particular task take unusually long, the tasks are restarted on other available nodes in the cluster automatically in the background by the JobTracker. Under speculative execution, if there is a free node available in the cluster, a redundant task will be assigned to it and the result will be collected from the node that finish executing the task first. 2.3.2 HadoopUnit HadoopUnit is a distributed execution environment proposed by Parveen and Tilley [46] that is tailored specifically for executing JUnit test cases concurrently in the cloud. HadoopUnit was originally developed to address the need of mitigating the lengthy test execution of the Hadoop production code by using the Hadoop platform itself, because the method of test execution at that time took very long times to run and could not provide timely feedbacks to the developers. Hadoop was created using Java and its test cases are in the form of JUnit, hence HadoopUnit was originally created specifically for executing JUnit test cases [60]. HadoopUnit Architecture HadoopUnit was built upon Hadoop with an adapted concept that a test suite is composed of a large set of test cases. Test cases are distributed all over machines in a cluster and the test execution is considered computing or processing upon them. The execution of test cases generates partial test results, which are part of the test suite. The final result is
- 39. aggregated from those partial test results from the execution of test cases that are parts of a complete test suite. HadoopUnit provides the infrastructure for JUnit test cases distribution and execution over a cluster of machines. HadoopUnit hides all the complexity of the cluster management and maintenance, job scheduling, resource allocation, and fault tolerance from the testers, thus letting them focus on what is important to them: testing. As shown in Figure 2.12, there are three core functional components in HadoopUnit. 1. Test Case Extraction: This component of HadoopUnit is responsible for gathering all the test cases to be executed from the test suite and generating a test case list in the form of line-delimited string with each line composed of test case name and test execution command pair, a key/value pair format suitable to be processed by Hadoop. The original version of HadoopUnit used Ant, a Java build tool, as a component to provide this functionality. 2. The Map Function: This component of HadoopUnit is a Hadoop Map function responsible for receiving and processing given test case name/test execution command pairs by using the test case name as a key and executing the corresponding command as a separate process. The Map function produces intermediary results in a form of a test case name/test result pair, a key/value pair format suitable to be processed by the next component, the Reduce function. 3. The Reduce Function: This component of HadoopUnit is a Hadoop Reduce function responsible for collecting the intermediary test results from the Map function, aggregating into one report, and placing it in the HDFS where testers can extract the report to their machine.
- 40. Figure 2.12: Overall architecture of HadoopUnit. Testing with HadoopUnit Testing with HadoopUnit begins with testers creating a HadoopUnit project and uploading the test suite to the file system where all the production code (a.k.a. the system under test), test cases for testing the production code, and dependent libraries were already in place in the HadoopUnit cluster. The test suite is then extracted and split into multiple sets of test cases that are distributed to machines in the Hadoop cluster. Each node executes the given set of test cases and returns back the test result. All the test results of all the execution of test cases are then being merged and reported back to the tester. In keeping with HadoopUnit’s goal of reducing the cost of running large regression test suites, it adopted the retest-all approach to software testing. While selective re-test techniques such as test case selection (test only what have been changed), test case prioritization (test the one with higher priority earlier), and test suite reduction (eliminate test cases from the test suite) can reduce the cost of regression testing, and indeed they can also be used in conjunction with HadoopUnit, they also reduce the chances that regression bugs can be revealed. Some set of test cases that could potentially expose those bugs might be eliminated in the reduction process [23, 50]. The retest-all approach is the simplest approach that avoids such errors and was selected by HadoopUnit; HadoopUnit shifts the focus to making the testing faster. To be able to be executed concurrently under a distributed environment such as HadoopUnit, a set of test cases must be independent from one another. They cannot be executed at the same time without violating temporal dependencies if these test cases are co-dependent. The tests should not affect each other and the order of execution should also not affect the final test result. This means that the test should be self-contained, isolated,
- 41. and fully functional. Although having this requirement in a set of test cases means that it can take longer to execute each test case, such isolated tests are valuable and provide higher quality feedback to the testers. This is because the failed test cases are not caused by any other dependent test cases in the test suite, which would require testers to further interpret the meaning of each failed test case [60, 7]. Requiring manual test case distribution not only slows down the whole testing process but also demands a substantial number of tasks be done by the testers themselves. After finishing the execution, testers also have to gather the results from the different machines to which they manually distributed the tests. This can be overwhelming, especially if the cluster size becomes large. HadoopUnit provides automated and transparent test case distribution, which is an ability that HadoopUnit inherited from Hadoop: it can push input data into nodes in the cluster automatically as well as to collect results and report back when finished. Test load distribution also is handled automatically by the underlying Hadoop platform for load balancing of the cluster. This is because if some of test cases take longer than another, they are distributed to other available nodes in the cluster automatically [9]. This approach of HadoopUnit better utilizes the available resources and prevents bottlenecks for rapid test execution.
- 42. CHAPTER 3 Using UI Automation with HadoopUnit In migrating from a traditional execution environment to a distributed execution environment like HadoopUnit, careful, up-front planning needs to be taken, especially when planning to automate the testing process. Simply using a good set of tools does not guarantee the success of the project. This chapter provides information on UI Automation test suites, HadoopUnit customization, and using HadoopUnit as a platform for performing GUI testing of iOS applications. 3.1 UI AUTOMATION TEST SUITES In order to migrate from a traditional GUI testing environment to the HadoopUnit distributed execution environment, the System Under Test (SUT) and its test suite must be prepared so that an execution of different test cases can be performed independently in parallel in the new environment. It is recommended practice that a test suite be developed in conjunction with the application so that the test could be run frequently as the development processes [48]. It is worth noting that an automated GUI testing project should be treated as a genuine programming project [33]. Once an application is released, it may be used or serviced for years. During the course of time, an application may inevitably need to be fixed (patches or bug fixes), changed (improvement or enhancements, or environmental changes such as operating system upgrade), or extended (adding new functionality to the application) [8]. The tests, once written, must also be maintained along with the application. Using capture/replay tools helps testers getting up and run to build a test suite quickly. However, for the test suite to be maintainable, some considerations on the test case design are needed so that the test cases will have such qualities. The test suite needs to evolve along with the software project [12]. The UI of an application will change throughout the life cycle of software development, especially in agile methodologies where changes in requirements are welcome, as long as the changes satisfy the users. Test cases need to be designed to be able to cope with this flexibility so the costs of such modification to the test case are kept to the minimum. The goal is to reduce the amount of work required to update the test scripts [39, 12]. 3.1.1 TEST CASE DESIGN Every feature of an application should be defined and written as a function within a test suite where test cases that need to test these features will need to access these functions. In other words, instead of having test cases to contain scripts for tapping a particular button to achieve some particular tasks, a function for such tasks should be created to wrap the underlying interactions with the UI around inside the function. This practice hides the low-level implementation of how the feature could be accessed. When the UI of the feature changes, the function that accesses that feature is the only place that needs to be
- 43. updated, instead of every test case that tests the feature. It’s also a good idea to pull out common scripts and create utility functions that other test cases can use, so when more functionality is added to the function, such as additional logging capabilities, then all test cases that use this function gain the additional functionality automatically [31, 35]. Jonathan Penn suggested a novel technique for designing UI Automation scripts for better maintenance by representing each screen of an application as a screen object with necessary methods to access GUI elements displayed on the screen [47]. The test cases that need to have access to the screens can import the script containing the screen object and perform testing on GUI elements as necessary with the provided methods. This way, if the user interface of the screen was changed, only the script representing the screen needs to be updated, instead of every place that needs to access that changed user interface, thus reducing the ripple effect from UI changes to test scripts. For HadoopUnit, while the dependency within the same test case is acceptable, different test cases in a test suite needs to be designed in such a way that they can be executed independently—meaning the order of execution of test cases should not affect the test result. Because the test case execution will be executed on different nodes on a cluster, such dependency can cause problems in test execution. This characteristic is also called test case contamination: the test case tA should not contaminate the test case tB no matter which one got executed first [60, 36]. A test case should set up the necessary data and other information required for testing without requiring other test cases (self-contained). There is also another benefit of keeping test cases independent: a test case could be updated, improved, or refactored without affecting the other test cases [27]. Since each test case is independent of each other, when a test case fails, testers know exactly which test case to look for rather than requiring the tester to analyze which other test cases also depend on the failed test case. 3.1.2 TEST CASE ANALYSIS Test case analysis is necessary for an application with existing UI Automation test suites in order to migrate to HadoopUnit environment. For example, the existing test suites might contain all test cases in one UI Automation script file. Distributing test cases in such a test suite having only one master UI Automation script file is not possible because it requires a mechanism to selectively execute test cases inside the test suite. This is due to the limitation of UI Automation that a parameter cannot be sent to the UI Automation engine through the command line terminal to selectively execute a specific test case within a test suite. Such types of test suites require test case analysis to break the test suite down into several files of test cases in order to prepare the test suites to run on HadoopUnit distributed environment. As mentioned in the previous section, the UI Automation test script file representing each test case should be self-contained and must be independent of one another. It can contain many tests but it should not require the other test file to work. If there are such dependencies between them, the tester should consider grouping them together into one
- 44. script file. The result of test case analysis is a set of UI Automation test script files, each representing a single test case. Each file should be executable with Instruments. 3.2 HadoopUnit CUSTOMIZATION The original version of HadoopUnit was designed specifically to execute JUnit test cases for Java applications, hence, it operates under a different environment from that of iOS applications. Some customization had to be made to set up HadoopUnit to prepare it for GUI testing of iOS applications. This section describes changes to the operational environment, gathering test results, and the resultant HadoopUnit architecture. 3.2.1 OPERATIONAL ENVIRONMENT In order to migrate GUI testing of iOS application from a traditional environment towards a distributed execution environment like HadoopUnit, a Hadoop cluster needs to be set up in a way that each node is be able to compile, build, and run iOS applications. This requires each node in the cluster to have the Xcode, Instruments, iOS simulator, iOS SDK, as well as the Command line tool for OS X installed on the node. These tools operate under the Mac OS X operating system—Apple’s proprietary operating system based on UNIX. This environmental requirement leads to several limitations and considerations that HadoopUnit has to accommodate. Only one test case can be run at a time during test execution. This limitation means that regardless of the number of test cases in a test suite, they have to be executed sequentially [4]. This is not a problem for a distributed execution environment like HadoopUnit since there are many different nodes of machines available in the cluster to execute the test case concurrently. However, HadoopUnit needs to be configured to execute only one Map function at a time to accommodate this limitation, rather than the default value of two. Only one instance of the iOS simulator can be run at a time. This is a limitation of the iOS simulator, a tool for running the application under test itself. This limitation is similar to the test case execution limitation, which means that another instance of iOS simulator cannot be started to run another set of tests on a single machine. In this case, the number of nodes available in the cluster directly correlates to the number of simulators available for test case execution. As described in Section 2.2.4, due to licensing restrictions, only two additional instances of an OS X virtual machines can be run and they have to run on Apple’s hardware only [3]. This requirement limits the size of the Hadoop cluster that could be constructed given available hardware resources. This restrictive requirement also makes it difficult to move this test execution environment to the cloud. Mac OS X isn’t available as an instance type on major cloud providers such as Amazon’s EC2 or Microsoft’s Azure. This means that a Hadoop cluster of Mac OS X machines for testing iOS applications currently needs to be constructed and used internally only. HadoopUnit as a platform for distributed test execution needs to account for these
- 45. Exploring the Variety of Random Documents with Different Content
- 46. “O that in England there might be A duty on hypocrisy! A tax on humbug, an excise On solemn plausibilities, A stamp on everything that canted! No millions more, if these were granted, Henceforward would be raised or wanted.” So an American politician, who, by caucus-packing, “wire-pulling,” and perhaps bribery, has contrived to get elected to a State legislature or to Congress, will publicly thank his fellow-citizens for having sent him there “by their voluntary, unbiased suffrages.” When the patriot, Patkul, was surrendered to the vengeance of Charles XII of Sweden, the following sentence was read over to him: “It is hereby made known to be the order of his Majesty, our most merciful sovereign, that this man, who is a traitor to his country, be broken on the wheel and quartered,” etc. “What mercy!” exclaimed the poor criminal. It was with the same mockery of benevolence that the Holy Inquisition was wont, when condemning a heretic to the torture, to express the tenderest concern for his temporal and eternal welfare. One of the most offensive forms of cant is the profession of extreme humility by men who are full of pride and arrogance. The haughtiest of all the Roman Pontiffs styled himself “the servant of the servants of God,” at the very time when he humiliated the Emperor of Germany by making him wait five days barefoot in his ante-chamber in the depth of winter, and expected all the Kings of Europe, when in his presence, to kiss his toe or hold his stirrup. Catherine of Russia was always mouthing the language of piety and benevolence, especially when about to wage war or do some rascally deed. Louis the Fourteenth’s paroxysms of repentance and devotion were always the occasion for fresh outrages upon the Huguenots; and Napoleon was always prating of his love of peace, and of being compelled to fight by his quarrelsome neighbors. While the French revolutionists were shouting “Liberty, Equality and Fraternity!” men were executed
- 47. in Paris without law and against law, and heads fell by cartloads from the knife of the guillotine. The favorite amusement of Couthon, one of the deadliest of Robespierre’s fellow-cutthroats, was the rearing of doves. The contemplation of their innocence, he said, made the charm of his existence, in consoling him for the wickedness of men. Even when he had reached the height of his “bad preëminence” as a terrorist, he was carried to the National Assembly or the Jacobin Club fondling little lapdogs, which he nestled in his bosom. It is told of one of his bloody compatriots, who was as fatal to men and as fond of dogs as himself, that when a distracted wife, who had pleaded to him in vain for her husband’s life, in retiring from his presence, chanced to tread on his favorite spaniel’s tail, he cried out, “Good heavens, Madam! have you no humanity?” “My children,” said Dr. Johnson, “clear your minds of cant.” If professional politicians should follow this advice, many of them would be likely to find their occupation clean gone. At elections they are so wont to simulate the sentiments and language of patriotism,—to pretend a zeal for this, an indignation for that, and a horror for another thing, about which they are known to be comparatively indifferent, as if any flummery might be crammed down the throats of the people,—that the voters, whom the old party hacks fancy they are gulling, are simply laughing in their sleeves at their transparent attempts at deception. Daniel O’Connell, the popular Irish orator, is said to have had a large vocabulary of stock political phrases, upon which he rang the changes with magical effect. He could whine, and wheedle, and wink with one eye, while he wept with the other; and if his flow of oratory was ever in danger of halting, he had always at hand certain stereotyped catch-words, such as his “own green isle,” his “Irish heart,” his “head upon the block,” his “hereditary bondsmen, know ye not,” etc., which never failed him in any emergency.
- 48. Offensive as are all these forms of speech without meaning, they are not more so than the hollow language of—strange to say,—some moral philosophers. Many persons have been so impressed by the ethical essays of Seneca, in which he sings the praises of poverty, and denounces in burning language the corruption of Rome and the extortion in the provinces, that they could account for the excellence of these writings only on the theory of a Christian influence; and a report gained credit that the Roman philosopher had met and conversed with the Apostle Paul. But what are these brilliant moral discourses? Reading them by the light of the author’s life and character, we find they are only words. A late German historian tells us that the same Seneca who could discourse so finely upon the abstemiousness and contentment of the philosopher, and who, on all occasions, paraded his contempt for earthly things as nothingness and vanity, amassed, during the four years of his greatest prosperity and power, a fortune of three hundred millions of sesterces,—over fifteen millions of dollars. While writing his treatise on “Poverty,” he had in his house five hundred citrus tables, tables of veined wood brought from Mount Atlas, which sometimes cost as much as twenty- five, and even seventy thousand dollars. The same Seneca, who denounced extortion with so virtuous anger, built his famous museum gardens with the gold and the tears of Numidia. The same Seneca, who preached so much about purity of morals, was openly accused of adultery with Julia and Agrippina, and led his pupil Nero into still more shameful practices. He wrote a work upon “Clemency,” yet had, beyond question, a large part of Nero’s atrocities upon his conscience. It was he who composed the letter in which Nero justified before the Senate the murder of his own mother.[13] Common, however, as are meaningless phrases on the stump and platform, and even in moral treatises, it is to be feared that they are hardly less so in the meeting-house, and there they are doubly offensive, if not unpardonable. It is a striking remark of Coleridge,
- 49. that truths, of all others the most awful and interesting, are too often considered so true that they lose all the power of truth, and lie bedridden in the dormitory of the soul, side by side with the most despised and exploded errors. Continual handling wears off the beauty and significance of words, and it is only by a distinct effort of the mind that we can restore their full meaning. Gradually the terms most vital to belief cease to mean what they meant when first used; the electric life goes out of them; and, for all practical purposes, they are dead. Hence it is that “the traditional maxims of old experience, though seldom questioned, have often so little effect on the conduct of life, because their meaning is never, by most persons, really felt, until personal experience has brought it home. And thus, also, it is that so many doctrines of religion, ethics, and even politics, so full of meaning and reality to first converts, have manifested a tendency to degenerate rapidly into lifeless dogmas, which tendency all the efforts of an education expressly and skilfully directed to keeping the meaning alive are barely found sufficient to counteract.”[14] There can be little doubt that many a man whose life is thoroughly selfish cheats himself into the belief that he is pious, because he parrots with ease the phrases of piety and orthodoxy. Who is not familiar with scores of such pet phrases and cant terms, which are repeated at this day apparently without a thought of their meaning? Who ever attended a missionary meeting without hearing “the Macedonian cry,” and an account of some “little interest,” and “fields white for the harvest”? Who is not weary of the ding-dong of “our Zion” and the solecism of “in our midst”; and who does not long for a verbal millennium when Christians shall no longer “feel to take” and “grant to give”? “How much I regret,” says Coleridge, “that so many religious persons of the present day think it necessary to adopt a certain cant of manner and phraseology as a token to each other! They must ‘improve’ this and that text, and they must do so and so in a ‘prayerful’ way; and so on. A young lady urged upon me, the
- 50. other day, that such and such feelings were the ‘marrow’ of all religion; upon which I recommended her to try to walk to London on her marrow bones only.” The language of prayer, both public and private, being made up more or less of technical expressions, tends continually to become effete. The scriptural and other phrases, which were used with good taste and judgment several generations ago, may have lost their significance to-day, and should, in that case, be exchanged for others which have a living meaning. Profound convictions, it has been truly said, are imperilled by the continued use of conventional phraseology after the life of it has gone out, so that nothing in the real experience of the people responds to it, when they hear it or when they use it. Mr. Spurgeon, in his “Lectures to Students,” remarks that “‘the poor unworthy dust’ is an epithet generally applied to themselves by the proudest men in the congregation, and not seldom by the most moneyed and grovelling; in which case the last words are not so very inappropriate. We have heard of a good man who, in pleading for his children and grandchildren, was so completely beclouded in the blinding influence of this expression, that he exclaimed, ‘O Lord, save thy dust, and thy dust’s dust, and thy dust’s dust’s dust.’ When Abraham said, ‘I have taken upon me to speak unto the Lord, which am but dust and ashes,’ the utterance was forcible and expressive; but in its misquoted, perverted, and abused form, the sooner it is consigned to its own element the better.” Many persons have very erroneous ideas of what constitutes religious conversation. That is not necessarily religious talk which is interlarded with religious phrases, or which is solely about divine things; but that which is permeated with religious feeling, which is full of truth, reverence, and love, whatever the theme may be. Who has not heard some men talk of the most worldly things in a way that made the hearer feel the electric current of spirituality playing through their words, and uplifting his whole spiritual being? And who has not heard other men talk about the divinest things in so dry,
- 51. formal, and soulless a way that their words seemed a profanation, and chilled him to the core? It is almost a justification of slang that it is generally an effort to obtain relief from words worn bare by the use of persons who put neither knowledge nor feeling into them, and which seem incapable of expressing anything real. When Lady Townsend was asked if Whitefield had recanted, she replied, “No; he has only canted.” Often, when there is no deliberate hypocrisy, good men use language so exaggerated and unreal as to do more harm than the grossest worldliness. We have often, in thinking upon this subject, called to mind a saying of Dr. Sharp, of Boston, a Baptist preacher, who was a hater of all cant and shams. “There’s Dr. ——,” said he, about the time of the first meeting of the Evangelical Alliance, “who went all the way to Europe to talk up brotherly love. If he should meet a poor Baptist minister in the street, he wouldn’t speak to him.” Robert Hall had an intense abhorrence of religious cant, to which he sometimes gave expression in blunt terms. A young preacher who was visiting him spent a day in sighing and in begging pardon for his suspirations, saying that they were caused by grief that he had so hard a heart. The great divine bore with him all the first day, but when the lamentations were resumed the next morning at breakfast, he said: “Why, sir, don’t be cast down; remember the compensating principle, and be thankful and still.” “Compensating principle!” exclaimed the young man; “what can compensate for a hard heart?” “Why, a soft head, to be sure,” said Hall, who, if rude, certainly had great provocation. Nothing is cheaper than pious or benevolent talk. A great many men would be positive forces of goodness in the world, if they did not let all their principles and enthusiasm escape in words. They are like locomotives which let off so much steam through the escape valves, that, though they fill the air with noise, they have not power enough left to move the train. There is hardly anything which so fritters spiritual energy as talk without deeds. “The fluent boaster is not the man who is steadiest before the enemy; it is well said to him that his courage is
- 52. better kept till it is wanted. Loud utterances of virtuous indignation against evil from the platform, or in the drawing-room, do not characterize the spiritual giant; so much indignation as is expressed has found vent; it is wasted; is taken away from the work of coping with evil; the man has so much less left. And hence he who restrains that love of talk lays up a fund of spiritual strength.”[15] “Prune thou thy words, the thoughts control That o’er thee swell and throng; They will condense within thy soul, And change to purpose strong. But he who lets his feelings run In soft luxurious flow, Shrinks when hard service must be done, And faints at every woe. Faith’s meanest deed more favor bears, Where hearts and wills are weigh’d. Than brightest transports, choicest prayers, Which bloom their hour and fade.”[16] It is said that Pambos, an illiterate saint of the middle ages, being unable to read, came to some one to be taught a psalm. Having learned the simple verse, “I said, I will take heed to my ways, that I offend not with my tongue,” he went away, saying that was enough if it was practically acquired. When asked six months, and again many years after, why he did not come to learn another verse, he answered that he had never been able truly to master this. A man may have a heart overflowing with love and sympathy, even though he is not in the habit of exhibiting on his cards “J. Good Soul, Philanthropist,” and was never known to unfold his cambric handkerchief, with the words, “Let us weep.” On the other hand, nothing is easier than to use a set phraseology without attaching to it any clear and definite meaning,—to cheat one’s self with the semblance of thought or feeling, when no thought or feeling exists. It has been truly said that when good men who have no deep
- 53. religious fervor use fervent language, which they have caught from others, or which was the natural expression of what they felt in other and better years,—above all, when they employ on mean and trivial occasions expressions which have been forged in the fires of affliction and hammered out in the shock of conflict,—they cannot easily imagine what a disastrous impression they produce on keen and discriminating minds. The cheat is at once detected, and the hasty inference is drawn that all expressions of religious earnestness are affected and artificial. The honest and irrepressible utterance of strong conviction and deep emotion commands respect; but intense words should never be used when the religious life is not intense. “Costing little, words are given prodigally, and sacrificial acts must toil for years to cover the space which a single fervid promise has stretched itself over. No wonder that the slow acts are superseded by the available words, the weighty bullion by the current paper money. If I have conveyed all I feel by language, I am tempted to fancy, by the relief experienced, that feeling has attained its end and realized itself. Farewell, then, to the toil of the ‘daily sacrifice!’ Devotion has found for itself a vent in words.”[17] Art, as well as literature, politics, and religion, has its cant, which is as offensive as any of its other forms. When Rossini was asked why he had ceased attending the opera in Paris, he replied, “I am embarrassed at listening to music with Frenchmen. In Italy or Germany, I am sitting quietly in the pit, and on each side of me is a man shabbily dressed, but who feels the music as I do; in Paris I have on each side of me a fine gentleman in straw-colored gloves, who explains to me all I feel, but who feels nothing. All he says is very clever, indeed, and it is often very true; but it takes the gloss off my own impression,—if I have any.”
- 54. FOOTNOTES: [13] Ulhorn’s “Conflict of Christianity with Heathenism;” pp. 93, 94. [14] Mill’s “Logic.” [15] Sermons, by Rev. F. W. Robertson. [16] Professor J. H. Newman. [17] “Life and Letters of F. W. Robertson.”
- 56. T C H A P T E R V I . SOME ABUSES OF WORDS. He that hath knowledge spareth his words.—Proverbs xvii, 27. Learn the value of a man’s words and expressions, and you know him.... He who has a superlative for everything wants a measure for the great or small.—Lavater. Words are women; deeds are men.—George Herbert. He that uses many words for the explaining of any subject, doth, like the cuttle- fish, hide himself for the most part in his own ink.—Ray. he old Roman poet Ennius was so proud of knowing three languages that he used to declare that he had three hearts. The Emperor Charles V expressed himself still more strongly, and declared that in proportion to the number of languages a man knows, is he more of a man. According to this theory, Cardinal Mezzofanti, who understood one hundred and fourteen languages, and spoke thirty with rare excellence, must have been many men condensed into one. Of all the human polyglots in ancient or modern times, he had perhaps the greatest knowledge of words. Yet, with all his marvellous linguistic knowledge, he was a mere prodigy or freak of nature, and, it has been well observed, scarcely deserves a higher place in the Pantheon of intellect than a blindfold chess-player or a calculating boy. Talking foreign languages with a fluency and accuracy which caused strangers to mistake him for a compatriot, he attempted no work of utility,—left no trace of his colossal powers; and therefore, in contemplating them, we can but wonder at his gifts, as we wonder at the Belgian giant or a five-legged lamb. In allusion to his hyperbolical acquisitions, De Quincey suggests that the following would be an appropriate epitaph for his eminence: “Here
- 57. lies a man who, in the act of dying, committed a robbery,— absconding from his fellow-creatures with a valuable polyglot dictionary.” Enormous, however, as were the linguistic acquisitions of Mezzofanti, no man was ever less vain of his acquirements,—priding himself, as he did, less upon his attainments than most persons upon a smattering of a single tongue. “What am I,” said he to a visitor, “but an ill-bound dictionary?” The saying of Catherine de Medicis is too often suggested by such prodigies of linguistic acquisition. When told that Scaliger understood twenty different languages,—“That’s twenty words for one idea,” said she; “I had rather have twenty ideas for one word.” In this reply she foreshadowed the great error of modern scholarship, which is too often made the be-all and the end- all of life, when its only relation to it should be that of a graceful handmaid. The story of the scholar who, dying, regretted at the end of his career that he had not concentrated all his energies upon the dative case, only burlesques an actual fact. The educated man is too often one who knows more of language than of idea,—more of the husk than of the kernel,—more of the vehicle than of the substance it bears. He has got together a heap of symbols,—of mere counters, —with which he feels himself to be an intellectual Rothschild; but of the substance of these shadows, the sterling gold of intellect, coin current throughout the realm, he has not an eagle. All his wealth is in paper,—paper like bad scrip, marked with a high nominal amount, but useless in exchange, and repudiated in real traffic. The great scholar is often an intellectual miser, who expends the spiritual energy that might make him a hero upon the detection of a wrong dot, a false syllable, or an inaccurate word. In this country, where fluency of speech is vouchsafed in so large a measure to the people, and every third man is an orator, it is easier to find persons with the twenty words for one idea, than persons with twenty ideas for one word. Of all the peoples on the globe, except perhaps the Irish, Americans are the most spendthrift of language. Not only in our court-houses and representative halls, but
- 58. everywhere, we are literally deluged with words,—words,—words. Everybody seems born to make long speeches, as the sparks to fly upward. The Aristotelian theory that Nature abhors a vacuum appears to be a universal belief, and all are laboring to fill up the realms of space with “mouthfuls of spoken wind.” The quantity of breath that is wasted at our public meetings,—religious, political, philanthropic, and literary,—is incalculable. Hardly a railroad or a canal is opened, but the occasion is seized on as a chance for speeches of “learned length and thundering sound”; and even a new hotel cannot throw open its doors without an amount of breath being expended, sufficient, if economically used, to waft a boat across a small lake. One is struck, in reading the “thrilling” addresses on various occasions, which are said to have “chained as with hooks of steel the attention of thousands,” and which confer on their authors “immortal reputations” that die within a year, to see what tasteless word-piling passes with many for eloquence. The advice given in Racine’s “Plaideurs,” by an ear-tortured judge to a long-winded lawyer, “to skip to the deluge,” might wisely be repeated to our thousand Ciceros and Chathams. The Baconian art of condensation seems nearly obsolete. Many of our orators are forever breaking butterflies on a wheel,—raising an ocean to drown a fly,—loading cannon to shoot at humming-birds. Thought and expression are supplanted by lungs and the dictionary. Instead of great thoughts couched in a few close, home, significant sentences,—the value of a thousand pounds sterling of sense concentrated into a cut and polished diamond,—we have a mass of verbiage, delivered with a pompous elocution. Instead of ideas brought before us, as South expresses it, like water in a well, where you have fulness in a little compass, we have the same “carried out into many petty, creeping rivulets, with length and shallowness together.”
- 59. It is in our legislative bodies that this evil has reached the highest climax. A member may have a thought or a fact which may settle a question; but if it may be couched in a sentence or two, he thinks it not worth delivering. Unless he can wire-draw it into a two-hours speech, or at least accompany it with some needless verbiage to plump it out in the report, he will sit stock still, and leave the floor to men who have fewer ideas and more words at command. The public mind, too, revolts sometimes against nourishment in highly concentrated forms; it requires bulk as well as nutriment, just as hay, as well as corn, is given to horses, to distend the stomach, and enable it to act with its full powers. Then, again,—and this, perhaps, is one of the main causes of long-winded speeches,—there is a sort of reverence entertained for a man who can “spout” two or three hours on the stretch; and the wonder is heightened, if he does it without making a fool of himself. Nothing, however, can be more absurd than to regard mere volubility as a proof of intellectual power. So far is this from being the case that it may be doubted whether any large-thoughted man, who was accustomed to grapple with the great problems of life and society, ever found it easy upon the rostrum to deliver his thoughts with fluency and grace. Bruce, the traveller, long ago remarked of the Abyssinians, that “they are all orators, as,” he adds, “are most barbarians.” It is often said of such tonguey men that they have “a great command of language,” when the simple fact is that language has a great command of them. As Whately says, they have the same command of language that a man has of a horse that runs away with him. A true command of language consists in the power of discrimination, selection, and rejection, rather than in that of multiplication. The greatest orators of ancient and modern times have been remarkable for their economy of words. Demosthenes, when he “Shook the arsenal, and fulmined over Greece To Macedon and Artaxerxes’ throne,”
- 60. rarely spoke over thirty minutes, and Cicero took even less time to blast Catiline with his lightnings. There are some of the Greek orator’s speeches which were spoken, as they may now be read with sufficient slowness and distinctness, in less than half an hour; yet they are the effusions of that rapid and mighty genius the effect of whose words the ancients exhausted their language in describing; which they could adequately describe only by comparing it to the workings of the most subtle and powerful agents of nature,—the ungovernable torrent, the resistless thunder. Chatham was often briefer still, and Mirabeau, the master-spirit of the French tribune, condensed his thunders into twenty minutes. It is said that not one of the three leading members of the convention that formed the Constitution of the United States spoke, in the debates upon it, over twenty minutes. Alexander Hamilton was reckoned one of the most diffuse speakers of his day; yet he did not occupy more than two hours and a half in his longest arguments at the bar, nor did his rival, Aaron Burr, occupy over half that time. A judge who was intimately acquainted with Burr and his practice declares that he repeatedly and successfully disposed of cases involving a large amount of property in half an hour. “Indeed,” says he, “on one occasion he talked to the jury seven minutes in such a manner that it took me, on the bench, half an hour to straighten them out.” He adds. “I once asked him, ‘Colonel Burr, why cannot lawyers always save the time, and spare the patience of the court and jury, by dwelling only on the important points in their cases?’ to which Burr replied, ‘Sir, you demand the greatest faculty of the human mind, selection.’” To these examples we may add that of a great English advocate. “I asked Sir James Scarlett,” says Buxton, “what was the secret of his preëminent success as an advocate. He replied that he took care to press home the one principal point of the case, without paying much regard to the others. He also said that he knew the secret of being short. ‘I find,’ said he, ‘that when I exceed half an hour, I am always doing mischief to my client. If I drive into
- 61. the heads of the jury unimportant matter, I drive out matter more important that I had previously lodged there.’” Joubert, a French author, cultivated verbal economy to such an extreme that he tried almost to do without words. “If there is a man on earth,” said he, “tormented by the cursed desire to get a whole book into a page, a whole page into a phrase, and this phrase into one word,—that man is myself.” The ambition of many American speakers, and not a few writers, is apparently the reverse of this. We do not seem to know that in many cases, as Hesiod says, a half is more than the whole; and that a speech or a treatise hammered out painfully in every part is often of less value than a few bright links, suggestive of the entire chain of thought. Who wants to swallow a whole ox, in order to get at the tenderloin? Prolixity, it has been well said, is more offensive now than it once was, because men think more rapidly. They are not more thoughtful than their ancestors, but they are more vivid, direct, and animated in their thinking. They are more impatient, therefore, of long- windedness, of a loose arrangement, and of a heavy, dragging movement in the presentation of truth. “A century ago men would listen to speeches and sermons,—to divisions and subdivisions,—that now would be regarded as utterly intolerable. As the human body is whisked through space at the rate of a mile a minute, so the human mind travels with an equally accelerated pace. Mental operations are on straight lines, and are far more rapid than they once were. The public audience now craves a short method, a distinct, sharp statement, and a rapid and accelerating movement, upon the part of its teachers.”[18] It is, in short, an age of steam and electricity that we live in, not of slow coaches; an age of locomotives, electric telegraphs, and phonography; and hence it is the cream of a speaker’s thoughts that men want,—the wheat, and not the chaff,— the kernel, and not the shell,—the strong, pungent essence, and not the thin, diluted mixture. The model discourse to-day is that which
- 62. gives, not all that can be said, even well said, on a subject, but the very apices rerum, the tops and sums of things reduced to their simplest expression,—the drop of oil extracted from thousands of roses, and condensing all their odors,—the healing power of a hundred weight of bark in a few grains of quinine. “Certainly the greatest and wisest conceptions that ever issued from the mind of man,” says South, “have been couched under, and delivered in, a few close, home, and significant words.... Was not the work of all the six days [of creation] transacted in so many words?... Heaven, and earth, and all the host of both, as it were, dropped from God’s mouth, and nature itself was but the product of a word.... The seven wise men of Greece, so famous for their wisdom all the world over, acquired all that fame, each of them by a single sentence consisting of two or three words. And γνῶθι σεαυτὸν still lives and flourishes in the mouths of all, while many vast volumes are extinct, and sunk into dust and utter oblivion.” Akin to the prolixity of style which weakens so many speeches, is the habitual exaggeration of language which deforms both our public and our private discourse. The most unmanageable of all parts of speech, with many persons, is the adjective. Voltaire has justly said that the adjectives are often the greatest enemies of the substantives, though they may agree in gender, number, and case. Generally the weakness of a composition is just in proportion to the frequency with which this class of words is introduced. As in gunnery the force of the discharge is proportioned, not to the amount of powder that can be used, but to the amount that can be thoroughly ignited, so it is not the multitude of words, but the exact number fired by the thought, that gives energy to expression. There are some writers and speakers who seem to have forgotten that there are three degrees of comparison. The only adjectives they ever use are the superlative, and even these are raised to the third power. With them there is no gradation, no lights and shadows. Every hill is
- 63. Alpine, every valley Tartarean; every virtue is godlike, every fault a felony; every breeze a tempest, and every molehill a mountain. Praise or blame beggars their vocabulary; epithets are heightened into superlatives; superlatives stretch themselves into hyperboles; and hyperboles themselves get out of breath, and die asthmatically of exhaustion. Of all the civilized peoples on the face of the globe, our Hibernian friends excepted, Americans are probably the most addicted to this exaggeration of speech. As our mountains, lakes and rivers are all on a gigantic scale, we seem to think our speech must be framed after the same pattern. Even our jokes are of the most stupendous kind; they set one to thinking of the Alleghanies, or suggest the immensity of the prairies. A Western orator, in portraying the most trivial incident, rolls along a Mississippian flood of eloquence, and the vastness of his metaphors makes you think you are living in the age of the megatheriums and saurians, and listening to one of a pre- Adamite race. Our political speeches, instead of being couched in plain and temperate language, too often bristle “With terms unsquared Which, from the tongue of roaring Typhon dropped, Would seem hyperboles.” In ordinary conversation, such is our enthusiasm or our poverty of expression, that we cannot talk upon the most ordinary themes, except in the most extravagant and enraptured terms. Everything that pleases us is positively “delicious,” “nice,” or “charming”; everything handsome is “elegant,” or “splendid”; everything that we dislike is “hateful,” “dreadful,” “horrible,” or “shocking.” Listen to a circle of lively young ladies for a few minutes, and you will learn that, within the compass of a dozen hours, they have met with more marvellous adventures and hairbreadth escapes,—passed through more thrilling experiences, and seen more gorgeous spectacles,— endured more fright, and enjoyed more rapture,—than could be
- 64. crowded into a whole life-time, even if spun out to threescore and ten. Ask a person what he thinks of the weather in a rainy season, and he will tell you that “it rains cats and dogs,” or that “it beats all the storms since the flood.” If his clothes get sprinkled in crossing the street, he has been “drenched to the skin.” All our winds blow a hurricane; all our fires are conflagrations,—even though only a hen- coop is burned; all our fogs can be cut with a knife. Nobody fails in this country; he “bursts up.” All our orators rival Demosthenes in eloquence; they beat Chillingworth in logic; and their sarcasm is more “withering” than that of Junius himself. Who ever heard of a public meeting in this country that was not “an immense demonstration”; of an actor’s benefit at which the house was not “crowded from pit to dome”; of a political nomination that was not “sweeping the country like wild-fire”? Where is the rich man who does not “roll in wealth”; or the poor man who is “worth the first red cent”? All our good men are paragons of virtue,—our villains, monsters of iniquity. Many of our public speakers seem incapable of expressing themselves in a plain, calm, truthful manner on any subject whatever. A great deal of our writing, too, is pitched on an unnatural, falsetto key. Quiet ease of style, like that of Cowley’s “Essays,” Goldsmith’s “Vicar of Wakefield,” or White’s “Natural History of Selborne,” is almost a lost art. Our newspaper literature is becoming more and more sensational; and it seems sometimes as if it would come to consist of head-lines and exclamation points. Some of the most popular correspondents are those whose communications are a perfect florilegium of fine words. They rival the “tulipomania” in their love of gaudy and glaring colors, and apparently care little how trite or feeble their thoughts may be, provided they have dragon-wings, all green and gold. It was said of Rufus Choate, whose brain teemed with a marvellous wealth of words, and who was very prodigal of
- 65. adjectives, that he “drove a substantive-and-six” whenever he spoke in public, and that he would be as pathetic as the grand lamentations in “Samson Agonistes” on the obstruction of fish-ways, and rise to the cathedral music of the universe on the right to manufacture India-rubber suspenders. When Chief-Justice Shaw, before whom he had often pleaded, heard that there was a new edition of “Worcester’s Dictionary,” containing two thousand five hundred new words, he exclaimed, “For heaven’s sake, don’t let Choate get hold of it!”[19] Even scientific writers, who might be expected to aim at some exactness, often caricature truth with equal grossness, describing microscopic things by colossal metaphors. Thus a French naturalist represents the blood of a louse as “rushing through his veins like a torrent!” Even in treating on this very subject of exaggeration, a writer in an English periodical, after rebuking sharply this American fault, himself outrages truth by declaring that “he would walk fifty miles on foot to see the man that never caricatures the subject on which he speaks!” To a critic who thus fails to reck his own rede, one may say with Sir Thomas Browne: “Thou who so hotly disclaimest the devil, be not thyself guilty of diabolism.” Seriously, when shall we have done with this habit of amplification and exaggeration,—of blowing up molehills into Himalayas and Chimborazos? Can anything be more obvious than the dangers of such a practice? Is it not evident that by applying super-superlatives to things petty or commonplace, we must exhaust our vocabulary, so that, when a really great thing is to be described, we shall be bankrupt of adjectives? It is true there is no more unpardonable sin than dulness; but, to avoid being drowsy, it is not necessary that our “good Homers” should be always electrifying us with a savage intensity of expression. There is nothing of which a reader tires so soon as of a continual blaze of brilliant periods,—a style in which a “qu’il mourut” and a “let there be light” are crowded into every line.
- 66. On the other hand, there is nothing which adds so much to the beauty of style as contrast. Where all men are giants, there are no giants; where all is emphatic in style, there is no emphasis. Travel a few months among the mountains, and you will grow as sick of the everlasting monotony of grandeur, of beetling cliffs and yawning chasms, as of an eternal succession of plains. Yet, in defiance of this obvious truth, the sensational writer thinks the reader will deem him dull unless every sentence blazes with meaning, and every paragraph is crammed with power. His intellect is always armed cap-a-pie, and every passage is an approved attitude of mental carte and tierce. If he were able to create a world, there would probably be no latent heat in it, and no twilight; and should he drop his pen and turn painter, his pictures would be all foreground, with no more perspective than those of the Chinese. De Quincey, speaking of the excitability of the French, says that, having appropriated all the phrases of passion to the service of trivial and ordinary life, they have no language of passion for the service of poetry, or of occasions really demanding it, because it has been already enfeebled by continual association with cases of an unimpassioned order. “Ah, Heavens!” or “O my God!” are exclamations so exclusively reserved by the English for cases of profound interest that, on hearing a woman even utter such words, they look round expecting to see her child in some situation of danger. But in France “Ciel!” and “O mon Dieu!” are uttered by every woman if a mouse does but run across the floor. There is much suggestive truth in this. By the habitual use of strong language men may blunt and petrify their feelings, as surely as by the excessive use of alcoholic stimulants they may deaden the sensibility of the palate. “Naturally the strongest word ought to be used to give expression to the strongest feeling. But strong words have been so blunted through frequent use that they have lost their sharp edge, and pass over our thick skin without even pricking our sensibility; while, at