IoT Software Testing Challenges: The IoT World Is Really Different

W6

Special
Topics

10/5/16
11:30

IoT
Software
Testing
Challenges:
The

IoT
World
Is
Really
Different

Presented
by:

Jon
Hagar

Grand
Software
Testing

Brought
to
you
by:

350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073

888-‐-‐-‐268-‐-‐-‐8770
·∙·∙
904-‐-‐-‐278-‐-‐-‐0524
-‐
info@techwell.com
-‐
http://www.starwest.techwell.com/

Jon
Hagar

Jon
Hagar
is
a
systems
software
engineer
and
testing
consultant,
supporting

software
product
integrity
and
verification
and
validation
(V&V),
with
a

specialization
in
mobile
and
embedded
software
system
testing.
For
more
than

thirty
years,
Jon
has
worked
in
software
testing
and
engineering
projects.
He

authored
Software
Test
Attacks
to
Break
Mobile
and
Embedded
Devices;
consults,

presents,
teaches,
and
writes
regularly
in
many
forums
on
software
testing
and

V&V;
and
is
lead
editor/author
on
committees
including
OMG
UTP
model-‐based
test

standard,
IEEE
1012
V&V
plans,
and
ISO/IEEE/IEC
29119
software
test
standard.

Contact
Jon
at
jon.d.hagar@gmail.com.

IoT Software Testing Challenges:
The IoT World Is Really Different
Jon D. Hagar, Consultant, Grand Software Testing
embedded@ecentral.com
1
• Challenges in both hardware and software development
– Can we produce quality within schedule and cost constraints?
– Merging of physical, cyber, and networked worlds
• All the problems of IT and Mobile Software
– It only takes a few minutes of using an App before users like or
dislike it
• Worse than that. . .
– IoT can Kill: You may be on the nightly news (bad press is not good)
– Companies want a piece of the IoT pie (4-10 trillion USD in next 10 years)
Copyright 2015 Jon D. Hagar - “Software Test Attacks to Break Mobile and Embedded Devices”
2
The IoT Opportunity

My Top IoT Challenges
(and which ones I cover in red)
• Complex software and hardware (for testing)
– Sensors and the “real world”
– How to conduct development
• Numbers of devices and configurations (and how to test)
– Configurations and compatibility
– Reliability and fault tolerance
• Big data and analytics
• Privacy and Security
• Connectivity (systems and systems of systems)
– Integration
• Safety
• Life cycle – unified hardware-software dev-test-ops (a dream)
– Tools to support development, ops and tests
– Cost and schedule
– Concurrent software and hardware development
• Integrated Operations - given the above
• International standards for devices and protocols
3Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Basic Definitions
• Test – the act of conducting experiments on something to
determine the quality (s) and provide information
– Many methods, techniques, approaches, levels, context
– Considerations: input, environment, output,
instrumentation
• Quality(ies) – Value to someone (that they will pay for)
– Functional
– Non-functional
– It “works”
– Does no harm
• Are there (critical) bugs?
4Copyright 2015 Jon D. Hagar - “Software Test Attacks to Break Mobile and Embedded Devices”

Technology Space
Physical
Systems
(circa 100,000 BC)
Cyber
Systems
(1950s)
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
(Cloud)
Many Options
Huge
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)
Cyber-Physical
Systems
Where are IoT Devices in Computer Space?
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
(Cloud)
Many Options
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)

• Embedded – Software contained in “specialized” hardware…
– Minimal networking-communications
PLUS
• Mobile and handheld smart devices—small,
held in the hand, highly connected (web, cloud, servers,….)
• IoT – Internet of Things are “traditional” and new
devices with software and communication added
What is an IoT Device
Test Brakes
Is IoT new?
We have had embedded, controllers, M-2-M, the internet
Why is IoT different?
It brings together:
• Connectivity
• Big data
• Resource limitations – size, batteries, processing, memory,
other
• Numbers and types of devices
• Mixes cloud, PC-IT, mobile, embedded, network, and user
• Security and privacy

Challenge 1:
Complex Software and Hardware
for Testing
In Parallel and Supporting Development
IoT Testing Opportunities
• Requirements verification checking
– Necessary but not sufficient
• Risk–based testing
– Historic but tried and true
• Pattern or attack-based exploratory tests
– Pattern 1: Model-based testing
– Pattern 2 (and Challenge 2) : Math-based testing
– Pattern 3: Skill/experience-based testing
– Pattern 4: Standards/process-based testing

Pattern 1 : Model-Based Testing
• Address systems, software,
and hardware test
• Developer and Independent
Modeling
• Improved Understanding
• Interest and use of model-based testing is growing in industry segments
• Telecom, finance, automotive, aerospace
• European and USA interests
• IoT “high integrity” areas will need it
• Model-based testing can support:
• Generation of test cases from models into test automated execution
engines directly using scripts or through the use of keywords
• Early testing with improved understanding of the system and risks
• Use of models to support simulations to drive test environments
• Verification via compares between development and test models
• Generation of test result oracles or judges
• Support of independent testing such as Independent V&V (IV&V)
• Model analysis and formal verification
Model-based Testing in IoT
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”

13
An Example Test Flow with Modeling for IoT
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
How to Address
Pattern 2 and Challenge 2
with one concept
How do we handle
many
configurations,
options, and even
test data sampling?

Pattern 2: Math-based Testing
Testing is a sampling problem:
How can testing use Math to help?
• Test systematically the numbers of devices, configurations,
networks, etc.
• Sampling in environments and quality control
• Selection of data from the input domain space
• Big Data analytics fed into testing
• Andriod
Use Math to Address Challenge 2:
Numbers of devices and configurations
16
• Routers
• IoT
devices
• Data
• Comm
Channels
• IoT Home
Protocol
How many Tests?
to address, data, configurations, devices
Comms, resources, integration, resources
10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests

Using the ACTS Combinatorial Tool:
Example
17
Parameters:
Andriod AppPlatform
[Device 1, Device 2, Device 3, Device 4, Device 6,
Device 7, Device 8, Device 9, Device 10]
IoTProtocolHome [true, false]
IoT Devices
[Refrig, Stove, mircrowave, TV, front door, Garage
door, Home gaurd, Stereo, Temp Control, Lights,
Drapes, Water Heater, window openers]
Routers [0, 1, 2, 3, 4, 5]
Comm providers
[Cell1, Broadband, cable, Cell 2, Space based,
Vendor godzilla]
Data [1, 0, -1, 99999, -99999, 100, -200]
Test Case# Andriod AppPlatform IoTsHome IoTDevices Routers
Comm
providers Data
0 Device 1 false Refrig 1 Broadband 0
1 Device 2 true Refrig 2 cable -1
2 Device 3 false Refrig 3 Cell 2 99999
3 Device 4 true Refrig 4
Space
based -99999
4 Device 6 false Refrig 5
Vendor
godzilla 100
5 Device 7 true Refrig 0 Cell1 -200
119 Test
Sample
Statistical Math Tools
General Technique
Concept Tool Examples
Examples of where
technique can be used
Specific sub- technique
examples
Combinatorial Testing
ACT, Hexawise
rdExpert
PICT
Medical, Automotive,
Aerospace, Information
Tech, avionics, controls,
User interfaces
Pairwise, orthogonal arrays,
3-way, and up to 6 way
pairing are now available
Design of Experiments
(DOE)
DOE ProXL
DOE++
JMP
Hardware, systems, and
software testing where
there are "unknowns"
needing to be evaluated
Taguchi
DOE
Random Testing and
Fuzz testing tools
(security)
Random number
generator feature
used from most
systems or languages
Chip makers,
manufacturing quality
control in hardware
selection
Testing with randomly
generated numbers includes:
fuzzing and use in model-
based simulations
Statistical Sampling SAS
Most sciences, engineering
experiments, hardware
testing, and manufacturing
Numerous statistical
methods are included with
most statistical tools
Software Black box
Domain Testing
Mostly used in manual
test design, though
some tools are now
coming available
All environments and types
of software tests. These
are “classic” test
techniques, but still
underused
Equivalence Class, Boundary
Value Analysis, decision
tables

Pattern 3: Skill/Experience-based Testing
Exploratory Testing - Definition
• Quoting James Bach: “The plainest definition of exploratory
testing is test design and test execution at the same time. This is
the opposite of scripted testing (predefined test procedures,
whether manual or automated). Exploratory tests, unlike
scripted tests, are not defined in advance and carried out
precisely according to plan.”
http://www.satisfice.com/articles/what_is_et.shtml
Credit to Jean Ann Harison2013
Pattern attack–based

Exploration: An Important Skill for Testers
• Some people think that all testing is exploratory
• Scientific methods
• Used at different times
– Early
– Performance
– On Hardware
– Late
• Can be based in patterns of “attacks or tours”
• Verification checking (tests) of requirements is common
• Expected necessary testing, but by itself is not sufficient
• Singular focus of many test teams that misses errors and
needed information
• Incorporates risk and attack testing within exploratory,
experience-based test planning
• Allows rapid test exploration without limitations of highly
scripted tests
• Requires “skilled” test teams
• Exploratory testing must be balanced
– Verification and Validation using standards
– Math-based and QC
– Model-based
IoT Hardware Test Planning with Exploration Concepts

Exploratory Testing In IoT
• Rapid feedback
• Learning
• Upfront rapid
(move-left) effort
• Attacking
• Address risk(s)
• Cover data
• Reliability
• Performance
• Independent assessment
• Targeting a defect
• Prototyping
• Need info for developers
• Test beyond the
requirements
• Can use the cloud
• Fault Tolerance checks
Credit to Jean Ann Harison2013Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Examples Software Attack Patterns for Exploratory Testing
Excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
24
Software Test Attack Type Attack Finds Notes on the Attack
Developer level attacks
Code and data structure
problems
Almost a quarter of errors in mobile and embedded can be found by
structural testing
Control system attacks
Hardware and software control
system errors
Many critical errors in mobile and embedded are centered in the
control logic, for example analog-to-digital and digital-to-analog
computation problems
Hardware-software attacks
Hardware and software
interface issues
The software should be tested to work with any unique hardware
Communication attacks
Digital communications
problems
Software communicates with hardware, network, and other
software with complex interfaces that should be tested
Time attacks
Time, performance, sequence,
and scenario errors
System software can have critical timing and performance factors
that testing can provide valuable information about
User interface attacks
Problems between man and
machine
The usability of devices and software are critical to success
Smart/Mobile/Hardware attacks
Issues specific to smart device
configurations including cloud
issues
Cloud-hybrid computing comprises a majority of the new software
systems being deployed
Security test hacking attacks
Software errors that can
expose devices to security
threats
Security of devices or systems is increasing in importance and
attacks include, for example, GPS and identity spoofing
Generic functional verification
attacks
Requirements and
interoperability errors
Basic checks that testers should conduct on systems and software
Static code analysis attacks
Hard to find errors that classic
testing often misses
Can often be done by the development group but sometimes the
test group must run this analysis

Pattern 4: Standards-based Testing
For Processes, Not Products
• IEEE 1012 is a standard that defines V&V processes
• Specific activities and related tasks
• Addresses V&V at system, hardware and software levels
• Can be applied to a full system, sub-system, or element
• Features in the standard include:
• Integrity levels
• Minimal V&V tasks for each integrity level
• Intensity and rigor consideration applied to V&V tasks
• Detailed criteria for V&V tasks
IEEE 1012-2012 Verification and Validation (V&V)
Planning Standard

ISO 29119 Software Test Standard
• Addressed basics of Modeling earlier
• UTP is a “language”
• ISO is considering a model-based process standard using
UTP 2.0
OMG UTP to Support Model Based Testing

Challenge 3:
IoT Data and Analytics
The Evolution of Computers is
The Evolution of Data Usage
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
(Cloud)
Many Options
Huge
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)
Data Used
by
The Few
Data
Pulled
(from whole web)
App that
Customizes a Data Slice
Limited
Data
(if any)
We do not
Know yet
? ? ?

IoT to Generate Huge Amounts of Data
(Petabyte, Exabyte, Zettabyte, Or a Yottabyte)
Current analytics focus is on marketing/sales
If user is a tester generating data…….
Testers will need to use data analytics
But for what?
IoT Data Analytics – One Future
SODA – Self Organizing Data Analytics
AI
Smart Analytics
• The tools and data are organized to support all aspects of IoT
with Artificial Intelligence and customized selection of data
based on nature of user
• Users = customer, middle men, governments, developers,
managers, testers, dev, etc.
• Research topic

Challenge 4:
IoT Security and Privacy
(Many experts think these are top priority)
• Your IoT App gets on the nightly news
• Your team sees security as someone else’s problem
• You lose personal data or your App makes personal data
available to anyone
Example of What Worries Me at Night:
Security and Privacy

• Mobile/IoT – IoT systems are highly integrated
hardware–software–system solutions which:
– Must be highly trustworthy since they handle sensitive data
– Often perform critical tasks
• Security holes and problems abound
– Coverity Scan 2010 Open Source Integrity Report - Android
• Static analysis test attack found 0.47 defects per 1,000 SLOC
• 359 defects in total, 88 of which were considered “high risk” in
the security domain
• Cars and medical devices hacked
The Current Security Situation
• Fraud – Identity
• Worms, virus, etc.
– Fault injection
• Processing on the
run
• Hacks impact
– Power
– Memory
– CPU usage
Security Error Data Analytics
(refinement of the software error data taxonomy)
• Eavesdropping – “yes everyone can
hear you”
• Hijacking
• Click-jacking
• Voice/Screen
• Physical Hacks
• File snooping
• Lost phone
Are you giving away
someone else’s keys?

• Attack 28 Penetration Attack Test
• Attack 28.1 Penetration Sub–Attacks: Authentication — Password
• Attack 28.2 Sub–Attack Fuzz Test
• Attack 29: Information Theft—Stealing Device Data
• Attack 29.1 Sub Attack –Identity Social Engineering
• Attack 30: Spoofing Attacks
• Attack 30.1 Location and/or User Profile Spoof Sub–Attack
• Attack 30.2 GPS Spoof Sub–Attack
Security Attacks
(from “Software Test Attacks to Break Mobile and Embedded Devices”)
Privacy – Restricted Data
• Different from security
»More of an issue in some countries
• Examples people might not want exposed

Challenge 5:
Connectivity
What separates IoT
from Embedded
Testing Options for Connectivity
Test Early
• Model-based testing
• Math-based testing
Test Often
• Test labs
Test Consistently
• Risk-based testing
• Requirements verification checking
• Automation

Connectivity
• To the User
Software
Hardware
Human
• To the system
• To the system of system
• To the data
Connectivity Test Concerns
• Time lag
• Data correctness
• Test on different configurations
• Data completeness
• Privacy and security (yet again)

IoT Testing Summary
• To defeat an enemy (bugs), you must know the bug
• The IoT test data is limited,
– What exists has implications
• There are challenges and patterns
• Software will be in very nearly everything
– Testing may be a limiting factor
43Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
References (my favorite books)
• “Software Test Attacks to Break Mobile and Embedded Devices”
– Jon Hagar
– IoT Tests Book in 2016
• “How to Break Software” James Whittaker, 2003
– And his other “How To Break…” books
• “A Practitioner’s Guide to Software Test Design” Copeland, 2004
• “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993
• “Computer Related Risks”, Neumann, 1995
• “Safeware: System Safety and Computers” Leveson, 1995
• Honorable mentions:
– “Systems Testing with an Attitude” Petschenik 2005
– “Software System Testing and Quality Assurance” Beizer, 1987
– “Testing Computer Software” Kaner et. al., 1988
– “Systematic Software Testing” Craig & Jaskiel, 2001
– “Managing the Testing Process” Black, 2002

More Resources
• www.stickyminds.com – Collection of test info
• www.embedded.com – info on attacks
• www.sqaforums.com - Mobile Devices, Mobile Apps - Embedded
Systems Testing forum
• Association of Software Testing
– BBST Classes http://www.testingeducation.org/BBST/
• Your favorite search engine
• My web sites and blogs
References for Statistical Math Tools
• IEEE 1012, Standard for System and Software Verification and Validation- http://standards.ieee.org/findstds/standard/1012-2012.html, IEEE press, 2012
• ISO 29119, Software Test Standard - http://www.softwaretestingstandard.org/
• Hagar, J. Software Test Attacks to Break Mobile and Embedded Devices, CRC press, 2013
• Kuhn, Kacker, Lei, Introduction to Combinatorial Testing, CRC press, 2013 (includes the tool ACTS)
• Tool: Hexawise - app.hexawise.com/
• Tool: rdExpert – www.phadkeassociates.com/
• Tool: PICT – msdn.microsoft.com/en-us/library/cc150619.aspx
• Reagan, Kiemele, Tool: DOE Pro XL - Design for Six Sigma, Air Academy Associates, self publish, 2000
• DOE++ - www.reliasoft.com/
• SAS - www.sas.com/
• Kaner, Hoffman, Padmanabhan, The Domain Testing Workbook, self publish, 2013
• Bailey, Design of Comparative Experiments. Cambridge University Press, 2008
• Kacker, Kuhn, Hagar, Wissink, "Introducing Combinatorial Testing to a Large System-Software Organization,” scheduled-2014, IEEE Software
• Whittaker, James 2003, How to Break Software, Pearson Addison Wesley
• Whittaker, James and Thompson, Herbert, How to Break Software Security, Pearson Addison Wesley, 2004
• Andrews, Whittaker, How to Break Web Software, Pearson Addison Wesley, 2006
• Levy, Tools of Critical Thinking: Metathoughts for Psychology, 1996
• Bach, Bolton, “Testing vs. Checking,” www.developsense.com/blog/2009/08/testing-vs-checking/
• Hagar, “Why didn’t testing find the embedded GM Truck fire system error?”- www.breakingembeddedsoftware.wordpress.com/
• OMG UTP 1.2, www.omg.org/spec/UTP/1.2/
• Baker, Dai, Grabowski, Schieferdecker, Williams, “Model-Driven Testing:Using the UML Testing Profile,” 2008
• Green, Hagar, “Testing Critical Software: Practical Experiences,” IFAC Conference 1995
• Boden, Hagar, “How to Build a 20-Year Successful Independent Verification and Validation (IV&V) Program for the Next Millennium,” Quality Week
Conference 1999
• Port, Nakao, Katahira, Motes, Challenges of COTS IV & V, Springer press, 2005

IoT Software Testing Challenges: The IoT World Is Really Different

More Related Content

What's hot (20)

Similar to IoT Software Testing Challenges: The IoT World Is Really Different (20)

More from TechWell (20)

Recently uploaded (20)

IoT Software Testing Challenges: The IoT World Is Really Different