TestowanieIoT2016

Testing in
the world of
Internet of
Things
Dawid Pacia
Tomasz Janiszewski

“
The best way to predict the future
is to invent it.
Alan Kay, Xerox PARC, 1971

What is IoT?
The Internet of Things (IoT) is the
network of physical objects - devices,
vehicles, buildings and other items
embedded with electronics, software,
sensors, and network connectivity - that
enables these objects to collect and
exchange data.
https://en.wikipedia.org/wiki/Internet_of_Things

What is going to be the next
"big thing"? IoT?

Silvair products
● Based on TI CC2540/41 &
Nordic nRF51/52
● Bluetooth Low Energy v4.0
● Radio range: up to 500 meters

Silvair products
● Android 5.0+ app
● iOS 9.0+ app

Testing Scope &
Challenges
Hardware
○ Drivers
○ Switches
○ Sensors
○ Luminaries
○ Motors
○ Light sources / bulbs
○ Multi-protocol
gateways
Firmware
○ Bluetooth Low
Energy v4.0
protocol support
○ Silvair Smart Mesh
protocol support
○ Provisioning
○ Discovery
○ Groups and Scenes
○ OTAU
○ High Security
Software
○ Network setup
○ On / Off, dimming, color change
○ Associations
○ Factory Reset
○ OTAU
○ Cloud

“In the beginning there was
only Chaos”
● Event-driven development (for CES, BT World, etc.)
● Chaotic environment with constantly changing
priorities (on a daily basis)
● Lack of time to test SF/HW/FW on all test levels
● Ignoring product/project risk analysis
● No TDD/FDD/XP, lack of Unit tests
● PoC….. PoC everywhere
● No documentation
● Not enough testers!

Main focus
● Building and maintaining test environment
● Test automation
● Helping other teams with their testing activities

Let’s imagine a simple test
scenario...
Story: The user turns on a light bulb from an app
and turns it off from a wireless switch
Step 1: Launch mobile app and create a home
network with 1x wireless switch and 2x light
bulbs
Step 2: Turn on the bulbs from an app
Step 3: Turn off the bulbs from a switch

Flashing
App
installation
Devices
pre-configuration
START Init
Test scenario flow:
Changing
bulb state
(Step 2)
Switch
control
(Step 3)
Test suite
execution
Closure
activities
Creating “Home”
network
(Step 1)
END

Simple… right?
Well... not quite...

○ How to flash a BLE module with the firmware
which is inside a bulb or a switch?
○ How to control hardware buttons on a switch on
demand?
○ How to verify whenever a bulb has been turned on
or not?
○ How to verify if desired color, brightness, color
temperature or saturation level has been reached?
Challenges:

○ How to combine the results from multiple
sources (mobile app, firmware, hardware, cloud)
and be sure that the test cases is passed and
functionality under test is working as expected?
○ How to to know which component has failed
and when?
○ How to detect potential problems like
unpleasant LED flickering while dimming?
○ How to reset everything to default state at the
end of the test scenario?
Challenges:

Hardware
Firmware
Embedded
platforms
Cloud
Vision
system
Continuous
integration
Electric
circuits
Mobile
applications
AUTOMATIZATIONAutomatic
reporting
REST
Multiple
platform
s

Want big
impact?
Use big image.
THEY SAID THIS JOB WOULD BE EASY
YOU GOTTA BE KITTEN ME

Let’s make it
step by
step !
● Following common methodology
● Facilitating external processes
● Automation tools freedom
● Considered architecture
● Brainstorming sessions
● Team autonomy

“...Possibilities kills creativity...”
Requirements of IoT E2E Test
Environment
● Dirt-cheap!
● Simplicity - built with a KISS
("Keep it simple, stupid!")
methodology, extremely
low entry point for system
newcomers
● Modularity - the system
should handle flawlessly a
multiple of different DUT’s
(Devices Under Test)

Requirements of IoT E2E Test
Environment
● Scalability - a multiple
instances of DUT modules
and Test Arbiters should be
able to operate
simultaneously without any
hassle
● Reliability and stability - the
testing environment should
be able to operate with a
MTBF of 7 days
“...Possibilities kills creativity...”

Every engineering
starts with LEGO

Cloud
REST API
SSH
UART
Module Flashing
M
A
S
T
E
R
SSH
REST
API
B
L
E
DEBUGGER UART
FW DB
AUTH
TOKEN
/ REST
B
L
E
DEBUGGER

class switchTest(unittest.TestCase):
self.authorization = ["rpi-devDav.local","qateam","123"]
…
self.channelOff2 = self.config["channelOff2"]
def testOnOff(self):
switch = switchLib(self.config["switchAddress"], self.config["login"],
self.config["password"], self.config["platform"], self.channelOn1,
self.channelOn2, self.channelOff1)
switch.switchClick(self.channelOn1)
…
switch.switchHold(self.channelOn1,3)
switch.switchHold(self.channelOff1,3)
switch.switchWakeUp()
switch.factortReset()
def tearDown(self):
pass
Switch Control

The very first thing - prepare your
workstation!
● Unchangeable
conditions
○ Lighting (!)
○ Filters
● Stability (!)
● Usability
○ Effort in
reconfiguration
○ Reusage
● Costs

The very first thing - prepare your workstation!

Trigger
signal
Set proper
parameters /
disable automatic
Send
output
How it really looks like:
Take a snapshot,
image
preprocessing
(filtering)
Kill the
processIs camera
buffer
released?
Image
processing (blob
detection, check
elements, etc.)
Thresholding
Create
binary ROI
matrix
Trigger
signal
Gather value
matrix
Send
output
How we would like it:
Vision System

v4l2-ctl -c exposure_auto=1 | v4l2-ctl -c exposure_auto_priority=0 | v4l2-ctl
-c white_balance_temperature_auto=0 | v4l2-ctl -c absolute_exposure=171
Set proper parameters
(always manually!)
*Collection of libraries which adds a thin abstraction layer on top of video4linux2 devices. The purpose is to make it easy for
application writers to support a wide variety of devices without having to write seperate code for different devices in the same class.
brightness (int) : min=0 max=255 step=1 default=128 value=128
contrast (int) : min=0 max=255 step=1 default=32 value=32
saturation (int) : min=0 max=255 step=1 default=32 value=32
white_balance_temperature_auto(bool) : default=1 value=0
gain (int) : min=0 max=255 step=1 default=0 value=0
power_line_frequency (menu) : min=0 max=2 default=2 value=2
white_balance_temperature (int) : min=0 max=10000 default=4000 value=4000
sharpness (int) : min=0 max=255 step=1 default=24 value=24
backlight_compensation (int) : min=0 max=1 step=1 default=1 value=1
exposure_auto (menu) : min=0 max=3 default=3 value=1
exposure_absolute (int) : min=1 max=10000 default=166 value=171
exposure_auto_priority (bool) : default=0 value=0

...and this is why
[109, 5, 2,
6, 2, 3,
3, 3, 104]
Good Exposure
(value 171)
[173, 58, 33,
54, 47, 51,
38, 48, 169]
Overexposure
(value 800)

Image preprocessing
(filtering)
filtering (blur),
thresholding
noise reduction,
morphological
(erosion)
(146, 193, 255)
(255, 64, 17)

Purpose of the extended
blob detection
# Filter by Area.
params.filterByArea = True
params.minArea = 1500
# Filter by Circularity
# (4 * PI * Area) / (Perimeter^2)
params.filterByCircularity = True
params.minCircularity = 0.6
# Filter by Inertia
# It measures how elongated a shape is
params.filterByInertia = True
params.minInertiaRatio = 0.8
# Filter by Convexity
# Area of the Blob / Area of its
# convex hull
params.filterByConvexity = True
params.minConvexity = 0.87

Purpose of the extended
blob detection - example
BAD VERY BAD OK

QA CI
Environment
Test Arbiter
JIRA TEST
RESULTS
iOS iOS
APK APK Cloud
TEST
RESULTS
E2E Test
Environment
Appium Appium

Test Arbiter
class Silvair(unittest.TestCase):
driver = None
app = None
@classmethod
def setUpClass(cls):
…
…
cls.restAPIClientBulb1 =
RestAPIClient(cls.config["restAPI_IP_bulb1"],cls.usernameRestAPI,cls.passwordRestAPI,
cls.projectName)
cls.restAPIClientBulb2 =
RestAPIClient(cls.config["restAPI_IP_bulb2"],cls.usernameRestAPI,cls.passwordRestAPI,
cls.projectName)
cls.restAPIClientSwitch =
RestAPIClient(cls.config["restAPI_IP_switch"],cls.usernameRestAPI,cls.passwordRestAP
I,cls.projectName)
cls.restAPIClientBulb1.fullFlashingFlow(cls.config["bulbOrderID"])
cls.restAPIClientBulb2.fullFlashingFlow(cls.config["bulbOrderID"])
cls.restAPIClientSwitch.fullFlashingFlow(cls.config["switchOrderID"])

Test Arbiter
...
print("[LOG] " + datetime.now().strftime("%Y-%m-%d %H:%M:%S") + " Appium
server started.")
downloadFromS3(cls.config["s3ApkPath"],
os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + "/apk",
cls.config["APP_NAME"])
cls.driver = create_driver('Android', configFile, bool(int(cls.config["reinstall_app"])))
cls.driver.implicitly_wait(5)
cls.bulbMac1 = cls.config["bulbMac1"]
cls.bulbMac2 = cls.config["bulbMac2"]
cls.switchMac = cls.config["switchMac"]
cls.retryNumber = int(cls.config["retryNumber"])
cls.bulbSlot1 = int(cls.config["bulbSlot1"])
cls.bulbSlot2 = int(cls.config["bulbSlot2"])
cls.bulbProvisioningWorkaround = cls.config["bulbProvisioningWorkaround"]
cls.factoryResetWorkaround = cls.config["factoryResetWorkaround"]
cls.HWbulbWorkaround = cls.config["HWbulbWorkaround"]

Test Arbiter
...
cls.androidLib = android(cls.driver)
cls.bulbEnv = BulbEnv(cls.config["deviceNumber"],
cls.config["resolutionX"], cls.config["resolutionY"], cls.config["rowsNum"],
cls.config["columnsNum"])
cls.switchEnv = switchLib(cls.config["switchAddress"], cls.config["login"],
cls.config["password"], cls.config["platform"], cls.config["channelOn1"],
cls.config["channelOn2"], cls.config["channelOff1"])
cls.app = Pages(cls.driver, cls.switchEnv)
cls.e2eEnv = e2eLib(cls.app, cls.bulbEnv, cls.switchEnv,
cls.config["minThreshold_on"], cls.config["maxThreshold_on"])

Number of commits 212
Number of code lines added 69293
Number of code lines deleted 46028
Number of Github
contributors
3
Number of Jenkins jobs 18
Number of builds 777
QA CI Environment
QA CI
Environment
Test Arbiter
JIRA
TEST
RESULTS
TEST
RESULTS
DUT DUT DUT

def test12_associations(self):
self.app.clickDeviceOnAddedList(self.switchMac)
self.switchEnv.switchWakeUp()
self.e2eEnv.switch_turnOffBulbs(self.config["channelOff2"], [self.bulbSlot1,
self.bulbSlot2], 0, self.retryNumber)
self.app.doAssociation(self.switchMac, 0, "none")
self.e2eEnv.switch_turnOffBulbs(self.config["channelOff2"], [self.bulbSlot1,
self.bulbSlot2], 0, self.retryNumber)
self.app.swipeUp()
self.app.doAssociation(self.switchMac, 1, self.bulbMac1, "Lightness")
self.driver.back()
Tests creation? Couldn’t be
simplier !

def test18_FRswitchHW(self):
self.androidLib.resetBT()
self.app.addDevice(self.switchMac)
self.app.findAddedDeviceOnList(self.switchMac)
self.e2eEnv.switch_FR(self.switchMac, 5)
def test08_provisioningOfNodOn(self):
self.switchEnv.switchFactoryReset()
self.app.addDevice(self.switchMac)
self.app.findAddedDeviceOnList(self.switchMac)
Tests creation? Couldn’t be
simplier !
def test14_sharing(self):
self.app.share("Gmail")
self.app.share("Save to Drive")

● Long time SSH connection (fast switch state
change)
● PWM high frequency (UART testing)
● Electrostatic Discharge
● Busy camera buffer
● Appium server / flashing / setting devices test
steps timeouts
● Different screen sizes / language versions /
platforms
but where is a problem...
Just a few problems met...

● Long time SSH connection
● PWM high frequency (UART
testing)
● Electrostatic Discharge
● Busy camera buffer
● Appium server / flashing /
setting devices test steps
timeouts
● Different screen sizes /
language versions /
platforms
...there is a solution !
● Cert
Authorization
● I/O stream log
deletion
● Humidifier
● Monitoring camera
process and usage
● Moved as
set-up activity
● Gather necessary
informations and
determine while class
initiation

… unfortunately, not for all
of them :(

Performance Mobile Phone
Number of ADV packets received from a Bulb
within 15 minutes The percentage of
packets received
[%]Clean RF environment
Noisy RF environment
(1200 packets per
second)
VERY GOOD Galaxy Tab 7.0 (T230) 3921 2312 59
GOOD
LG G2 3395 1594 47
Nexus 6 5294 1948 37
Samsung Galaxy S6 5818 2130 37
AVERAGE
Samsung Galaxy Note 4 5070 1607 32
Nexus 9 5377 1541 29
Sony Xperia Z2 5862 1604 27
Nexus 5 5964 1518 25
BAD
Samsung Galaxy S5 mini 6397 1197 19
Samsung Galaxy Note 10.1 5268 751 14
VERY BAD
Xiaomi Mi3 3237 278 9
LG G2 mini 4081 328 8
Samsung Galaxy A3 4050 319 8
Samsung Galaxy S4 mini 4037 329 8
HTC One M8 3868 279 7
Sony Xperia T3 4040 283 7

Takeaway...
● Does Scrum really work? Yes, it does!
● Think big, but act small
● Iterate and refine
● Unleash team creativity
● Focus on interoperability
● Modularity is the key
● Incorporate RBT - focus on risky functionalities,
e.g. OTAU
● Physics is ruthless
● IoT is also hardware
● Security!
Want to know more about IoT protocols? Just grab
our free ebook at: https://silvair.com/media

Thanks!
Any questions?
You can find us at:
Dawid Pacia:
◎ e-mail: paciadawid@gmail.com
◎ / paciadawid
Tomasz Janiszewski:
◎ e-mail: janiszewski.tomasz@gmail.com
◎ / tjaniszewski
https://silvair.com/careers

TestowanieIoT2016

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to TestowanieIoT2016 (20)

More from kraqa (20)

Recently uploaded (20)

TestowanieIoT2016