Video Testing Best Practices: How to Guarantee High-Quality Video for your Customers

VIDEO EXPERIENCES: THE NEW DIGITAL APP FRONTIER
ENSURING VIDEO QUALITY IN APPS
BT UK VIDEO STRATEGY AND TESTING
DEMO, Q&A
Stephen Dolan
Product Manager
Accenture
Amir Rozenberg
Director, Product
Perfecto Mobile
Kedhar Reddy
Head of BT TV
E2E Test, BT

Source: WOWZA
Source: Adélie Studios

2- Process alignment (to your SDLC)
Use your Automation Framework
Execute tests every build
Quality Visibility
‘Test in Production’
Remote Debug to the Cloud
Shift left

3- Use the right tools (for coverage and automation)
<=>

Copyright © 2017 Accenture All rights reserved. 1111Copyright © 2017 Accenture All rights reserved.
Accelerates build and
release cycles
• Automate routine tests and
schedule for 24x7 e.g. regression
tests
• Execute simpler scripts by
abstracting navigation tests
• Reproduces errors instantly with
powerful logging
• Test Framework built to
accommodate changes
Improves product & service
reliability
• Execute more test types, more
intensively
• Run tests not possible with manual
testing
• Validate services end-to-end
Validates the multiscreen
customer experience
• Test second screen inter-device and
synchronization
• Test exactly what the customer
sees
• Measure and track system
performance

CPE Stack Testing
Multi-Site Testing
Multiscreen Testing OTT Service Testing
Applications
Functional and performance testing of
video apps:
• Validate OTT service delivery across
different devices
• Test catalogue availability
• Perceptual video quality testing
Support testing across multiple
locations and projects:
• Remote test script DEV
• Remote test execution
and analysis
• Remote access to debug‘ live’ devices in
the network
Functional and performance testing on
any screen:
• Synchronization between devices
• Interaction between devices
• Ensure no inter-device interference
• Perceptual video quality testing
Functional and performance testing of
new CPE:
• Application validation
• Software API development and
integration
• Low-level hardware test
Stress-Testing
Cost effectively execute long duration
use cases:
• Simultaneously test multiple releases on
a large number of devices
• Execute long duration use cases e.g.
stress, reliability testing
End-to-end validation from
playout to receiver:
• CAS
• EPG content
• OSS interactions
• Recommendation delivery
E2E Validation

Validate the Multiscreen Experience
• Validate QoE across the widest range of devices
‐ STB, iOS, Android, Xbox, PlayStation, Connected TV, and media streamers
• Test on real devices to replicate end user experience
• Focus on real QoE metrics:
• Video startup speed
• Video quality
• Audio quality
• Test devices as a ‘group’ to simulate real user actions
‐ Test synchronization across devices. Test companion device interactions.
‐ Simulate the ‘home’ environment
Assure the customer experience across all devices
Replicate and measure customer interaction across
multiple linked devices
Execute complex, interconnected tasks with
minimal manual intervention
A/V Quality Measurement
Task:
Challenge:
Solution:

Quality of Experience (QoE) is defined by a number of different measurable parameters
What do we mean by ‘Quality’
1. Subjective Video quality measurement as defined in ITU-R BT.500 Mean Opinion Score (MOS)
2. Subjective Audio quality measurement as defined in ITU-T P.800 MOS
3. Performance of the user experience, for example:
a) Time taken to start playing back video
b) Number and length of video buffering gaps in the video
c) Time taken to populate on screen components (e.g. video catalog)
d) Responsive of menu interface
“In assessing quality,
users are quick to criticise,
slow to forgive”

Subjective MOS measurement is expensive and time consuming. An algorithm is required for
deterministic calculation of MOS scores on video without a reference
From Subjective to Objective Quality Measurement
Stream Captured
through StormTest
HDMI input
Frames are analyzed
individually
Jerkiness Blockiness Blur Contrast
0 0.313548 0.361336 0.160398
Distortions measurement is
done on each frame Pooling of
measures
Output
Quality
score
MOS 85.56123
Frame
Quality
83.762743
1721
Temporal
Frame
Quality
83.762743
1721
User input the codec being
used: MPEG2/H264

Video Quality Analysis – Quality Score
Frame Quality
– Perceived quality of each frames
a.Jerkiness + b.Blockiness + D.Blur +…
Temporal Frame Quality
– Take into account human visual system “quick to criticize, slow
to forgive”
MOS Mean opinion score
– Integration over 5s of frame quality scores
𝑡
𝑡+5𝑠
𝑓𝑟𝑎𝑚𝑒 𝑞𝑢𝑎𝑙𝑖𝑡𝑦 + t
Scale:
< 20 Bad Quality
20 - 40 Poor Quality
40 - 60 Rather Good Quality
60 - 80 Good Quality
80 - 100 Excellent Quality

Consistent, continuous MOS measurement
0
10
20
30
40
50
60
70
80
90
100
0 200 400 600 800 1000 1200 1400 1600 1800
MOS
FRAME NUMBER
MOS VS BPS
1000 2000 3000 9000 12000

Full frame rate video quality analysis testing scenarios, combining
the strengths of StormTest and Perfecto
Mobile devices
connected to
mirroring devices via
wifiSTBs, consoles and media
players connected to LAN
A/V capture and
analysis
Automation
Python automation test
script controlling devices
in StormTest and Perfecto
racks
Whole Home Environment

Automation of Video Quality Measurement delivers the ability to:
Benefits of Video Quality Analysis
• Quickly assess the severity and troubleshoot video quality problems
• Monitor in real time the quality of a linear or on-demand video services
• Accurately determine consumers perceived video QoE
• Set quality thresholds and validate video quality against multiple performance criteria e.g.
macro-blocking, blurring, etc
• Accurately compare quality factors on video clips across different devices and encoding
profiles

Video Quality – What it means to BT
 Mobile application landscape is becoming more complex with increase in the number of mobile
devices and Operating Systems.
 BT’s business has been continuously evolving and it’s focus is on delivering the digital apps to the
customers for Set Top Boxes and Mobile devices with consistent user experience.
 With all the efforts and cost invested in design, development and test phases – the aim is to provide a
quality product to end user. Thus as a business, BT sees video quality becoming an intrinsic factor for
customer satisfaction (specially during sport matches).
 We are using StormTest and Perfecto for functional test automation and are exploring further options
to make the best use of these tools to measure the video quality aspects.

• Sign up for “Factors”, Perfecto’s quarterly reference guide for planning,
building and releasing apps.
http://info.perfectomobile.com/factors-magazine.html
• Watch Perfecto’s next webinar: http://info.perfectomobile.com/Four-
Milestoneson-the-Path-to-Efficient-DevOps-Webinar.html
• View additional resources on StormTest, including data sheets and
brochures at
https://www.accenturestormtest.com/
• Sign up for the next Perfecto Mobile webinar:

Video Testing Best Practices: How to Guarantee High-Quality Video for your Customers

More Related Content

What's hot (20)

Similar to Video Testing Best Practices: How to Guarantee High-Quality Video for your Customers (20)

More from Perfecto by Perforce (20)

Recently uploaded (20)

Video Testing Best Practices: How to Guarantee High-Quality Video for your Customers