Quality of Experience of Web-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing
4 likes1,544 views
This document summarizes a presentation on evaluating the quality of experience (QoE) of adaptive HTTP streaming clients like DASH in real-world environments using crowdsourcing. The study evaluated three clients (DASH-JS, dash.js, YouTube) playing video content with different quality representations. Results found that DASH-JS had high startup times but low stalls and good throughput/QoE, while dash.js had low startup but high stalls and lower throughput/QoE. YouTube performed best with low startup, few stalls, and highest throughput and QoE. The methodology and results provide insights into QoE aspects of DASH clients over the open internet.
![Quality
of
Experience
• Quality
of
Experience
– “…
is
the
degree
of
delight
or
annoyance
of
the
user
of
an
applica+on
or
service…”
– Factors
influencing
/
features
of
QoE
may
lead
to
applica+on-‐specific
defini+ons
• Subjec+ve
quality
assessments
– Laboratory
environment
[ITU-‐T
B.500
/
P.910]
– Crowdsourcing
with
special
plaVorms
or
social
networks
• QoE
of
DASH-‐based
services
– Startup
delay
(low)
– Buffer
underrun
/
stalls
(zero)
– Quality
switches
(low)
and
media
throughput
(high)
P.
Le
Callet,
S.
Möller
and
A.
Perkis,
eds.,
“Qualinet
White
Paper
on
Defini+ons
of
Quality
of
Experience
(2012)”,
European
Network
on
Quality
of
Experience
in
Mul>media
Systems
and
Services
(COST
Ac>on
IC
1003),
Lausanne,
Switzerland,
Version
1.2,
March
2013."
December
2,
2014
VideoNext
2014,
Sydney
10](https://image.slidesharecdn.com/videonext2014timmerer-141201212617-conversion-gate02/85/Quality-of-Experience-of-Web-based-Adaptive-HTTP-Streaming-Clients-in-Real-World-Environments-using-Crowdsourcing-10-320.jpg)
![Methodology
• Quality
of
Experience
…
– Mean
Opinion
Score
[0..100]
– [other
objec+ve
metrics:
start-‐up
+me,
throughput,
number
of
stalls]
• …
Web-‐based
Adap+ve
HTTP
Streaming
Clients
…
– HTML5+MSE:
DASH-‐JS
(dash.itec.aau.at),
dash.js
(DASH-‐IF,
v1.1.2),
YouTube
• …
Real-‐World
Environments
…
– DASH-‐JS,
dash.js
hosted
at
ITEC/AAU
(~
10Gbit/s)
– YouTube
hosted
at
Google
data
centers
– Content:
Tears
of
Steel
@
144p
(250
kbit/s),
240p
(380
kbit/s),
360p
(740
kbit/
s),
480p
(1308
kbit/s),
and
720p
(2300
kbit/s);
segment
size:
2s
– Users
access
content
over
the
open
Internet
• …
Crowdsourcing
– Campaign
at
Microworker
plaVorm
(others
also
possible:
Mechanical
Turk,
social
networks)
limited
to
Europe,
USA/Canada,
India
– Screening
Techniques:
Browser
fingerprin+ng,
s+mulus
presenta+on
+me,
QoE
ra+ngs
and
pre-‐ques+onnaire
December
2,
2014
VideoNext
2014,
Sydney
11](https://image.slidesharecdn.com/videonext2014timmerer-141201212617-conversion-gate02/85/Quality-of-Experience-of-Web-based-Adaptive-HTTP-Streaming-Clients-in-Real-World-Environments-using-Crowdsourcing-11-320.jpg)
Quality of Experience of Web-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing
- 1. Slides: hVp://www.slideshare.net/chris8an.8mmerer Quality of Experience of Web-‐based Adap8ve HTTP Streaming Clients in Real-‐World Environments using Crowdsourcing Benjamin Rainer and Chris8an Timmerer Alpen-‐Adria-‐Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Informa8on Technology (ITEC) w Mul8media Communica8on (MMC) w Sensory Experience Lab (SELab) h"p://blog.+mmerer.com w h"p://dash.itec.aau.at/ w h"p://selab.itec.aau.at mailto:chris+an.+mmerer@itec.uni-‐klu.ac.at December 2, 2014
- 2. Outline • Introduc+on • How to evaluate DASH and QoE • Methodology • Results • Conclusions December 2, 2014 VideoNext 2014, Sydney 2
- 3. Mul+media is Predominant on the Internet • Real-‐+me entertainment – Streaming video and audio – More than 50% of Internet traffic at peak periods • Popular services – NeVlix (34.9%), YouTube (14.0%), Amazon Video (2.6%), Hulu (1.4%) – All delivered over-‐the-‐top (OTT) – MPEG Dynamic Adap+ve Streaming over HTTP Global Internet Phenomena Report: 2H 2014 December 2, 2014 VideoNext 2014, Sydney 3
- 4. Over-‐The-‐Top – Adap+ve Media Streaming • In a nutshell … Adapta8on logic is within the client, not norma8vely specified by the standard, subject to research and development C. Timmerer and A. C. Begen, “Over-‐the-‐Top Content Delivery: State of the Art and Challenges Ahead”, In Proceedings of the ACM interna+onal conference on Mul+media (MM '14), Orlando, FL, USA, Nov. 2014. h"p://www.slideshare.net/chris+an.+mmerer/over-‐the-‐top-‐content-‐delivery-‐state-‐of-‐the-‐art-‐and-‐challenges-‐ahead December 2, 2014 VideoNext 2014, Sydney 4
- 5. MPEG Dynamic Adap+ve Streaming over HTTP What is specified – and what is not? Media Presenta+on on HTTP Server Media Presenta8on DASH-‐enabled Client Descrip8on Segment … . . . . . . Segment … Segment … . . . . . . Segment … … Segments located by HTTP-‐URLs DASH Control Engine HTTP/1.1 MPD Parser On-‐8me HTTP requests to segments HTTP Client Media Engine December 2, 2014 VideoNext 2014, Sydney 5
- 6. MPEG Dynamic Adap+ve Streaming over HTTP What is specified – and what is not? Media Presenta+on on HTTP Server Media Presenta8on DASH-‐enabled Client Descrip8on Segment … . . . . . . Segment … Segment … . . . . . . Segment … … Segments located by HTTP-‐URLs DASH Control Engine HTTP/1.1 MPD Parser On-‐8me HTTP requests to segments HTTP Client Media Engine December 2, 2014 VideoNext 2014, Sydney 6
- 7. DASH Data Model Segment Info Initialization Segment http://bitmov.in/500/init.mp4 Media Presentation Period, start=0s … Period, start=100s … Period, start=200s … … Period start=100 baseURL=http://… bitmov.in/ AdaptationSet 1 500-1500 kbit/s AdaptationSet 2 1500-3000 kbit/s … Media Segment 1 start=100s http://bitmov.in/500/seg-1.m4s Media Segment 2 start=102s http://bitmov.in/500/seg-2.m4s Media Segment 3 start=104s http://bitmov.in/500/seg-3.m4s Media Segment 50 start=198s http://bitmov.in/500/seg-50.m4s AdaptationSet 1 width=640-1280 height=360-720 … Representation 1 500 Kbit/s Representation 2 1500 Kbit/s … Representation 2 bandwidth=1500 kbit/s width=960, height=540 … Segment Info duration=2s Template: 500/seg-$Number$.m4s Initialization: 500/init.mp4 December 2, 2014 VideoNext 2014, Sydney 7
- 8. December 2, 2014 VideoNext 2014, Sydney 8
- 9. How to evaluate DASH? • Methodology – Dataset, tools (see backup slides for details) – Common evalua+on setup – Bandwidth traces (real/synthe+c) vs. models • Metrics – Average media bitrate/throughput at the client – Number of representa+on/quality switches – Number of stalls (in seconds) – buffer level C. Mueller, S. Lederer, C. Timmerer, “An Evalua+on of Dynamic Adap+ve Streaming over HTTP in Vehicular Environments”, In Proceedings of the Fourth Annual ACM SIGMM Workshop on Mobile Video (MoVid12), Chapel Hill, North Carolina, February 2012. December 2, 2014 VideoNext 2014, Sydney 9
- 10. Quality of Experience • Quality of Experience – “… is the degree of delight or annoyance of the user of an applica+on or service…” – Factors influencing / features of QoE may lead to applica+on-‐specific defini+ons • Subjec+ve quality assessments – Laboratory environment [ITU-‐T B.500 / P.910] – Crowdsourcing with special plaVorms or social networks • QoE of DASH-‐based services – Startup delay (low) – Buffer underrun / stalls (zero) – Quality switches (low) and media throughput (high) P. Le Callet, S. Möller and A. Perkis, eds., “Qualinet White Paper on Defini+ons of Quality of Experience (2012)”, European Network on Quality of Experience in Mul>media Systems and Services (COST Ac>on IC 1003), Lausanne, Switzerland, Version 1.2, March 2013." December 2, 2014 VideoNext 2014, Sydney 10
- 11. Methodology • Quality of Experience … – Mean Opinion Score [0..100] – [other objec+ve metrics: start-‐up +me, throughput, number of stalls] • … Web-‐based Adap+ve HTTP Streaming Clients … – HTML5+MSE: DASH-‐JS (dash.itec.aau.at), dash.js (DASH-‐IF, v1.1.2), YouTube • … Real-‐World Environments … – DASH-‐JS, dash.js hosted at ITEC/AAU (~ 10Gbit/s) – YouTube hosted at Google data centers – Content: Tears of Steel @ 144p (250 kbit/s), 240p (380 kbit/s), 360p (740 kbit/ s), 480p (1308 kbit/s), and 720p (2300 kbit/s); segment size: 2s – Users access content over the open Internet • … Crowdsourcing – Campaign at Microworker plaVorm (others also possible: Mechanical Turk, social networks) limited to Europe, USA/Canada, India – Screening Techniques: Browser fingerprin+ng, s+mulus presenta+on +me, QoE ra+ngs and pre-‐ques+onnaire December 2, 2014 VideoNext 2014, Sydney 11
- 12. Results: QoE December 2, 2014 VideoNext 2014, Sydney 12
- 13. Results: Media Throughput December 2, 2014 VideoNext 2014, Sydney 13
- 14. Results: Start-‐Up Time December 2, 2014 VideoNext 2014, Sydney 14
- 15. Results: Number of Switches December 2, 2014 VideoNext 2014, Sydney 15
- 16. Results: Number of Stalls December 2, 2014 VideoNext 2014, Sydney 16
- 17. Results: Summary • DASH-‐JS – High start-‐up +me – Low number of stalls – Good throughput, QoE • dash.js – Low start-‐up +me – High # stalls – Low throughput – Low QoE • YouTube – Low start-‐up +me – Low number of stalls – Best throughput, QoE December 2, 2014 VideoNext 2014, Sydney 17
- 18. Conclusions • QoE evalua+on of DASH-‐like systems in real-‐world environments using crowdsourcing – Detailed methodology described in the paper – Results indicate that the delivered representa+on bitrate (media throughput) and the number of stalls are the main influence factors on the QoE – Results confirmed by previous evalua+ons but within controlled environments – Evidence about QoE aspects of DASH-‐enabled Web clients within real-‐ world environments – Feasibility of using crowdsourcing for subjec+ve quality assessments • Future work – Comprehensive evalua+on of various adapta+on logics (both objec+ve and subjec+ve) and – the impact of dedicated delivery infrastructures aiming to improve DASH-‐based services December 2, 2014 VideoNext 2014, Sydney 18
- 19. Thank you for your a"en+on ... ques+ons, comments, etc. are welcome … Priv.-‐Doz. Dipl.-‐Ing. Dr. Chris+an Timmerer Associate Professor Klagenfurt University, Department of Informa+on Technology (ITEC) Universitätsstrasse 65-‐67, A-‐9020 Klagenfurt, AUSTRIA chris+an.+mmerer@itec.uni-‐klu.ac.at h"p://research.+mmerer.com/ Tel: +43/463/2700 3621 Fax: +43/463/2700 3699 © Copyright: Chris>an Timmerer December 2, 2014 VideoNext 2014, Sydney 19
- 20. December 2, 2014 VideoNext 2014, Sydney 20
- 21. December 2, 2014 VideoNext 2014, Sydney 21
- 22. BACKUP SLIDES December 2, 2014 VideoNext 2014, Sydney 22
- 23. End-‐to-‐End DASH System Aspects • (Distributed) dataset – Full movie length in high quality – Various bitrate, resolu+ons, segment lengths (2-‐15s), (sub-‐)segments – Distributed: ini+al 3 sites, now 9 in Europe, USA, Taiwan • DASH encoder – Encoding + Mul+plexing + MPD genera+on – Fully configurable using a configura+on file – Enables batch processing – x264/ffmpeg + GPAC MP4Box S. Lederer, C. Müller, C. Timmerer, “Dynamic Adap+ve Streaming over HTTP Dataset”, In Proceedings of the ACM Conference on Mul+media Systems 2012, Chapel Hill, North Carolina, February 2012. // S. Lederer, C. Mueller, C. Timmerer, C. Concolato, J. Le Feuvre, K. Fliegel, “Distributed DASH Dataset”, In Proceedings of the ACM Conference on Mul+media Systems 2013, Oslo, Norway, 2013. December 2, 2014 VideoNext 2014, Sydney 23
- 24. End-‐to-‐End DASH System Aspects • Playback – VLC plugin (first implementa+on) – DASH-‐JS (HTML5 + MSE) – libdash / qtsampleplayer • MPD valida+on – XML schema valida+on – Xlink resolver & processing – Addi+onal valida+on rules (Schematron) • Experimental – DASH over Content-‐Centric Networks (CCN) – VLC + libdash C. Müller and C. Timmerer, “A VLC Media Player Plugin enabling Dynamic Adap+ve Streaming over HTTP”, In Proceedings of the ACM Mul+media 2011, Sco"sdale, Arizona, November 2011. // B. Rainer, S. Lederer, C. Müller, C. Timmerer, “A Seamless Web Integra+on of Adap+ve HTTP Streaming”, In Proceedings of the 20th European Signal Processing Conference 2012, Bucharest, Romania, August 2012. h"p://records.sigmm.ndlab.net/2013/04/open-‐source-‐column-‐dynamic-‐adap+ve-‐streaming-‐over-‐h"p-‐toolset/ December 2, 2014 VideoNext 2014, Sydney 24