A Framework for Adaptive Delivery of Omnidirectional Video

A Framework for Adaptive Delivery of
Omnidirectional Video
Priv.-Doz. Dr. Christian Timmerer
Alpen-Adria-Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Information
Technology (ITEC) w Multimedia Communication (MMC) w Sensory Experience Lab (SELab)
http://blog.timmerer.com w http://selab.itec.aau.at/ w http://dash.itec.aau.at w christian.timmerer@itec.aau.at
Chief Innovation Officer (CIO) at bitmovin GmbH
http://www.bitmovin.com w christian.timmerer@bitmovin.com
Human Vision and Electronic Imaging 2018, January 31, 2018
http://www.slideshare.net/christian.timmerer

Outline
• Introduction
• Framework
• Encoding options and omnidirectional HAS
• Standardization overview
• Conclusion
January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 2

Multimedia Traffic on the Internet
• Real-time entertainment: Streaming video and audio; >70% of Internet traffic at peak periods
• Forecast: Visual Networking Index (VNI) 2016-2021
– IP video traffic will be 82% of all consumer Internet traffic by 2021 (up from 73% in 2016); will grow threefold from 2016 to 2021
– Live Internet video will account for 13% of Internet video traffic by 2021; will grow 15-fold from 2016 to 2021
• Popular services
– YouTube (17.53%), Netflix (35.15%), Amazon Video (4.26%), Hulu (2.68%); all delivered over-the-top (OTT)
• More people now subscribe to Netflix (50.85M) than cable TV (48.61M) in the US (Q1 2017)
Global Internet Phenomena Report: 2016
https://www.businessinsider.com/netflix-more-popular-than-cable-us-chart-2017-6/

How does it work? In a nutshell…
Adaptation logic is within the
client, not normatively
specified by the standard,
subject to research and
development

Omnidirectional Video / 360° Video
Capturing Devices
• Stitching, Projection Formats
• Encoding, Encryption, Encapsulation
• Storage, Content Distribution, Delivery
• Processing, Decoding, Rendering, …
Consumer Devices

Functional Architecture
Encode, Encrypt,
Encapsulate
Decode, Decrypt,
Decapsulate
Project &
Render
Fuse, Stitch
& Edit
①
②
③
④
⑤
Store & Deliver
Capture Consume
Content Creation NetworkServer Client
Encoding
Encryption
Encapsulation
Storage
Delivery Decryption
Decapsulation
Decoding RenderingEditing
Processing Processing
Capture
Acquisition
Consumption
Distribution
Adaptive Delivery of Omnidirectional Video
Interaction
From ecosystem…
To buidling blocks…

Adaptive Streaming Options
• Traditional, viewport-agnostic streaming
– Simple, easy, deployed today
– Bandwidth waste, quality issues
• Viewport-adaptive streaming
– Multiple versions for predefined viewports
– Various projection techniques (pyramid)
– Bandwidth waste reduced, increased storage
and CDN costs, limited flexibility
• Tile-based streaming
– Use tiling technique of modern video codecs
– High complexity, full flexibility
– Multiple challenges
X. Corbillon, et al., "Viewport-adaptive navigable 360-degree video
delivery," 2017 IEEE International Conference on Communications
(ICC), Paris, 2017, https://doi.org/10.1109/ICC.2017.7996611
M. Graf, et al. 2017. Towards Bandwidth Efficient Adaptive
Streaming of Omnidirectional Video over HTTP: Design,
Implementation, and Evaluation. Proc. ACM MMSys'17.
https://doi.org/10.1145/3083187.3084016
C. Concolato, et al., "Adaptive Streaming of HEVC
Tiled Videos using MPEG-DASH," IEEE TCSVT, 2017.
https://doi.org/10.1109/TCSVT.2017.2688491

Tile-based Adaptive Streaming
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Encoding &
Packaging
Tile 7 Tile 8 Tile 9 Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile 7 Tile 8 Tile 9 Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
DifferentQuality
Representations
Delivery
MPEG-HEVC/H.265
Tiles in ISOBMFF
Adaptive
Player
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile-based streaming of VR/360°
content with MPEG-DASH SRD
…
…
…
…
Adaptive Streaming using
MPEG-DASH SRD
Head Mounted Displays
Browsers, Smart (Mobile) Devices
(Stereo) 2D, (Stereo) 3D

Encoding Options
• AVC dominates the market
• HEVC, VP9, AV1 support tiles
– Divides a picture into
independent, rectangular regions
– Tradeoff: bitrate, quality, flexibility
• Multiple tiling options available
– Uniform vs. non-uniform tiling
– Same vs. mixed resolutions
• New quality metrics, mostly based on PSNR but subjective
quality assessments/metrics increasing
Bitmovin 2017 Video Developer Report
https://bitmovin.com/2017-video-developer-report/
I. D.D. Curcio, et al. 2017. Bandwidth Reduction of Omnidirectional Viewport-Dependent Video Streaming
via Subjective Quality Assessment. Proc. AltMM'17. https://doi.org/10.1145/3132361.3132364
A. Zare, et al. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality
Applications. Proc. ACM MM'16. http://dx.doi.org/10.1145/2964284.2967292

Dataset
• Segment length / Intra period
– 1s (tiled content) vs 1, 2, 4s (monolithic content)
• Tiling pattern (columns × rows): 1×1, (i.e., tiles
monolithic), 3×2, 5×3, 6×4, and 8×5
• Resolution: 1920×960, 3840×1920 and 7680×3840
• Map projection: equirectangular format
• Quantization parameter: QP={22,27,32,37,42}
• Head motion recordings for V-PSNR evaluation

Bitrate Overhead due to Tiling
500 1000 1500 2000 2500
384042444648
Bitrate [kbps]
Y−PSNR[dB]
●
●
●
●
●
●
● 1x1 tiles
3x2 tiles
5x3 tiles
6x4 tiles
8x5 tiles
Tile Overhead for Resolution: 1920x960
Sequence: AssassinsCreed
2000 4000 6000 8000
30323436384042
Bitrate [kbps]
Y−PSNR[dB]
●
●
●
●
●
●
● 1x1 tiles
3x2 tiles
5x3 tiles
6x4 tiles
8x5 tiles
Tile Overhead for Resolution: 1920x960
Sequence: ExploreTheWorld

Bandwidth Requirements

Adaptive Streaming Issues
• Increasing number of segment/tile requests
– HTTP/2 server push, query parameters, proprietary protocols
– Additional functionality at server – breaks fundamental HAS requirements
• Low latency streaming
– Reducing segment size impacts coding efficiency (1s vs. 4s)
– CMAF chunks + other enhancements to enable sub-second latency
– Remember: live internet video will grow 15-fold from 2016 to 2021
• Viewport prediction
– Allows prefetching (caching) but cannot predict to much into future (1s)
– Impact on segment size but situation will get better the more data is available –
machine learning/AI will help
• Quality of Experience: QoE
– Still in its infancy but situation much better than one year ago
– Requires datasets, subjective studies, quality models, metrics
M. Xu, et al., "A subjective visual quality assessment method of panoramic
videos," Proc. ICME’17. https://doi.org/10.1109/ICME.2017.8019351
R. Schatz, et al., "Towards subjective quality of experience assessment
for omnidirectional video streaming," Proc. QoMEX’17,
https://doi.org/10.1109/QoMEX.2017.7965657
Y. Rai, at al. 2017. A Dataset of Head and Eye Movements for 360 Degree
Images. Proc. ACM MMSys’17. https://doi.org/10.1145/3083187.3083218
S. Petrangeli, et al. 2017. An HTTP/2-Based Adaptive Streaming
Framework for 360° Virtual Reality Videos. Proc. ACM MM'17.
https://doi.org/10.1145/3123266.3123453
N. Bouzakaria, et al., "Overhead and performance of low latency live
streaming using MPEG-DASH," Proc. IISA’14.
https://doi.org/10.1109/IISA.2014.6878732
C.-L. Fan, et al. 2017. Fixation Prediction for 360° Video Streaming in Head-Mounted
Virtual Reality. Proc. NOSSDAV'17. https://doi.org/10.1145/3083165.3083180
Stay tuned for QoMEX'18, MMSys'18, PV'18, ICME'18, MM'18…

Standardization Overview
R. Skupin, et al., "Standardization Status of 360 degree Video Coding and Delivery,” Proc. IEEE
VCIP’17.
C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE
Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017.
https://doi.org/10.1109/MCOMSTD.2017.1700038 B. Choi, et al., (editors) "Information Technology -- Coded Representation of Immersive Media
(MPEG-I) -- Part 2: Omnidirectional Media Format," ISO/IEC 23090-2 Final Draft International
Standard (FDIS), Dec. 2017.

Conclusions
• In general, we know how it works and how
to do it, but many challenges exist
• Optimization required everywhere from
creation to consumption
• Need to reduce complexity to foster adoption
and innovation
• This framework…
– Provides an overview;
– Identifies building blocks;
– Outlines state of the art;
– Raises research questions and helps with
respect to gap analysis.
January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 16http://www.diamandis.com/peters-laws
[10] When faced without a challenge, make one.
[13] When in doubt: THINK.
[26] If you can’t measure it, you can’t improve it.
[22] The day before something is a
breakthrough it’s a crazy idea.

Thank you for your attention
... questions, comments, etc. are welcome …
Priv.-Doz. Dipl.-Ing. Dr. Christian Timmerer
Associate Professor
Alpen-Adria-Universität Klagenfurt, Department of Information Technology (ITEC)
Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA
christian.timmerer@itec.uni-klu.ac.at
http://blog.timmerer.com/
Tel: +43/463/2700 3621 Fax: +43/463/2700 3699
© Copyright: Christian Timmerer

A Framework for Adaptive Delivery of Omnidirectional Video

More Related Content

What's hot (9)

Similar to A Framework for Adaptive Delivery of Omnidirectional Video (20)

More from Alpen-Adria-Universität (20)

Recently uploaded (20)

A Framework for Adaptive Delivery of Omnidirectional Video