Evaluation of Quality of Experience of ABR Schemes in Gaming Stream

Quality of Experience Evaluation of Stall Events, and
Quality Switches in Game Streaming
Game Studies and Engineering
Reza Ebrahimi
Supervisors: Christian Timmerer, Babak Taraghi
Nov 29th, 2023

Presentation Outline
1. Introduction
2. Purpose
3. Methods
4. Statistical Analysis
5. Results
6. Discussion
7. Conclusions
2

Introduction
As the Internet becomes increasingly central to multimedia services, the
Quality of Experience (QoE) in game streaming has gained importance. My
thesis focuses on the critical aspects of QoE, particularly in the context of
HTTP Adaptive Streaming (HAS).
3

Thesis Objectives and Goals
Investigate the impact of QoE characteristics in three game streaming scenes (Normal/Action). The
study examines how quality switching and stall events impact QoE in game streaming by performing
three scenarios. Three subjective quality assessments:
1- Quality Switching vs Stall Events (QvS)
2- Multiple Short Stalls vs Long Stalls (SSvLS)
3- Noticeable Quality Switching (NQS)
4

State of the Art
➢ Investigation on latency and packet loss in cloud gaming. Packet
loss was the most significant component of QoE in slow-paced
and medium-paced. Fast-paced games are more delay sensitive
than slower-paced
➢ Investigation on Multiplayer Online Role-Playing Games packet
loss. Packet loss in both directions, client-to-server, and server-to-
client, significantly impacts players’ QoE
An Evaluation of Delay and Packet Loss on QoE in Cloud Gaming
1. V. Clincy and B. Wilgor, “Subjective evaluation of latency and
packet loss in a cloudbased game,” in 2013 10th International
Conference on Information Technology: New Generations. IEEE,
2013, pp. 473–476.
2. K.-T. Chen, P. Huang, and C.-L. Lei, “How sensitive are online
gamers to network quality?” Communications of the ACM, vol.
49, no. 11, pp. 34–38, 2006.
5

State of the Art
➢ Investigation on the bitrate impacts on the quality of experience
of active participants and passive spectators in cloud gaming
systems. Passive viewers were more critical of video quality
➢ The impact of stream bitrate on the quality of experience for
Steam's In-Home streaming was examined in 2015 in both a first-
person shooter (FPS) and an action RPG games. A significant
correlations between video bitrate and perceived graphics quality.
➢ An effects of video quality variations in a cloud gaming
environment using self-assessment questionnaires and
electroencephalography (EEG) in low and high quality.
Investigations into Video Quality and Bitrate in Game Streaming
1. A. Sackl, R. Schatz, T. Hossfeld, F. Metzger, D. Lister, and R.
Irmer, “Qoe management made uneasy: The case of cloud
gaming,” in 2016 IEEE International Conference on
Communications Workshops (ICC). IEEE, 2016, pp. 492–497.
2. I. Slivar, M. Suznjevic, and L. Skorin-Kapov, “The impact of video
encoding parameters and game type on qoe for cloud gaming:
A case study using the steam platform,” in 2015 Seventh
International Workshop on Quality of Multimedia Experience
(QoMEX). IEEE, 2015, pp. 1–6.
3. J. Beyer, R. Varbelow, J.-N. Antons, and S. M ̈oller, “Using
electroencephalography and subjective self-assessment to
measure the influence of quality variations in cloud gaming,” in
2015 Seventh International Workshop on Quality of Multimedia
Experience (QoMEX). IEEE, 2015, pp. 1–6.
6

State of the Art
➢ An objective and subjective study conducted to assess the impact
of stall events and quality switches on the perceived and predicted
Quality of Experience (QoE) in HTTP Adaptive Streaming.
➢ A comprehensive multilayer evaluation model based on classic
network quality of service (QoS) and application QoS The paper
reveals that shorter stall events have a higher negative impact on
QoE of HTTP video streaming.
➢ A video complexity analyzer (VCA) provides an efficient method
for determining the optimal encoding decisions for each video
segment by analyzing the spatial and temporal complexity of the
video.
Impact of Buffering, Stall Events, and Quality Switching on Streaming Video QoE
1. B. Taraghi, M. Nguyen, H. Amirpour, and C. Timmerer, “Intense:
In-depth studies on stall events and quality switches and their
impact on the quality of experience in http adaptive streaming,”
IEEE Access, vol. 9, pp. 118 087–118 098, 2021.
2. Wang, R., Geng, Y., Ding, Y., Yang, Y., & Li, W. (2014, September).
Assessing the quality of experience of HTTP video streaming
considering the effects of pause position. In The 16th Asia-
Pacific Network Operations and Management Symposium (pp.
1-4). IEEE.
3. Menon, V. V., Feldmann, C., Amirpour, H., Ghanbari, M., &
Timmerer, C. (2022, June). VCA: Video complexity analyzer. In
Proceedings of the 13th ACM Multimedia Systems Conference
(pp. 259-264).
7

State of the Problem and Gaps
• Limited number of literature review studies have been conducted on QoE of video game
streaming and there is still a lot of research potential in this area.
• No study has been done comparing stall events and quality switching with a specific focus on
different genres, camera view and the temporal information or component of scenes in game
streaming.
8

Chapter 1: Introduction and Overview
•Key Aspects: Highlights main components of the thesis.
•Objectives & Goals: Defines the purpose and targets of the study.
•Structure Outline: Presents a roadmap of the thesis content.
9

Research Questions
1. What is the end-user preference concerning quality switching and stall events in
normal and action scenes from different video game streaming?
2. What is the impact and preference of users in seeing stall events with different
patterns?
3. How is the effect of changing the stream quality from 6000 bitrate 1080p to 3000
bitrate 720p on QoE of the audience in different video games?
Three research questions
10

Game Streaming - Passive Spectators
✓ Share their hobby
✓ Passive user interaction
✓ Replaced to old activities
✓ Encouraging streamers to establish unique relationships with users
✓ Not only time but also spend money
This Photo by Unknown Author is licensed under CC BY-SA-NC
One of the most notable examples is “Twitch”
The platforms provide their users with channels through which they can stream their favorite
entertainment live while others watch a game, chat, and interact with the broadcaster.
CHALLENGES: Avoid Stall-Events Keep The Best Quality
11

Chapter 2: Background and Context
1. QoE Streaming State-of-the-Art Review
2. Video Game Genre: Overview Game Genre challenges and selection for the thesis.
3. QoE in Passive Game Streaming: Analyzing user experience in a non-interactive environment.
4. Video Compression & Encoding: Techniques and practices in video streaming.
5. HTTP Adaptive Streaming: Exploration of adaptive streaming protocols.
6. Metrics and Tools: Introduction to methods and instruments used in this research.
12

Game Genres
FPS Genre Element Description
Gameplay Mechanics
Fast - paced action , shooting as the primary
gameplay
Camera Perspective
First - person view , player experiences the
game through the
eyes of the character
Pace High intensity and rapid gameplay
Aesthetics Realistic graphics , immersive environments
Dynamics
Player - driven interactions , strategic decision
- making
Player Demographics
Attracts players who enjoy action - packed
experiences
Streaming Experience
Hypothesis
The FPS genre provides an exciting and
engaging streaming
experience that draws viewers into the action
.
Adventure Genre
Element
Description
Gameplay Mechanics
Exploration , puzzle - solving , narrative -
driven
Camera Perspective
Third - person view , player controls a
character
Pace
Variable, includes both slow and fast-
paced moments
Aesthetics
Diverse environments , atmospheric
design
Dynamics
Story - driven choices , character
progression
Player Demographics
Appeals to players who enjoy story – rich
experiences
Streaming Experience
Engaging narrative , captivates viewers
with
unfolding story
MOBA Genre
Element
Description
Gameplay Mechanics
Team - based multiplayer battles , strategic
objectives , hero selection
Camera Perspective
Top - down view , players have an aerial
perspective of the battlefield
Pace
Fast - paced gameplay , intense battles with
frequent action and
decision - making
Aesthetics
Diverse fantasy or sci - fi settings , colorful
visuals , unique hero designs
Dynamics
Team coordination and communication ,
competitive and dynamic
matches
Player Demographics
Attracts competitive players who enjoy
strategic team - based
gameplay
Streaming
Experience
Hypothesis
The MOBA genre provides an action - packed
and competitive
streaming experience , attracting viewers
who enjoy intense battles
and strategic gameplay .
Sources [17], [18], [19], [20]
13

Chapter 3: Methodology and Experimental Setup
•Video Sequences: Describes the video content used in quality assessments.
•Subjective Quality Assessments: QvS, SSvLS, NQS
•Evaluation Setup: Outlines the framework for conducting assessments.
•Participant Recruitment: Process of using Amazon Mechanical Turk (MTurk)
•Implementation Details: Provides comprehensive technical specifics of the
methodology.
14

Quality Switching vs Stall Event Evaluation (QvS)
Set A - Case 1 and Case 2: Normal Scenes
Set B – Case 1 and Case 2: Action Scenes
Video Duration: 30 seconds
Total videos: 12 videos
Survey Duration: 20 minutes
QoE factors: 8 seconds lower quality vs 6 seconds stalls
Set A – Normal Scene Set B – Action Scene
Audio Coding: AAC
Video Coding: AVC/H.264
Total Participants Number: 69
Valid Participants: 19
Data Analyse: The repeated measures ANOVA
15

Longer Stall Events vs. Short Stall Events Evaluation (SSvLS)
Contents: An FPS Video (Valorant)
QoE factors: 5 different stall events’ patterns
Stall Event Pattern for 60 Seconds
Audio Coding: AAC
16

Noticeable Quality Switching Evaluation (NQS)
Contents: 3 different video games (including normal and action scenes)
QoE factors: Quality Switching
Noticeable Quality Switching Evaluation (QS) 6000 kbps
1080p resolution to 3000 kbps 720p resolution
Audio Coding: AAC
17

Experimental Setup
Contents Genres Camera
Temporal
Information
(h)
NQS QvS SSvLS
Valorant FPS 1st-person view Fast
Grand Theft Auto V Adventure game 3rd-person view Medium
League of Legends MOBA Isometric view Slow
FPS: First Person Shooting
MOBA: Multiplayer Online Battle Arena Video Games
MQS: Minimum Quality Switching
QvS: Quality vs Stall
SSvLS: Short Stalls vs Long Stalls
Table of Game Contents with Experimental Setup
18

Experimental Setup
Comparison of spatial information (E) / temporal information (h) in three video sequences GTA V
(Orange), Valorant (Gray) and League of Legends (Blue)
E
h
VCA: Video Complexity Analyzer
19

Experimental Setup
Bitrate Ladder and Encoding Configurations Table
TWITCH APPLE NETFLIX Selected
Resolution Bitrate Resolution Bitrate Resolution Bitrate Resolution Bitrate
1920x1080 6000 kbps 1920x1080 7800 kbps 1920x1080 5800 kbps 1920x1080 6000 kbps
1280x720 3000 kbps 1280x720 3000 kbps 1280x720 2350 kbps
960x540 2000 kbps 720x480 1750 kbps
768x432 730 kbps 640x480 1050 kbps
640x360 365 kbps 512x384 560 kbps
20

Subject Evaluation Setup Procedure
1. Web Server is live by starting the Node.js npm
2. MTurk workers click on the link
3. AWS EC2 return the required libraries, HTML, and JavaScript
files
4. Participants download the test sequences from S3 bucket in
their browser by IndexedDB (avoid user network fluctuation)
21

Subject Evaluation Setup
5. Minimum screen size, 1280 pixels in width and 720 pixels in
height
6. Continue pop-up button when the test sequence ends
7. No video control bar
8. Subjects’ votes will be captured and stored in the MongoDB
database
9. Reliability question included
10. Questions must answer
11. Next test sequence loaded in background
22

Subject Evaluation Setup
12. A completion code generate for each participants (using UUID)
13. Each campaign has its time constraints
14. Wrong answers to reliability question will remove after downloading the DB
Using UUID to generate completion code
23

Chapter 4: Methodology and Experimental Setup
1. Analysis of Subjective Quality Assessments: Examines key findings from the conducted studies.
2. Data Analysis Techniques: Describes the methodologies used for analyzing results.
3. Validation of MTurk Assessments: Details how evaluations meet research requirements.
4. Results Presentation: Showcases outcomes from the three subjective quality assessments.
24

Quality Switching vs Stall Event Evaluation (QvS)
Results QoE factors: 8 seconds lowest quality vs 6 seconds stalls
Mean opinion score QoE between quality switching (blue bar) and stall events
(orange line) in normal scenes as Set A (the first three) and action scenes as Set B (last
three) for 30 seconds playback 25

Statistical Analysis
Analysis of variance (One-way repeated-measures ANOVA) comparing stall events
patterns to each other in the SSvLS survey.
A B df Mean Square F P-Value
(6-1) (.5-12) 1 0.074 2.08 0.161
(1-6)
(.5-12) 1 0.296296 4.521 0.043
(6-1) 1 0.666667 7.428 0.011
(3-1)
(.5-12) 1 0.296 4.521 0.043
(6-1) 1 0.666 7.428 0.011
(1-6) 1 Same result Same result Same result
(1-3)
(.5-12) 1 0.462 5.909 0.022
(6-1) 1 0.907 9.1 0.005
(1-6) 1 0.018 1 0.326
(3-1) 1 0.018 1 0.326
(0-0)
(.5-12) 1 1.851 15.294 0.000
(6-1) 1 2.666 20.8 0.000
(1-6) 1 0.666 7.428 0.011
(3-1) 1 0.666 7.428 0.011
(1-3) 1 0.462 5.909 0.022
26

Chapter 5: Insightful Analysis and Key Findings
•Deep Analysis: Examines subjective quality assessments in detail.
•Main Finding: Quality switching is favored over stall events in all game genres.
•Impact on QoE: Quality switching has a negligible effect on Quality of Experience.
•Game Genre Insights: Specifically in first-person shooters:
• Preference for multiple short stalls over one long stall event.
•Recommendations for Future Work: Offers guidance based on the study's conclusions.
27

Summary Results - Quality Switching vs Stall Events
The rate of QoE in watching the quality switching stream is higher than the stall event in
all games. Stall event had a more negative impact on the users’ QoE than the quality
switching. This means that users tend to see quality switching more than stall event in
different games in all scenes (normal and action).
First Research Question Answer
28

Summary Results - Short Multiple Stall Events vs One Long Stall Event
Multiple short stall events are preferred over a longer stall event with the same total
duration in FPS game streaming
• Contrast to “INTENSE: In-Depth Studies on Stall Events and Quality Switches and Their
Impact on the Quality of Experience in HTTP Adaptive Streaming” paper where authors
found a longer stall event is preferred over multiple short stall events.
• Contrast to the conclusion of “Assessing the quality of experience of HTTP video
streaming considering the effects of pause position,” paper where authors found
shorter stall events had a greater negative impact on the QoE
Second Research Question Answer
29

Summary Results - Noticeable Quality Switching
1- More than 52% of the participants were unaware of the quality change while watching
the games.
2- The highest percentage of users' unawareness was in LOL game compared to other
games. This means that games with lower temporal information (i.e., MOBA genre) less
notice to the quality change compared to other games.
3- In first-person shooting games like Valorant with high temporal information, users
noticed a change in quality.
4- Participants’ QoE who noticed the quality switching was not significant or negatively
affected.
Third Research Question Answer
30

Resources
[1] “Adaptive Bitrate Streaming " What Is It? [2022].” Bitmovin, January 15, 2021. https://bitmovin.com/adaptive-streaming/.
[2] GTASeriesVideos. “GTA 5 PS5 - Mission #47 - Minor Turbulence [Gold Medal Guide - 4K 60fps].” YouTube. YouTube, March 26, 2022.
https://www.youtube.com/watch?v=3WjuidQg1Bg%5C&%3Bt=654s.
[3] Throneful. “Valorant (2021) - Gameplay (PC UHD) [4K60FPS].” YouTube. YouTube, January 5, 2021. https://www.youtube.com/watch?v=5RSdUErFgX0.
[4] EpicSkillshot. “FNC vs T1 | Day 2 Lol Worlds 2022 Main Group Stage | Fnatic vs T1 - Groups Full Game.” YouTube. YouTube, October 8, 2022.
https://www.youtube.com/watch?v=ctogqoYowDI.
[5] Claypool, Mark. "Motion and scene complexity for streaming video games." In Proceedings of the 4th International Conference on Foundations of Digital
Games, pp. 34-41. 2009.
[6] V. Clincy and B. Wilgor, “Subjective evaluation of latency and packet loss in a cloudbased game,” in 2013 10th International Conference on Information
Technology: New Generations. IEEE, 2013, pp. 473–476.
[7] K.-T. Chen, P. Huang, and C.-L. Lei, “How sensitive are online gamers to network quality?” Communications of the ACM, vol. 49, no. 11, pp. 34–38, 2006.
[8] Sackl, R. Schatz, T. Hossfeld, F. Metzger, D. Lister, and R. Irmer, “Qoe management made uneasy: The case of cloud gaming,” in 2016 IEEE International
Conference on Communications Workshops (ICC). IEEE, 2016, pp. 492–497.
[9] Brunnström, K., Beker, S. A., De Moor, K., Dooms, A., Egger, S., Garcia, M. N., ... & Zgank, A. (2013). Qualinet white paper on definitions of quality of
experience.
32

Resources
[10] I. Slivar, M. Suznjevic, and L. Skorin-Kapov, “The impact of video encoding parameters and game type on qoe for cloud gaming: A case study using the steam
platform,” in 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX). IEEE, 2015, pp. 1–6.
[11] J. Beyer, R. Varbelow, J.-N. Antons, and S. M ̈oller, “Using electroencephalography and subjective self-assessment to measure the influence of quality variations in
cloud gaming,” in 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX). IEEE, 2015, pp. 1–6.
[12] B. Taraghi, M. Nguyen, H. Amirpour, and C. Timmerer, “Intense: In-depth studies on stall events and quality switches and their impact on the quality of experience
in http adaptive streaming,” IEEE Access, vol. 9, pp. 118 087–118 098, 2021.
[13] J. Allard, A. Roskuski, and M. Claypool, “Measuring and modeling the impact of buffering and interrupts on streaming video quality of experience,” in Proceedings
of the 18th International Conference on Advances in Mobile Computing & Multimedia, 2020, pp. 153–160.
[14] T. Hoßfeld, S. Egger, R. Schatz, M. Fiedler, K. Masuch, and C. Lorentzen, “Initial delay vs. interruptions: Between the devil and the deep blue sea,” in 2012 Fourth
International Workshop on Quality of Multimedia Experience. IEEE, 2012, pp. 1–6.
[15] N. Staelens, P. Coppens, N. Van Kets, G. Van Wallendaef, W. Van den Broeck, J. De Cock, and F. De Turek, “On the impact of video stalling and video quality in the
case of camera switching during adaptive streaming of sports content,” in 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX). IEEE,
2015, pp. 1–6.
[16] Wang, R., Geng, Y., Ding, Y., Yang, Y., & Li, W. (2014, September). Assessing the quality of experience of HTTP video streaming considering the effects of pause
position. In The 16th Asia-Pacific Network Operations and Management Symposium (pp. 1-4). IEEE.
[17] C. Therrien, “Inspecting video game historiography through critical lens: Etymol- ogy of the first-person shooter genre by carl therrien,” Game Studies, vol. 15, 01
2015.
33

Resources
[18] M. J. Wolf, The medium of the video game. Chapter 6. University of Texas Press, 2001, pp. 113–134. [Online]. Available: https://doi.org/10.7560/791480
[19] S. Egenfeldt-Nielsen, J. H. Smith, and S. P. Tosca, Understanding Video Games: The Essential Introduction, 4th ed. Routledge, 2019, ch. 5.
[20] “Riot games’ competitive 5v5 character-based tactical shooter,” Accessed: 2022-11-04. [Online]. Available: https://playvalorant.com/en-us/
34

Evaluation of Quality of Experience of ABR Schemes in Gaming Stream

More Related Content

Similar to Evaluation of Quality of Experience of ABR Schemes in Gaming Stream (20)

More from Alpen-Adria-Universität (20)

Recently uploaded (20)

Evaluation of Quality of Experience of ABR Schemes in Gaming Stream