Towards 3D visual quality assessment for future multimedia

1

Vers une mesure
de qualité visuelle 3D
pour le multimédia
de demain
Touradj Ebrahimi
Touradj.Ebrahimi@epfl.ch

CORESA
24-25 mai 2012, Lille, France

1

Vers une mesure
Towards 3Dvisuelle 3D
de qualité visual quality
assessment for future
pour le multimédia
de demain
multimedia
Touradj Ebrahimi
Touradj.Ebrahimi@epfl.ch

CORESA

2
Outline

• Motivations
• 3D perception
• 3D processing chain and sources of distortions
• 3D quality
– Subjective quality assessment
– Objective quality assessment
• Some last words

Anaglyph glasses recommended

CORESA

3

INTRODUCTION

CORESA

4
Motivation

• Evolution of visual content towards greater realism
– Black and white
– Color
– High resolution
– 3DTV
• 3 major factors for success
– Technology: Ability to capture, process and show 3D
– Content: Availability of interesting 3D content
– Quality: Attractive to consumers

CORESA

5
History of 3D
• 1840: Invention of stereoscopy and
stereoscope by C. Wheatstone
• 1890: First patent for 3D motion pictures
using stereoscope
• 1915: First 3D footage in cinema using
anaglyph glasses
• 1922: Invention of „Teleview“ a shutter
based technique
• 1936: First demonstration of polarization
based projection
• 1952: Golden era of 3D movies due to
invention of television
• 1961: Single film solution „Space-Vision
3D“ using polarization
• 1980: IMAX 70mm projectors for non-
fiction short films
• 2003: First full length 3D feature film for
IMAX screens by J. Cameron
• 2004: Animation „Polar Express“ makes
14 times more revenue in 3D than 2D
CORESA

6
Example of some recent 3D successes
• Movies
– Beowulf (2007)
– Avatar (2009)
• Music
– U2 3D (2008)
– In Concert 3D (2009)
• Documentary
– Biodiversity (2009)
• Sports
– NBA All Star Game (2009)
– Six Nations Cup (2010)
– FIFA World Cup (2010)
• Games
– 19 PS3 titles in 2010
• 3D Blu-ray

CORESA

7
Good vs. Bad 3D Quality
• Avatar (2009) • Clash of the Titans (2010)
– Combination of 3D shooting, – Shot completely in 2D and
motion capture and and CGI converted to 3D in 8 weeks
– Budget of 237 million USD – Budget of 125 million USD
– "It was an absolute marvel and I – "The film redefines 3-D but in the
am left in awe after seeing it.“ wrong way.”

CORESA

8

3D Perception

CORESA

9
3D perception

• Human visual system uses several depth cues
to build a mental model of a perceived 3D scene
• Understanding the human visual perception of
3D is an important aspect to develop 3D quality
metrics and to come up with efficient 3D
systems and services

CORESA

10
Depth cues

CORESA

11
Oculomotor depth cues

• Accommodation
– Change of the focal length of the lens
– Quite a weak depth cue and only effective for distances < 2 m
• Convergence
– Rotation of the eyes towards each other for closer objects
– Quite a weak depth cue and only effective for distances < 10 m
• Myosis
– Size of the pupil determines both amount of light and depth of
field (DOF)
– Very weak depth cue for short distances

CORESA

12
Monocular depth cues
• Mainly experiential and learned over
time
• Shadow
• Illuminations
• Relative sizes differences
• Aerial perspective
• Linear perspective
• Interposition
• Texture gradient
• Intensity gradient
• ...

CORESA

13
Stereopsis

• Human eyes are separated
horizontally by approx. 6.3 cm
• Existence of different retinal
images leads to binocular
disparity

CORESA

14
Binocular depth cues
• Two eyes observe scene from two slightly different angles
• Most important depth cue for medium viewing distances
• Basic idea behind any stereoscopic display technology
• 5%-10% of population have difficulties with binocular depth

CORESA

15
Motion parallax

• Motion cues are created
when the viewer moves
his eyes or head
• Relative object motion
around a fixation point
serves as depth cue
• Very important depth cue
for a large range of scene
depths

CORESA

16
Depth perception layers

CORESA

17

3D Processing chain and sources
of distortions
CORESA

18
3D processing chain

CORESA

19
3D creation

• Computer generated imagery (CGI)
• Stereoscopic cameras
• Video + depth sensor
• 2D to 3D conversion

CORESA

20
Stereoscopic camera

• Matching cameras
• Matching photography
– White balance
– Sensitivity
– Shutter speed
– Aperture
• Matching optics
– Focal length
• Matching geometry
– Distance
– Angle
• Synchronization
CORESA

21
Camera distance

• Single most important
parameter in stereoscopy
• Regulates strength of 3D
effect and object size
• Optimal camera distance
depends on many factors
– Near and far distance
– Focal length
– Maximum disparity

CORESA

22
Camera distance

CORESA

23
Camera convergence
• Second most important
parameter in stereoscopy
• Parallel camera setup
– All objects are in front of
screen
– Depth position can be set
later
• Converged camera setup
– Objects partially behind the
screen
– Depth position fixed to
objects where optical axes
converge
– May lead to keystone effect

CORESA

24
Camera convergence

CORESA

25
3D post-processing
• Geometric alignment
– Horizontal and vertical alignment of cameras may not be perfect
– Align images to avoid vertical disparities and adjust depth position
• Color adjustment
– White balance and exposure of individual cameras may differ
– Leads to color and brightness variations between stereo images
– Correction using histogram matching techniques

CORESA

26
3D scene representation

• Stereoscopic
– Left-Right
– Above-Below
– Interlaced
• Image + Depth
– Synchronized 2D and depth
image or video
• Multiview
– Several synchronized 2D video
streams
– Also possible to include depth for
each view

CORESA

27
3D compression standards
• Simulcast
– Possible with any video
coding standard
• Stereoscopic
– MPEG-4/AVC: Stereo SEI
message and different L/R
packings
• Image+depth
– MPEG-C Part A: Auxiliary data
representation for depth maps
• Multiview video coding (MVC)
– H.264/MVC: Extension of H.
264/AVC to multiview video
• Multiview+depth
– 3D Video Coding (3DVC)

CORESA

28
Asymmetric coding
• Individual views are coded independently
• Each view is coded with a different quality
• According to binocular suppression theory the perceived
quality is dominated by the higher quality view
• Any video coding standard can be used

CORESA

29
3D content displays (devices)

• Head Mounted Displays (HMD)
• Monitors
– Anaglyph
– Polarization
– Shutter
– Autostereoscopic
• Projectors
– Anaglyph
– Polarization
– Shutter

CORESA

30
3D content display (technologies)

• Stereograms
• Passive anaglyph
• Active shutter
• Passive polarization
– Linear
– Circular
• Autostereoscopic
– Single view
– Multi view

CORESA

31
Autostereoscopic displays
• Requires no glasses
• Sends the left and right views
directly to the eyes
• Lenticular lens
– Tiny cylindrical plastic lenses
attached to the front of the screen
– Does not support 2D mode
• Parallax barrier
– Fine grating of liquid crystals
placed in front of the screen
– Supports 2D/3D mode by
switching parallax barrier off/on

CORESA

32
Sweet spot and crosstalk

• Correct 3D perception
depends largely on the
viewers position and the
used display technology
• Wrong distance to the
screen leads to crosstalk
between the left and the
right images
• Wrong horizontal position
leads to inverted images
and causes headache

CORESA

33

3D Quality

CORESA

34
3D Quality
• Improved quality of experience (QoE) is an important aspect for
the success of any new technology such as 3DTV
• Emerging 3D video quality assessment needs to consider
additional aspects such as depth perception, immersion,
naturalness, and presence

CORESA

35
3D Quality

Nintendo Warns
Parents Not To Let
Children Under 6
Play 3DS
December 29th, 2010

CORESA

36
2D artifacts
• Not specific to 3D video but also present in 2D video
• Mainly degrade the 2D quality but may also affect depth
perception
• Examples include blocking, ringing, blurring, color bleeding,
jerkiness

CORESA

37
3D artifacts
• Distortions that lead to a perceptual difference between the
real 3D structure of a scene and its representation
• May have different effects on the depth perception from a
feeling of unnaturalness to visual discomfort and headache
• Includes keystone distortion, puppet theater effect, crosstalk,
cardboard effect, shear distortion, disocclusions

CORESA

38
A key challenge in 3D

• Subjective and objective quality evaluation protocols
and metrics that measure:
– Artifacts of 3D
processing chain
– Naturalness
– Sense of presence
– Fatigue
– Eye strain
– Headache
– …

CORESA

39

3D subjective evaluations

CORESA

40
Evaluation methodology in MPEG 3DVC
• ISO subjective test
campaign for the evaluation 1)
of proponent technologies Registration
and
for the 3D video coding Screening

standard (3DV) during the
month of October 2011 2)
Training
• Coordinated by QUALINET
COST Action 3)
Test
• Network of 12 labs all
around the world
• Circa 120 subjects in each
test laboratory taking part,
during 1 week

CORESA

41
Evaluation methodology in MPEG 3DVC

• Double Stimulus Impairment Scale (DSIS) evaluation
“Rate the quality of each stimulus B,
keeping in mind that of stimulus A”

• 11-grade numerical categorical scale
– 10: highest quality (i.e. test sequence indistinguishable from the
reference)
– 0: lowest quality

CORESA

Scatter plots (EPFL – UBC) 42

Autostereo data

CORESA

Scatter plots (NTNU – Acreo) 43

Stereo data

CORESA

44
Paired comparison subjective evaluation

• Evaluation of motion parallax versus
binocular based mobile 3D display

CORESA

45
Wiggle stereoscopy
• Rapid change between two views provides motion parallax cues
• Automatic alternation leads to annoyingly jerky image
• Quality of wiggle stereoscopy can be improved in two ways
– Use multiple views to achieve smoother transition
– Interactive alternation between images depending on the viewing position

CORESA

Video 46

CORESA

Subjective quality evaluation 47

• Comparison of methods in terms of overall and depth quality
• Paired comparison with sequential presentation and 3 rating
possibilities (first, equal, second)
• Multiview content from the MPEG 3DV dataset (Ballons,
Kendo, Mobile)
• 7 different test conditions
– 2D as reference (2D)
– Anaglyph with narrow (ANN) and wide (ANW) camera baseline
– Motion Parallax with narrow (MVN) and wide (MVW) camera
baseline
– Autostereoscopic with narrow (Sn) and wide (Sw) camera baseline
• 15 participants (11 male, 4 female)

CORESA

Overall quality 48

(a) Ballons (b) Kendo

ions from multi-

ultiscopic image (c) Mobile (d) Overall
eline resulting in
image of 21 im- Fig. 5. CORESA and tie probabilities of the individual pairs
Preference
in larger motion considering the overall quality.

Depth quality 49

(a) Ballons (b) Kendo
(a) Overall quali

Fig. 7. Preferenc

(c) Mobile (d) Overall

Fig. 6.CORESA and tie probabilities of the individual pairs
Preference
considering the perceived depth.
(a) Overall quali

50
Preference probabilities

(a) Overall quality (b) Perceived depth

Fig. 7. Preference probabilities of condition A vs. B.
CORESA

51
Probability of preference & Confidence Interval
Fig. 7. Preference probabilities of condition A vs. B.


Fig. 8. Preference probability of choosing the given restitu-
tion technique among the others.
CORESA

52
Normalized MOS and CI


Fig. 9. MOS/CI of the individual test conditions.
CORESA

53
Subjective 3D quality assessment

• Psycho-perceptual approaches
– Examine relationship between physical stimuli and
sensorial experience
– Usually conducted in highly controlled laboratory
environment
– Judgement of overall quality or predefined quality
attributes (e.g. depth)
• User-centered approaches
– Relate quality evaluation to the use of the system or
service
– Takes into account typical users (persona) and required
system characteristics
– Goal-related evaluation in potential usage scenarios
CORESA

54
Psycho-perceptual evaluation
• Long history of subjective quality evaluation according to
standardized protocols
• Methods for 2D video quality are standardized in ITU-R BT.
500-11
• Methods differ mainly in
– Number of stimuli (single, double)
– Type of scale (quality, impairment, comparison)
– Granularity of scale (discrete, continuous)
• According to ITU-R BT.1438 also recommended for 3D video
quality
• Assessment predefined quality attributes (overall quality, depth
quality, naturalness)
• No support for experienced quality factors and ergonomic
aspects

CORESA

55
Methods overview [MOBILE3DTV D4.2]

CORESA

56
User centered evaluation
• Increasing importance to evaluate quality from a user perspective
rather than from a system perspective
• Constrains the tests to potential users and stimuli and evaluates the
acceptability with respect to the task and the context
• Identify underlying quality factors beside the overall quality

CORESA

Personas and scenarios [Strohmeier2009] 57

• Personas (user profiles)
– Archetypical user representing the needs, behaviors
and goals of a particular group of users
– Not real people but fictional characters derived from
user requirements
• Scenarios (context)
– Realistic usage environment

CORESA

58
Discussion
• Current status
– Most 3D studies are based on psycho-perceptual evaluation methodologies
– Perceived 3D quality is difficult to describe using fixed quality attributes
– Only a few studies consider ergonomical aspects of 3D video
– Common standards for subjective evaluation of 3D quality are missing
• Future directions
– Assess suitability of established 2D methodologies for 3D quality evaluation
considering the fundamental differences between 2D and 3D perception
and sources of distortions
– User centered evaluation to identify the various quality factors for 3D video
– Need for long-term studies which consider visual fatique and motion
sickness
– Definition of standard protocols for subjective evaluation of 3D quality
including display and viewing conditions
– Evaluation of systems in realistic usage scenarios and with relevant content

CORESA

59
Objective quality assessment

• Subjective tests are time consuming and expensive and
can not be user for online quality evaluation
• Objective quality metrics are mathematical models that
approximate subjective quality evaluation
• High correlation between subjective and objective scores is
desired
• Can be grouped based on the availability of a reference
– Full reference (FR)
– Reduced reference (RR)
– No reference (NR)
• Can be categorized according to the approach
– Model based (top down)
– Feature based (bottom up)

CORESA

Objective 3D quality metrics 60

• Fairly new research topic with only a few proposed
metrics so far
• Straightforward way is to apply 2D image quality metrics
to individual channels
– Works for impairment that affect both channels equally
– Does not consider inter channel masking effects
• Several factors affect the 3D image quality
– Binocular suppression
– Inter channel relations
– Image content
• Most of the proposed metrics are full reference metrics
– Not suitable for online scenarios or when the reference is not
available

CORESA

Left/right image [Campisi2007] 61

• Applies common 2D image quality metrics to left and right image
• Combines scores using average, main eye or visual acuity
approach
CORESA

62
3D Quality ǂ Σ2D
• Encode left/right images with
JPEG and different QPs (0-100)
• Show images with decreasing
quality to the subjects
• Determine limit of transparency
for left, right and stereo image
• Compute PSNR of left and right
images and average for stereo
• Find PSNR which corresponds
to the QP limit for each image
• Average PSNRs for each image
across the individual subjects

CORESA

3D Quality ǂ Σ2D 63

CORESA

64
MVC assessment using PSNR as metric
Sample comparison of simulcast vs inter-view prediction

8 views (640 480), consider rate for all views
~25% bit rate savings over all views

Ballroom Race1
40 42

39 41

40
38

39
37
38

PSNR (db)
PSNR (db)

36
37
35
36

34
35

33
Simulcast 34 Simulcast
32 MVC 33 MVC

31 32
0 200 400 600 800 1000 1200 1400 1600 1800 0 200 400 600 800 1000 1200 1400 1600

Bitrate (Kb/s) Bitrate (Kb/s)

CORESA
24-25 mai 2012, Lille, MPEG 3D Video Coding Activity
Vetro: Update on France

MVC:assessment by subjective evaluation
MVC Subjective Performance 65

• Main finding: MVC achieves comparable quality to
simulcast with as little as 25% rate for dependent view

4.50
Mean Opinion Score

4.00

3.50

3.00

2.50

2.00

1.50

1.00
al

)

t
ct

ct

ct

ct

ct

ct
VC

c
in

5P
P

P

P

P

P

P
rig

50

35

25

20

15

10
+A

L_
O

L_

L_

L_

L_

L_

L_
VC

12
12

12

12

12

12

12
(A
t
cas
ul
m
Si

Base view fixed at 12Mbps
Dependent view at varying percentage of base view rate

CORESA Vetro: Update on MPEG 3D Video Coding Activity

66
Performance of PSNR in 3D video quality evaluation

Left

View Depth 3D Video View
Intermediate Synthesized
Synthesis Estimation Codec Synthesis

Right Decoded

CORESA

and is irrelevant for the goal of the measure performance anal- ever, they are more clustered for either the decoded view or
ysis. At the same time, it avoids the risk of data over fit- the average of the decoded and synthesized views, especially 67
Performance of PSNR in
ting, which may occur when considering non-linear regres- 3D sequence S01. Therefore,evaluation
for video quality the PSNR of the decoded view
sion. The linear regression is of the form: and the average PSNR of the decoded and synthesized views
MOSp (VQR) = a · VQR + b should have a better correlation with the perceived quality.
For some sequences (S03, S05 and S06), a few synthe-
Then, the Pearson linear correlation coefficient (PCC) and sized (intermediate) views have a low PSNR value but the cor-
the root-mean-square error (RMSE) are computed between responding stereo pairs have a high MOS. These data points
a) PSNR of the decoded view, calculated between the decoded view and the original view
b) PSNR of the intermediate view, calculated between the synthesized view at the decoder side and the intermediate view from the original data (if available)
MOSp and MOS to estimateview, calculated between theVQR. To es- at the decoder side and the distant from thethe encoder side scatter plot of
c) PSNR of the synthesized the accuracy of the synthesized view are significantly synthesized view at trend in the
timate monotonicity and consistency,
d) Average PSNR of the decoded view andthe intermediate view, computed as the synthesized (intermediate) view. It is known that one pro-
the Spearman rank or- the mean value of a) and b)
e) Average PSNR of the decoded view and the synthesized view, computed as the mean value of a) and c)
der correlation coefficient (SCC) and the outlier ratio (OR), ponent used a different view synthesis algorithm. Our hy-
are computed between MOSp and MOS, respectively [13]. pothesis is that those results are from this specific proponent

a) b) c) d) e) a) b) c) d) e)
Table 2. Accuracy, monotonicity, and consistency indexes of the objective video quality models under consideration.
Decoded and Decoded and Decoded and Decoded and
Decoded Intermediate Synthesized Decoded Intermediate Synthesized
intermediate synthesized intermediate synthesized
Pearson linear correlation coefficient Spearman rank order correlation coefficient
S01 0.9216 - 0.4489 - 0.8502 0.9452 - 0.4211 - 0.8509
S02 0.9469 - 0.8158 - 0.9406 0.9534 - 0.7958 - 0.9373
S03 0.9366 0.8130 0.8370 0.9199 0.9240 0.9362 0.8504 0.8674 0.9219 0.9266
S04 0.8889 - 0.8476 - 0.8843 0.8372 - 0.8055 - 0.8412
S05 0.9073 0.4592 0.4877 0.8498 0.8309 0.9078 0.5910 0.5831 0.8182 0.8019
S06 0.9558 0.6477 0.6145 0.9235 0.9039 0.9773 0.6370 0.5934 0.9288 0.9049
S07 0.9219 0.1119 0.3788 0.9186 0.8284 0.9158 0.1935 0.4704 0.9278 0.8723
S08 0.8813 0.8972 0.9043 0.9024 0.9044 0.8185 0.8449 0.9006 0.8492 0.8747
Mean 0.9200 0.5858 0.6668 0.9028 0.8834 0.9114 0.6234 0.6797 0.8892 0.8762
Root-mean-square error Outliers ratio
S01 0.9476 - 2.1812 - 1.2850 0.0220 - 0.2088 - 0.0659
S02 0.7882 - CORESA -
1.4169 0.8316 0.0000 - 0.0978 - 0.0217
S03 0.7861 1.3062 1.2277 mai 2012,
24-25 0.8795 Lille, France 0.0000
0.8575 0.0312 0.0208 0.0104 0.0104
S04 1.1044 - 1.2790 - 1.1255 0.0217 - 0.0543 - 0.0217

68
Disparity map and cyclopean image [Boev2006]

• Compare cyclopean images using perceptual quality metric
(SSIM)
• Compare disparity maps using absolute difference
CORESA

69
Discussion
• Current status
– Current 3D quality metrics are rather simple extensions to 2D quality
metrics
– They do not take into account the interaction between 2D and 3D
perception
– Mainly full reference metrics not suitable for assessment of 3D creation
– Do not consider 3D display characteristics and influence on 3D
perception
• Future directions
– Important to quantify the influence of 3D distortions originating from
every step within the whole processing chain
– More accurate models for 2D and 3D human visual perception are
needed
– Consider interaction between monocular and binocular depth cues
– Incorporate information about 3D content creation and display
– Consider focus of attention and accommodation/convergence rivalry
CORESA

70

A FEW LAST WORDS

CORESA

71
Take home message
• 3DTV is among one of the emerging multimedia trends
• Success will largely depend on improved quality of experience
• Interesting content and good 3D quality provides novel experience
to users (e.g. Avatar)
• Bad 3D quality may limit the acceptance of 3D technologies (e.g.
Clash of the Titans)
• Subjective and objective quality assessment of 3D image and video
not mature yet
• Methodologies and metrics need to be adopted for 3D considering
the special characteristics of 3D perception
• Huge opportunities in research, technology, art, and business

CORESA

72
Thanks for your attention

CORESA

Towards 3D visual quality assessment for future multimedia

More Related Content

Viewers also liked (12)

Similar to Towards 3D visual quality assessment for future multimedia (20)

More from Touradj Ebrahimi (20)

Recently uploaded (20)

Towards 3D visual quality assessment for future multimedia

Editor's Notes