Current developments in video quality: From the emerging HEVC standard to temporal video quality assessment and classification

Current developments in Video Quality :
From the emerging HEVC standard to temporal video
quality assessment and classification
Dr. Harilaos G. Koumaras
Friday 22 June 2012
FTW, Vienna, Austria
1
Dr. Harilaos Koumaras

• Research Interests
▫ Video Quality, QoE, Prediction and Assessment
Methods, Video Coding Standards, IMS
• EC Research Activities
▫ Coordination and Participation in FP7
▫ ICT and SEC
▫ Independent Expert Objective 1.5
• National Research Activities
▫ Evaluator of the Entrepreneurship Contest “Η
Ελλάδα Καινοτομεί”
2

Outline
• Advances on Video Encoding
▫ HEVC (H.265)
▫ Performance Expectations
▫ Main Characteristics
▫ Performance Analysis
▫ Perceived Performance Analysis
• Advances on Video Quality Prediction
▫ Current Situation
▫ Enhanced content classification (uncompressed)
▫ Enhanced prediction (uncompressed->compressed)
3

Outline
▫ HEVC (H.265)
4

Video Coding - Timeline
5

Codec Evolution (milestones)
• Transition from MPEG2 to H.264/AVC
– ~2:1 improvement in performance
– Significantly more complex, but neutralized by
Moore’s law
• Next generation H.265/HEVC in development
– 30-50% improvement in performance
– Significantly more complex yet
• Performance strongly dependent on encoder
6

Codec Comparison
http://www.compression.ru/video/codec_comparison/index_en.html
7

Outline
▫ HEVC (H.265)
8

HEVC – The project objectives
• Quality
▫ To increase the quality of the coded signal
• Compression
▫ To enhance the compression ratio
▫ A good compression rate is important to be able to use the
video at limitations such as Internet connection bandwidth,
radio bandwidth or limited space of DVD and Blue-Ray
discs.
• Computational Complexity
▫ Keep it as low as possible
▫ Support real-time applications, which rely on fast decoding
process.
• Improvements in one of these three factors often comes
at the cost of the others.
9

Outline
▫ HEVC (H.265)
10

Reference Machine
• The Reference Machine used to test new algorithms
against is the test model developed by members of
JCT-VC.
• The test model is written in C++ and have the
currently best algorithms chosen by JCT-VC
meetings implemented.
• At first (up to version 0.9) the test model was called
TMuC (Test Model under Consideration).
• When JCT-VC officially adopted the test model the
name changed to HM (HEVC test Model).
• Currently HM has reached version 7 (!)
11

HEVC – New Key Features
• Flexible block structure to support arbitrary min & max
unit sizes
▫ Coding Unit (CU)
▫ Prediction Unit (PU)
▫ Transform Unit (TU)
• Consistent syntax representation, independent of size
• Asymmetric motion partitions
• Greater than ¼ pixel motion accuracy with new
interpolation filter
• Large integer transforms up to 64x64
• New rotational transform
• New motion vector prediction method
• New in-loop filtering methods
• New intra-coding prediction methods
• New entropy coding with explicit scan order signaling
12

Reference Configurations
• 6 reference
configurations are
specified
▫ (a) High Efficiency
(HE) and Low
Complexity (LC)
settings;
▫ (b) Intra Only,
Random Access, and
Low Delay settings.
• Six test conditions
(or configurations)
are formed by picking
up one from the first
group and one form
the second group.
13

HEVC – Unit Definition (CU)
• Coding Unit (CU)
• CU is the basic processing block
▫ Used for quad-tree based segmentation of
regions
▫ Plays a similar role to macroblock
▫ Can take various sizes
 Always power of 2 size
 Always square shape
• Range of allowed sizes specified in
Sequence Parameter Set
▫ Largest CU (LCU)
▫ Maximum hierarchical depth
▫ Easily adapted for various applications
• Recursive structure with split flag
▫ Single 2Nx2N or four NxN
LCU size = 128 (N=64), maximum hierarchical depth = 5
14

CU example #2
• LCU=64
• Max Hier. Depth=3
• Split Flag 0/1
15

Benefits of CU structure
• Supports large CU size
▫ Virtually no limit to maximum
size
▫ Maximum of 128x128 used in
CfP submissions
• Flexible structure
▫ Can be optimized for content,
device or application
• Size-independent syntax
▫ Each CU has an identical
syntax regardless of its size
▫ Reduces complexity of
parsing
64
64
32
32
16
16
8
8
Resolution: 1920x1080
LCU size : 64
Maximum depth =2
LCU size : 64
Maximum depth = 4
Resolution: 1920x1080
Resolution: 352x288
LCU size : 16
Maximum depth =2
32
32
16
16
8
8
4
4
16

HEVC – Unit Definition (PU)
• Prediction Unit (PU)
• PU is the basic unit for prediction
▫ Largest allowed PU size is equal to the CU size
▫ Other allowed PU sizes depend on prediction type
 Includes asymmetric splitting options for inter-prediction
2Nx2N NxN
2Nx2N Nx2N 2NxnU2NxN NxN 2NxnD nLx2N nRx2N
Intra
Inter
2Nx2N
Skip
• Example of 128x128 CU
– Skip: PU = 128x128
– Intra: PU = 128x128 or 64x64
– Inter: PU = 128x128, 128x64, 64x128, 64x64, 128x32, 128x96,
32x128 or 96x128
17

HEVC – Unit Definition (TU)
• Transform Unit
• TU is the basic unit for transform and quantization
▫ May exceed size of PU, but not CU
• Only two TU options are allowed, signalled by transform unit size flag
▫ Transform unit size flag = 0  2Nx2N - same as CU
▫ Transform unit size flag = 1  square units of smaller size
 NxN when PU splitting is symmetric
 N/2xN/2 when PU splitting is asymmetric
2N
2N
0 1
2 3
N
N
2N
2N
N/2
N/2
transform unit size flag = 0 transform unit size flag = 0transform unit size flag = 1 transform unit size flag = 1
(a) 2Nx2N, 2NxN, Nx2N, NxN case (b) 2NxnU, 2NxnD, nLx2N, nRxN case
18

Relationship of CU, PU and TU
CU
PU
TU
2Nx2N 2NxN Nx2N NxN 2NxnU 2NxnD nLx2N nRx2N
Symmetric type Asymmetric type
TU size flag = 0
TU size flag = 1
TU
TU size flag = 0
TU size flag = 1
19

Intra Prediction
• The current intra prediction in HM unified two
directional intra prediction methods
▫ Arbitrary Direction Intra (ADI) introduced in
JCTVC-A124
▫ Angular Intra Prediction introduced in JCTVC-
A119
• With simplification for parallel processing
possibility, leading to a simplified unified intra
prediction (JCTVC-B100, JCTVC-C042).
20

Unified Intra Prediction
• In current HM, unified intra prediction
provides up to 34 directional prediction
modes for different PUs.
• With the PU size of 4×4, 8×8, 16×16, 32×32,
64×64, there are 17, 34, 34, 34, and 5
prediction modes available respectively.
• The prediction directions in the unified intra
prediction have the angles of +/- [0, 2, 5, 9,
13, 17, 21, 26, 32]/32.
• The angle is given by displacement of the
bottom row of the PU and the reference row
above the PU in case of vertical prediction,
or displacement of the rightmost column of
the PU and the reference column left from
the PU in case of horizontal prediction.
• Figure 1 shows an example of prediction
directions for 32×32 block size.
PU size
Number of
prediction
modes
4×4 17
8×8 34
16×16 34
32×32 34
64×64 5
21

Intra Prediction Example
• When the intra prediction angle is
positive, (blue lines), only the
samples from the main array are
used for prediction.
• When the intra prediction angle is
negative, (red lines), a per-sample
test should be performed to
determine whether samples from
the main or the side array should
be used for prediction
22

Inter Prediction
• Instead of adaptive
interpolation filter, HM
adopts the fixed 12-tap
DCT-based interpolation
filter to provide fractional
pel accuracy interpolation
by replacing the
combination of Wiener and
bilinear filters with a set of
interpolation filters at the
desired fractional accuracy.
• More specifically, only one
filtering procedure is
needed to provide the
interpolation pixel to any
pixel accuracy, instead of a
combination of 6-tap and
bilinear filtering procedures
in H.264/AVC.
23

Outline
▫ HEVC (H.265)
24

Performance Results
• Four Test Signals
• The video clips were encoded from their original
uncompressed YUV format to
• ISO AVC Main Profile (MP)
• Random Access Profile (RAP)
• Random Access Low Complexity Profile
(RALCP)
• Low Delay Profile (LDP)
• Low Delay P Profile (LDPP)
• For HEVC the Test Model (HM) Reference
Software v5.1
Frames Resolution
Apocalypto Trailer 990 352x288
Batman Trailer 913 352x288
Bubbles 501 416x240
Horses 300 416x240
25

Experimental Results
26

Performance Outcomes
• Based on experimental data :
• HEVC can retain the same video quality level as
AVC
• HEVC achieves 32% to 62% improvement in the
compression and coding efficiency
• Complexity doubles
27

Outline
▫ HEVC (H.265)
28

Objective Assessment (SSIM)
• Maintain the same quality level
29

.. HEVC @ Low Bit Rates ??
• AVC‟s decoded video for
QP equal to 51, proves
not viewable, while
HEVC achieves
acceptable video quality
and smooth playback.
(b) (c)
Decoded frame of Race Horses signal
for HEVC RAP and AVC MP (QP=51)
(a) Initial Video, (b) HEVC RAP, (c) AVC MP
30

HEVC – The project milestones
• Quality
▫ Maintains the same quality as AVC/H.264
▫ Can achieve acceptable quality at very low bit rates
• Compression
▫ Doubles the compression efficiency in comparison
to AVC/H.264
▫ Can achieve very high compression at acceptable
quality
• Computational Complexity
▫ Depending on the profile used, it may be double in
comparison to AVC/H.264.
31

Outline
▫ HEVC (H.265)
32

Video Quality Degradation
• Lossy compression methods introduce
distortions whose visibility depends highly on
the content.
• These artifacts result in perceived quality
degradation.
• The parameters with strong influence on the
video quality are the encoding-related
▫ (the bit rate, the frame rate and the resolution)
• Thus, the issue of the user satisfaction in
correlation with the encoding parameters
has been raised.
33

Video Quality Evaluation
Subjective Methods
• Subjective experiments
▫ An audience is questioned for the perceived
quality evaluation of a encoded signal
▫ Subjective experiments to date are the only widely
recognized method of determining the actual
perceived quality
▫ They are complex and time-consuming, both
in their preparation and execution.
34

Subjective Methods
• The subjective test methods
▫ Proposed by ITU and VQEG
▫ ITU-R Rec. T.500-11 (2002) and ITU-T Rec. P.910 (1999)
▫ involve an audience of people, who watch a video sequence
and score its quality as perceived by them, under specific
and controlled watching conditions.
▫ Afterwards, the statistical analysis of the collected data is
used for the evaluation of the perceived quality.
 The Mean Opinion Score (MOS) is regarded as the most
reliable method of quality measurement and has been
applied on the most known subjective techniques
35

Subjective Methods
• Double Stimulus Impairment Scale (DSIS)
▫ Scene pairs (reference scene is always first)
▫ Overall impression scale of impairment
• Single Stimulus (SS) Methods
▫ Multiple separate scenes are shown
▫ Three different scoring methods are used:
 Adjectival: the aforementioned 5-grade impairment scale,
however half-grades may be allowed.
 Numerical: an 11-grade numerical scale, useful if a reference
is not available.
 Non-categorical: a continuous scale with no numbers or a
large range, e.g. 0-100
36

Subjective Methods
• Single Stimulus Continuous Quality Evaluation
(SSCQE)
▫ The viewers watch a program of typically 20–30
minutes without the original reference to be shown
▫ The subjects/viewers using a slider continuously rate
the instantaneously PQoS on scale from „bad‟ to
„excellent‟ (0 to 100).
• Double Stimulus Continuous Quality Scale
(DSCQS)
▫ Scene pairs (reference scene is always first)
▫ The subjects/viewers using a slider continuously rate
the instantaneously PQoS on scale from „bad‟ to
„excellent‟ (0 to 100)
37

Objective Methods
Subjective Methods
Time consuming, Expensive, Require Sophisticated Equipment
Objective Methods
Exploiting Mathematical Models
for Emulating the Results of Subjective Procedures
38

Objective Methods
Objective Methods
“Psychophysical approach",
where metric design is based on
models of the human visual system.
(e.g. VQ metric by Stefan Winkler,
VDP by Daly, VDM by Lubin…)
“Engineering approach", where metrics
make assumptions about the artifacts that are
introduced by the compression process or
transmission link. (e.g. SSIM metric Z. Wang,
DVQ metric by Watson…)
• Full-reference metrics (frame-by-frame comparison
between a reference video and the video under test)
• No-reference metrics (no need of reference information)
• Reduced-reference metrics (extract a number of features from the
reference and degraded video (e.g. amount of motion, spatial detail)
39

Objective Methods
- Advantages
• Faster
• Cheaper
• Economically Affordable
• No audience is required
• No statistical analysis is needed
• …
40

Objective Methods - Cons
• Full Reference Methods
 Initial undistorted clips are not always available.
 Synchronization predicaments between the
undistorted and the distorted signal (which may
have experienced frame loss)
• Reduced Reference
 Very few implementations
 Similar problems to Full Reference
• No Reference
 Usually Codec Specific
41

Practical Limitations
The 3G/4G vision is the provision of audiovisual
content at various quality and price levels
(P. Seeling, 2004)
All the aforementioned subjective/objective post-
encoding methods require repeating tests in
order to determine the encoding parameters that
satisfy a specific level of user satisfaction.
42

Need for predicting Video Quality
There is need for developing methods for predicting
quickly and easily the video quality level.
These methods will enable the determination of the
specific encoding parameters that will satisfy a
certain quality level.
43

Assuming that available network resources can be
efficiently confronted by traffic control
techniques
Video Quality is mostly depended on
i) Encoding parameters
ii) Video content
44

Our contribution…
Evaluation of VQ today:
- Subjective procedures
(time-consuming, expensive)
- Objective procedures
(many repeated tests required)
We have proposed:
- Objective pre-encoding evaluation method
- For ISO MPEG-4/AVC Clips
- Based on a single metric and a single test encoding process
Post-encoding
evaluation
45

Our contribution (2)
• Harilaos Koumaras, Fidel Liberal, Lingfen Sun, “Quality of experience issues in multimedia provision”
(Editorial), Int. Journal of Telecommunications Systems, Springer, pp.1-3, DOI: 10.1007/s11235-010-9349-4, March 2012
• Julien Arnaud, Daniel Négru, Mamadou Sidibé, Julien Pauty, Harilaos Koumaras, “Adaptive IPTV services based on a
novel IP Multimedia Subsystem”, Multimedia Tools & Applications, Springer, Vol. 55(2),333-352, DOI:
10.1007/s11042-010-0576-1, November 2011
• H. Koumaras, C-H Lin, C-K Shieh, A. Kourtis , “A Framework for End-to-End Video Quality Prediction of MPEG
Video”, Journal of Visual Communication and Image Representation, Elsevier, July 2009,
• M. Sidibe, H. Koumaras, I. Kofler, A. Mehaoua, A. Kourtis, C. Timmerer, “A Novel Cross Layer Monitoring
Architecture for Media Services Adaption Based on Network QoS to Perceived QoS Mapping”, International
Journal of Signal, Image and Video Processing Special Issue on “Multimedia Semantics, Adaptation & Personalization”,
Vol.2, No.4, pp307-320, DOI 10.1007/s11760-008-0083-2, ISSN-1863-1703, December 2008
• Harilaos Koumaras, Michail-Alexandros Kourtis, Drakoulis Martakos and Christian, “Impact of H.264 Advanced Video
Coding Inter-Frame Block Sizes on Video Quality”
The International Conference on Computer Vision Theory and Applications VISAPP 2012, Rome, Italy, February 24-26.
• H. Koumaras, Julien Arnaud, Daniel Negru, A. Kourtis, ” An Experimental Approach of Video Quality Level
Dependence on Video Content Dynamics” , MobiMedia 2009, 3rd European Symposium on Mobile Media Delivery
(EUMOB), London, U.K., 7-9 September 2009
• M. Sibide, H. Koumaras, G. Xilouris, “A Perceived Quality-aware Cross Layer Monitoring Framework for Real-
Time Media Content Adaptation”, The IEEE Global Information Infrastructure Symposium (IEEE GIIS 2009) June 23-
26, 2009, Hammamet, Tunisia
• H. Koumaras, A. Kourtis, “Video Quality Prediction Based on the Spatial and Temporal Classification of the
Uncompressed Content” , The 18th Annual IEEE International Symposium on Personal, Indoor and Mobile Radio
Comm. (PIMRC), Athens, Greece, 3-7 Sep. 2007
• H. Koumaras, A. Kourtis, C-H Lin, C-K Shieh, “A Theoretical Framework for End-to-End Video Quality Prediction
of MPEG-based Sequences” BEST PAPER AWARD
• The Third Inter. Conf. on Networking and Services – ICNS07, Athens, Greece, June 19-25 2007
• H. Koumaras, T. Pliakas, A. Kourtis, “A Novel Method for Pre-Encoding Video Quality Prediction”, IST Mobile
Summit 2007, Budapest, Hungary, 1-5 July 2007
• G. Gardikis, H.Koumaras, G.Xilouris, E.Pallis, A.Kourtis , “Real-time, Dynamic Resource Allocation in DVB-
S.2/RCS Networks” , Eighth International Symposium on Interworking, Santiago, Chile, January 15 – 19, 2007
46

… more prediction methods??
• YES!
• Our current works is focused on video quality
fingerprint
• We try to specify each video clip/test signal with
a unique fingerprint, which describes its unique
characteristics.
47

Video Fingerprint
• We were looking for a method of describing the
temporal and spatial video quality
characteristics of each video signal
• We have proposed a 2-D metric (called video
fingerprint), which provides a perceived and
unique classification of the video content
characteristics
• The form of each fingerprint is unique and
representative of each video signal.
48

Video Fingerprint
• Video Fingerprint
combines:
▫ Temporal Video
quality aspects of the
test signal (TSSIM,
MovIe)
▫ Spatial Video Quality
aspects of the test
signals (SSIM, VQM)
49
Temporal Video Quality Metric
SpatialVideoQualityMetric
Low Temporal
High Spatial
HighTemporal
High Spatial
Low Temporal
Low Spatial
High Temporal
Low Spatial

Proposed Method
Initial Encoding
at a predefined
Resolution and
Bit Rate
Generation
of
Fingerprint
Mapping of each
fingerprint
according to
reference Video
Quality vs. Bit
Rate curves
50
i) Low Bit Rate
ii) Low Resolution
iii) Fast process

Fingerprint Characteristics
51
Spatial Spatial Spatial Spatial
Spatial Spatial Spatial Spatial
Temporal
Temporal
Temporal
Temporal
Temporal
Temporal
Temporal
Temporal
Homogeneous
Low Spatial
Low Temporal
Homogeneous
Low Spatial
High Temporal
Homogeneous
High Spatial
High Temporal
Homogeneous
High Spatial
Low Temporal
Heterogeneous
Mixed Spatial
Mixed Temporal
Heterogeneous
Low Spatial
Mixed Temporal
Heterogeneous
High Spatial
Mixed Temporal
Heterogeneous
Mixed Spatial
High Temporal

Fingerprint Pros and Cons
Pros Cons
• Unique Characterization of
each test signal
• Both temporal and spatial
description
• Homogeneous and
Heterogeneous classification
of the videos
• One test encoding is necessary
• Full reference comparison
with reference signal
• Difficult decision on highly
heterogeneous videos
52

Next Steps…
• Extensive testing of Video Quality Fingerprint
• Extension of the fingerprint to uncompressed
signal
• If succeeded, we will provide a method for video
quality prediction directly from the
uncompressed
53

References
• [1] JCTVC-A119, Video coding technology proposal by Tandberg, Nokia, and Ericsson
• [2] JCTVC-A124, Video coding technology proposal by Samsung (and BBC)
• [3] JCTVC-B093, Simplified angular intra prediction
• [4] JCTVC-B100, Unification of the Directional Intra Prediction Methods in TMuC
• [5] JCTVC-B118, Angular intra prediction and ADI simplification
• [6] JCTVC-C042, TE5: Results for Simplification of Unified Intra Prediction
• [7] JCTVC-C207, Encoder improvement of unified intra prediction
• [8] H. Koumaras, A. Kourtis, D. Martakos, J. Lauterjung, “Quantified PQoS Assessment Based on
Fast Estimation of the Spatial and Temporal Activity Level”, Multimedia Tools and
Applications, Springer Editions, Published online (IF 0.416)
• [9] H. Koumaras, A. Kourtis, D. Martakos, “Evaluation of Video Quality Based on Objectively
Estimated Metric”, Journal of Communications and Networking, KICS, (IF 0.479)
• Vol. 7(3), Sep 2005, Technically cosponsored by IEEE ComSoc
• [10] Harilaos Koumaras, Michail-Alexandros Kourtis, Spyros Mantzouratos, Drakoulis Martakos,,
“Quantitative Performance Evaluation Of the Emerging HEVC/H.265 Video Codec”,
QoEMCS 2012 workshop, Euro ITV 2012, Berlin, Germany, 04 – 06 July 2012. (Accepted)
• [11] Harilaos Koumaras, Michail-Alexandros Kourtis, Drakoulis Martakos, “Benchmarking the
Encoding Efficiency of H.265/HEVC and H.264/AVC”, Future Network & Mobile Summit 2012 4
– 6 July 2012, Berlin, Germany, July 4-6. (
54

Current developments in video quality: From the emerging HEVC standard to temporal video quality assessment and classification

More Related Content

What's hot (20)

Similar to Current developments in video quality: From the emerging HEVC standard to temporal video quality assessment and classification (20)

Recently uploaded (20)

Current developments in video quality: From the emerging HEVC standard to temporal video quality assessment and classification