How video codec work
3 likes7,835 views
The document explains the fundamentals of video codecs, emphasizing the importance of digital video compression to manage data size and maintain quality. It covers concepts including color models, temporal and spatial redundancy, and various encoding techniques such as intra and inter prediction. The document also compares the efficiency of newer codecs like HEVC against older ones like AVC.
![TRANSFORM
Double [3] f(x): x + x => [ 6]
Plus10 [3] f(x): x + 10 => [ 13]
Divide2 [3] f(x): x / 2 => [1.5]](https://image.slidesharecdn.com/howvideocodecwork-171001113931/85/How-video-codec-work-65-320.jpg)
How video codec work
- 2. I'm not a runway but I don't have either twitter or FB But you still can ask questions using #qacodec on twitter PS: I do have a github leandromoreira
- 4. OUR GOAL To learn the whys, whats and hows about digital video compression.
- 5. WHY DO WE NEED COMPRESSION?
- 6. #qacodec IN THE BEGINNING A SINGLE PIXEL (PICTURE ELEMENT)
- 7. #qacodec HOW CAN WE ENCODE THIS PIXEL? 1B = 8 bits 1 bit 2 bits
- 8. #qacodec LET'S DEFINE AN IMAGE (2D IS ON) 4 x 4 x 1 = 16B width height
- 9. #qacodec LET'S COLORIZE IT (RGB PRIMARY COLORS)
- 10. X
- 12. https://lightingstudio.wordpress.com/2012/03/27/week5-light-object-shadow-contrast/
- 13. #qacodec NOW WE'RE DEALING WITH COLOR (3D IS ON) width height color
- 14. #qacodec THE COLOR COST 4 x 4 x 3 = 48B
- 15. BEFORE WE MOVE ON...
- 16. #qacodec
- 17. #qacodec MATRIX OF NUMBERS 25 14 0 0 14 14 16 13 13 13 15 0 15 15 20 10 20 10 0 0 15 15 0 0 10 10
- 18. #qacodec BEHOLD, THE 4TH DIMENSION 4 x 4 x 3 x 30 = 1440B time
- 19. A SINGLE TV SHOW EPISODE
- 20. #qacodec 30 x 60 x 24 x 3 x 1080 x 1920 = 250.28GB 1080p 24 fps 30 min long time in sec colorfps resolution
- 21. #qacodec 24 x 3 x 1080 x 1920 x 8 = 1.11Gbps colorfps resolution bits
- 22. WHAT CAN WE DO?
- 25. #qacodec OUR EYES: AN OVERSIMPLIFICATION The eye contains about 120M rod cells and 6M cone cells.
- 26. #qacodec WE'RE BETTER TO SEE LUMA THAN COLOR
- 27. #qacodec AN ALTERNATIVE TO RGB
- 28. #qacodec From RGB to YCbCr Y = 0.299R + 0.587G + 0.114B Cb = 0.564(B - Y) | Cr = 0.713(R - Y) From YCbCr to RGB R = Y + 1.402Cr | B = Y + 1.772Cb | G = Y - 0.344Cb - 0.714Cr *ITU-R BT.601-7
- 29. #qacodec COLOR MODEL YUV (YCBCR, YPBPR) Y (luma) U (chroma blue) V (chroma red)
- 31. #qacodec
- 32. 24 bits per pixel 12 bits per pixel
- 33. #qacodec 30 x 60 x 24 x 3 x 1080 x 1920 = 250.28GB time in sec colorfps resolution
- 34. #qacodec 30 x 60 x 24 x 3 x 1080 x 1920 = 250.28GB time in sec colorfps resolution 30 x 60 x 24 x 1.5 x 1080 x 1920 = 125.14GB time in sec colorfps resolution
- 36. #qacodec TEMPORAL REDUNDANCY frame 0 frame 1 frame 2 frame 3
- 37. #qacodec ORIGINAL FRAMES |||||||||| (103Kb) || (4Kb)
- 40. #qacodec WHAT IS THE COST? frame 0 frame 1 diff + reference
- 41. CAN WE DO BETTER?
- 42. #qacodec DESCRIBE THE MOTION (ESTIMATION) frame 1 frame 2 x=34, y=26x=12, y=25
- 43. #qacodec FRAME DIFFERENCE VS MOTION ESTIMATION
- 44. ffmpeg -flags2 +export_mvs -i in.mp4 -vf codecview=mv=pf+bf+bb out.mp4
- 45. #qacodec SO, CAN WE JUST LINK ALL THE FRAMES?
- 47. FRAMES THAT CAN'T BE EASILY EXPLOITED BY TEMPORAL REDUNDANCY
- 52. SPATIAL REDUNDANCY (INTRA PREDICTION) 100 100 100 200 100 ??? ??? ??? 100 ??? ??? ??? 100 ??? ??? ??? 100 100 100 200 100 100 100 200 100 100 100 200 100 100 100 200 100 100 100 200 100 100 100 200 100 100 100 200 100 100 120 210 100 100 100 200 100 0 0 0 100 0 0 0 100 0 20 10 unknown values direction of the prediction real values difference highly compressible
- 53. WHAT IS THE COST OF INTRA PREDICTION? 100 100 100 200 100 0 0 0 100 0 0 0 100 0 20 10 direction of the prediction residual values
- 55. PIED PIPER OUR IMAGINARY VIDEO CODEC * Pied Piper is a startup company focused on "multi-platform technology based on a proprietary universal compression algorithm" featured in the HBO series Silicon Valley
- 56. CODEC “A codec is a device or computer program for encoding or decoding a digital data stream or signal.”
- 57. CODEC *Vcodex: Introduction to Video Coding Compress (enCOde) Decompress (DECode) Video Source Compressed Video Display
- 58. CODEC *Vcodex: Introduction to Video Coding Compress (enCOde) Decompress (DECode) Video Source Compressed Video Display
- 59. ENCODER BLOCKS picture partitioning predictions transform quantization entropy coding redundancy removal entropy reduction lossless compression dct, dwt, intra-prediction, inter-prediction, motion estimation / compensation linear, logarithm huffman, lzw ... technique implementation
- 60. picture partitioning predictions transform quantization entropy coding redundancy removal entropy reduction lossless compression dct, dwt, intra-prediction, inter-prediction, motion estimation / compensation linear, logarithm huffman, lzw ... ENCODER technique implementation
- 62. ENCODER picture partitioning predictions transform quantization entropy coding redundancy removal entropy reduction lossless compression dct, dwt, intra-prediction, inter-prediction, motion estimation / compensation linear, logarithm huffman, lzw ... technique implementation
- 63. (INTRA|INTER)-PREDICTION (SPACE AND TIME) direction of the prediction 100 100 100 200 100 100 100 200 100 100 100 200 100 100 100 200
- 64. ENCODER picture partitioning predictions transform quantization entropy coding redundancy removal entropy reduction lossless compression dct, dwt, intra-prediction, inter-prediction, motion estimation / compensation linear, logarithm huffman, lzw ... technique implementation
- 65. TRANSFORM Double [3] f(x): x + x => [ 6] Plus10 [3] f(x): x + 10 => [ 13] Divide2 [3] f(x): x / 2 => [1.5]
- 66. TRANSFORM (DCT)
- 68. WALKTHROUGH
- 69. DCT APPLIED
- 70. ENCODER picture partitioning predictions transform quantization entropy coding redundancy removal entropy reduction lossless compression dct, dwt, intra-prediction, inter-prediction, motion estimation / compensation linear, logarithm huffman, lzw ... technique implementation
- 72. ORIGINAL VS QUANTIZED (50%)
- 73. ORIGINAL VS QUANTIZED (85%) https://www.mathworks.com/help/images/discrete-cosine-transform.html
- 74. ENCODER picture partitioning predictions transform quantization entropy coding redundancy removal entropy reduction lossless compression dct, dwt, intra-prediction, inter-prediction, motion estimation / compensation linear, logarithm huffman, lzw ... technique implementation
- 75. ENTROPY CODING VLC 27 10 0 0 10 0 0 0 0 0 0 0 0 0 0 0
- 76. ZIG-ZAG SCAN (FROM 2D TO 1D) 27 10 0 0 10 0 0 0 0 0 0 0 0 0 0 0 27,10,10,0,0,0,0,0,0,0,0,0,0,0,0,0
- 77. PROBABILITY OF EACH SYMBOL 27,10,10,0,0,0,0,0,0,0,0,0,0,0,0,0 symbols 27 10 0 probability 1/16 2/16 13/16
- 78. BITCODE symbols 27 10 0 probability 1/16 2/16 13/16 binary code 110 10 0
- 79. ENCODING BIT CODES *CAVLC example binary code 0 10 110 27,10,10,0,0,0,0,0,0,0,0,0,0,0,0,0 110 10 10 0 0 0 .... 16B 3B + symbol table
- 80. FROM MANY BYTES TO FEW BITS 010101101101 + code table 0001101100001010000010100000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000
- 81. ENCODER picture partitioning predictions transform quantization entropy coding redundancy removal entropy reduction lossless compression dct, dwt, intra-prediction, inter-prediction, motion estimation / compensation linear, logarithm huffman, lzw ... technique implementation
- 82. ENCODER picture partitioning predictions (subtract) transform quantization entropy coding picture reconstructing predictions (add) inverse transform dequantization entropy decoding DECODER
- 83. #qacodec 30 x 60 x 24 x 3 x 1080 x 1920 = 250.28GB time in sec colorfps resolution
- 84. #qacodec 30 x 60 x 24 x 3 x 1080 x 1920 = 250.28GB time in sec colorfps resolution 30 x 60 x 24 x 1.5 x 1080 x 1920 = 125.14GB time in sec colorfps resolution
- 85. #qacodec 30 x 60 x 24 x 3 x 1080 x 1920 = 250.28GB time in sec colorfps resolution 30 x 60 x 24 x 1.5 x 1080 x 1920 = 125.14GB time in sec colorfps resolution 30 x 60 x 24 x 0.127 x 1080 x 1920 = 1.32GB time in sec colorfps resolution
- 86. #qacodec 24 x 3 x 1080 x 1920 x 8 = 1.11Gbps colorfps resolution bits 24 x 0.127 x 1080 x 1920 ~= 6.02Mbps colorfps resolution
- 87. FINALLY, HOW NEWER CODECS BEAT OLDER ONES?
- 88. HEVC @ 2MbpsAVC @ 4Mbps
- 89. AVC @ 400kbps HEVC @ 400kbps
- 91. References ● https://github.com/leandromoreira/digital_video_introduction ● https://en.wikipedia.org/wiki/List_of_monochrome_and_RGB_p alettes ● https://globoplay.globo.com/os-dias-eram-assim/p/9955/ ● https://commons.wikimedia.org/wiki/File:Ressorts_de_compre ssion_coniques.jpg ● https://commons.wikimedia.org/wiki/File:Sony_DME7000_Digit al_Video_Multi_Effects_Processor_(13577533863).jpg ● https://commons.wikimedia.org/wiki/File:High-speed.jpg ● https://en.wikipedia.org/wiki/List_of_monochrome_and_RGB_p alettes ● https://www.youtube.com/watch?v=o0DYP-u1rNM ● http://silicon-valley.wikia.com/wiki/Pied_Piper_(company) ● http://www.nintendo.com/
- 92. LEANDRO MOREIRA leandromoreira.com.br Disclaimer: The views and opinions expressed in this presentation are those of the authors and do not necessarily reflect the policy or position of Globo. bit.ly/intro_codec SEE MORE: