Motion estimation overview
Download as PPT, PDF3 likes1,869 views
1. The document discusses and compares various motion estimation methods used in video compression standards, including translational and affine motion models. 2. It describes pixel domain block matching and frequency domain matching techniques. 3. It provides details on parameters for block matching motion estimation such as search area size, sub-pixel precision, and hierarchical and early termination techniques to improve efficiency.
Motion estimation overview
- 1. 1 Motion Estimation methods Review and comparison
- 2. 2 Motion Prediction Models Translational Model the prediction signal for each block is a block of same size MXN from other frames the prediction block is specified by the translational motion vector and reference frame(s) index. Affine Model motion in 3-D is described by affine transformations (a composition of scaling, rotation, sheering and translation).
- 3. 3 Translational Model Translational model maps rectangle to rectangle of same size and it is non- adequate for 3-D motion (e.g. imagine the case when a rectangular object quickly approaching to the camera, its size is increasing). This model works good for small motions: Subdivide current frame into blocks. Find one displacement vector for each block: Within a search range, find a “best match” that minimizes an error measure. In Translational model all pixels with a current block are displaced by same vector from the reference block. It’s like the reference block is displaced (without rotation and scaling) to a new position.
- 5. 5 Taxonomy of Motion Estimation Methods Pixel Domain Methods Matching algorithms Block Matching (most popular): full-search, Three Step, diamond etc. Feature Matching: Integral Projection matching, Successive Elimination Gradient-based algorithms pel-recursive block-recursive Frequency Domain Methods Phase correlation matching in wavelet domain matching in DCT domain
- 6. 6 Motion estimation Parameters Search Area in case of significant (or fast) motion large search area impacts significantly on Motion Estimation effectiveness. On the other hand ME complexity increases. Sub-pixel mode Motion is not limited to pixel granularity, therefore sub-pixel prediction (with accuracy up to 1/8 of pixel) is applied
- 7. 7 Motion estimation error measures
- 8. 8 Block Matching Motion Estimation Parameters Hierarchical Architecture: To reduce complexity and/or to pipeline Motion Estimation two hierarchical levels are commonly used: First stage: coarse motion estimation (usually on decimated search region) Second stage: fine motion estimation tuning around “best” coarse motion vectors obtained in the previous stage. Speed Up Techniques: Early termination – exclude current candidate if its preliminary cost exceeding the minimal cost (already obtained). Exclude candidates – not all candidates are checked (e.g. logarithmic search schemas).
- 9. 9 Inter Prediction Comparison: AVC/H.264, HEVC/H.265, VP9, AV1 Features AVC/H.264 HEVC/H.265 VP9 AV1 Square blocks Only yes no yes no Weighted prediction yes yes no yes Bi-Prediction yes yes Yes as superframe* Yes as superframe* Number of references Up to 16 (depending on level) Up to 16 (depending on level) 3 Up to 7 Sub-pixel Precision ¼-pel for luma 1/8-pel for chroma ¼-pel for luma 1/8-pel for chroma 1/8-pel for luma 1/16-pel for chroma 1/8-pel for luma 1/16-pel for chroma *To avoid patent infringements B-frame is coded as a couple of non-displayable frame plus displayable frame consisting of skip blocks. This pair of frames is called ‘superframe’
- 10. 10 Rectangular Prediction Blocks in HEVC/H.265 and AV1 HEVC/H.265 AV1 2 0 1 Horizontal A Split 0 1 2 Horizontal B Split Sub-blocks are not split further 0 1 2 Vertical A Split 1 2 0 Vertical B Split 0 1 2 3 Vertical 4:1 Split 0 1 2 3 Horizontal 4:1 Split nLx2N nRx2 N 2Nxn U 2Nxn D
- 11. 11 Rectangular Prediction Blocks in HEVC/H.265 and AV1 (cont.) 2NxnU 2NxnD nRx2NnLx2N Benefits of rectangular partitioning (HEVC( Benefits of rectangular partitioning (AV1(
- 12. 12 Sub-pixel Precision in AVC/H.264, HEVC/H.265, VP9 and AV1 AVC sub-pel precision: is ¼ for luma and ⅛ for chroma respectively (4:2:0(. The interpolation filters for both luma and chroma are fixed (non- adaptive(. For luma the interpolation is pipelined and it is executed in two non-balanced serial stages for each direction (horizontal and vertical(: 6-tap filter for half-pels (high complex( bilinear filter for quarter-pels (low complex( For chroma a fixed 4-tap filter is used for all fractional positions (similar to HEVC(. HEVC sub-pel precision: ¼ for luma and ⅛ for chroma respectively The interpolation filters for generating sub-pel data for both luma and chroma are fixed (non-adaptive(: For luma pixels a fixed 8-tap filter is applied for both half-pels and quarter-pels. The luma interpolation process is pipelined, it consists of two stages: horizontal and vertical filtering. For chroma a fixed 4-tap filter is used for all fractional positions.
- 13. 13 Sub-pixel Precision in H.264, H.265, VP9 & AV1 VP9 sub-pel precision ¼ for luma and ⅛ for chroma respectively (if 4:2:0(. The interpolation filters for generating sub-pel data can be adaptively chosen at frame-level, available filters kernels: Normal Smooth - slightly smooths or blurs the prediction block Sharp - slightly sharpens the prediction block. Interpolation filtering is pipelined: firstly a corresponding horizontal filter is used to build up a temporary array, and then at the second stage this array is vertically filtered to obtain the final prediction. Note: important advantage of HEVC over VP9 is a separation of filters for half and quarter pel (can be realized in stages, friendly for HW(.
- 14. 14 Sub-pixel Precision in H.264, H.265, VP9 & AV1 AV1 sub-pel precision Up-to 1/8-pel sub-pel precision for luma (1/8 and 1/16 precision for chroma respectively due to 4:2:0(, the precision level is specified at frame level. There are four interpolation kernels (up to 8 taps(, filter can be block- level adaptive: EIGHTTAP, EIGHTTAP_SMOOTH, EIGHTTAP_SHARP, BILINEAR Each filter is separable (i.e. filtering process is pipelined(: firstly horizontal filtering is performed and then vertical filtering. Interpolation filter can be fixed within a frame, in such case one of four kernels is selected at frame header. Interpolation filter can be switchable at block-level There is a special mode - dual filtering, where kernel for each direction can be different. Justification for dual filtering - signals can possess distinctive statistics in vertical and horizontal directions.
- 15. 15 Use Case: HEVC/H.265 Motion Estimation Details Variable inter-prediction block sizes – from 8x8 to 64x64, including non-square sizes like 32x16 (actually 4x8 and 8x4 blocks are also permitted with some constraints(. Chroma block sizes mimic luma, for 4:2:0 case with the scaling factor 1/2 (although for small luma blocks the scaling factor is 1(. Bi-directional prediction: two prediction blocks from previous and future pictures are mixed (averaged( to produce the final prediction signal (it’s a kind of interpolation(. weighted prediction (e.g. to compensate fading(. Sub-pixel precision: up to 1/4-th for luma and up to 1/8 for chroma
- 16. 16 Weighted Prediction in HEVC/H.265 Fwd Ref Horizont al Filter Vertical Filter << 6 << 2 8bits per pixel 10bits per pix Bwd Ref Horizont al Filter Vertical Filter << 6 << 2 8bits per pixel 10bits per pix Merge Predicted signal
- 17. 17 AV1 Motion Estimation Details AV1 supports Global Motion mode which is divided into the following categories: Translation (panning video) Rotation Zoom Affine (suitable for 3D motion) AV1 supports OBMC (Overlapped Block Motion Compensation) AV1 supports Warped motion per superblock Examples: In case of translation a global Motion Vector is applied for the whole frame. In case of Zoom and Rotation Motion Vector is depending on block location
- 18. 18 AV1 Motion Estimation Details – General Idea of OBMC Justification of OBMC - MV is most reliable in the center of the block (where prediction errors tend to be smaller than those at the corners). For a block it’s better to assign several MVs (its own and nearby blocks) and to blend reference samples: Block MV0 MV1 MV2 MV3 r c
- 19. 19 AV1 Motion Estimation Details – Technical Details of OBMC In AV1 OBMC predicted block is associated with a single vector MV0 corresponding to the block’s center while corner MVs are taken from causal (already decoded) neighbors. Blending is executed in two separable stages: firstly according to vertical direction and then according to horizontal direction (the filter coefficients are pre-defined in the AV1 spec.) shadow of block2 Block 1 block2 MV1 MV2 shadowofBlock1 MV0 Get prediction samples according to MV0 Get prediction samples for overlap area of block1 according to MV1 Get prediction samples for overlap area of block2 according to MV2 block3 block4 Shadow of block3 Shadow of4 MV3 MV4 Get prediction samples for overlap area of block3 according to MV3 Get prediction samples for overlap area of block4 according to MV4 blending blending