TWI872431B - Video coding method and apparatus thereof - Google Patents

Abstract

A video coder receives data for a block of pixels to be encoded or decoded as the current block of a current picture of a video. The video coder identifies multiple candidate coding modes applicable to the current block. The video coder identifies a first group of candidate coding modes that is a subset of the plurality of candidate coding modes. The first group of candidate coding modes may be the highest priority candidate coding modes identified based on cost. The number of candidate coding modes in the first group is less than the number of candidate coding modes in the plurality of candidate coding modes. The video coder selects a candidate coding mode in the first group of candidate coding modes. the video coder encodes or decodes the current block by using the selected candidate coding mode.

Description

Video encoding and decoding method and device

The present disclosure generally relates to video coding and decoding. In particular, the present disclosure relates to the sorting of candidate coding modes based on boundary matching.

Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims listed below and are not admitted to be prior art by reason of inclusion in this section.

High-Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion compensation DCT transform-like coding and decoding architecture. The basic unit of compression, called the coding unit (CU), is a 2Nx2N square block. Each CU can be recursively divided into four smaller CUs until a predetermined minimum size is reached. Each CU contains one or more prediction units (PUs).

Versatile Video Coding (VVC) is a codec designed to meet the upcoming needs of video conferencing, OTT (over-the-top) streaming, mobile phones, etc. VVC is designed to provide multiple functions to meet all video needs from low resolution and low bit rate to high resolution and high bit rate, high dynamic range (HDR), 360 omnidirectional, etc.

For each inter-frame predicted CU, motion parameters consisting of motion vectors, reference picture indices and reference picture list usage indices as well as additional information are used for inter-frame predicted sample generation. Motion parameters can be sent explicitly or implicitly. When the CU is encoded and decoded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no encoded motion vector increments or reference picture indices. A merge mode is specified, and the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as additional schedules introduced in VVC. The merge mode can be applied to any inter-frame predicted CU. An alternative to merge mode is explicit transmission of motion parameters, where motion vectors, the corresponding reference picture index for each reference picture list and the reference picture list usage flag, along with other required information, are sent explicitly per CU.

In addition to the inter-frame codec features in HEVC, VVC also includes several new and improved inter-frame prediction codec tools, listed below: - Extended merge prediction - Merge mode with MVD (MMVD) - Symmetric MVD (SMVD) delivery - Affine motion compensation prediction - Subblock-based temporal motion vector prediction (SbTMVP) - Adaptive motion vector resolution (AMVR) - Motion field storage: 1/16 precision luminance sample MV storage and 8x8 motion field compression - Bi-prediction with CU level weight (BCW) - Bi-directional optical flow (BCW) flow, BDOF for short) - Decoder side motion vector refinement (DMVR for short) - Geometric partitioning mode (GPM for short) - Combined inter and intra prediction (CIIP for short)

The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected but not all implementations are further described in the detailed description below. Therefore, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

Some embodiments provide a method for using cost to select a candidate codec mode to encode or decode a current block. A video codec receives data of a pixel block, which is to be encoded or decoded as a current block of a current picture of a video. The video codec identifies multiple candidate codec modes applicable to the current block. The video codec identifies a first candidate codec mode group, which is a subset of multiple candidate codec modes. The first candidate codec mode group can be the highest priority candidate codec mode identified based on the cost. The number of candidate codec modes in the first candidate codec mode group is less than the number of candidate codec modes in the multiple candidate codec modes. The video codec selects a candidate codec mode from the first candidate codec mode group. The video codec uses the selected candidate codec mode to encode or decode the current block.

In some embodiments, multiple candidate coding modes include merge candidates for the current block. The merge candidates for the current block may include (i) merge candidates using combined inter-frame and intra-frame predictions and/or (ii) merge candidates using affine transform motion compensation predictions. In some embodiments, multiple candidate coding modes include candidate coding modes corresponding to different combinations, which are used to refine the distance and offset of motion information. In some embodiments, multiple candidate coding modes include candidate coding modes corresponding to different linear models, and the candidate coding modes of the different linear models are used to derive a predictor of the chrominance samples of the current block based on the luminance samples of the current block. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different candidate weights for combining inter-frame predictions in different directions.

In some embodiments, the first candidate codec mode set is a highest priority candidate codec mode identified based on the cost of the plurality of candidate codec modes. The codec may index the candidate codec modes in the identified candidate codec mode set or assign codewords to the candidate codec modes in the identified candidate codec mode set according to the priority of the candidate codec mode. In some embodiments, the cost of the candidate codec mode is calculated by comparing: (i) reconstructed samples adjacent to the current block and (ii) predicted samples of the current block along the current block boundary generated according to the candidate codec mode.

In some embodiments, each candidate codec mode in a plurality of candidate codec modes is assigned to one candidate codec mode group in a plurality of candidate codec mode groups. For example, in some embodiments, each candidate codec mode in a plurality of candidate codec modes is associated with an original candidate index, wherein each candidate codec mode is assigned to one candidate codec mode group in K candidate codec mode groups based on the result of the original index modulo K or the original index divided by K. In some embodiments, candidate codec modes corresponding to spatial merge candidates are assigned to the same candidate codec mode group. In some embodiments, candidate codec modes corresponding to spatial merge candidates are assigned to different candidate codec mode groups. In some embodiments, candidate codec modes of merged candidates having motion differences less than a threshold are assigned to the same candidate codec mode group. In some embodiments, candidate codec modes of merged candidates having motion differences greater than a threshold are assigned to the same candidate codec mode group. In some embodiments, the encoder identifies a candidate codec mode group having a lowest representative cost among multiple candidate codec mode groups, and sends an index for selecting a candidate codec mode from the identified candidate codec mode group. The representative cost of the identified group can be an average, maximum, or minimum value of the costs (e.g., boundary matching costs) of the candidate codec modes of the identified group. In some embodiments, the encoder sends an index for selecting a set of candidate codec modes and identifies a candidate codec mode from the selected set of candidate codec modes based on a cost (e.g., a boundary matching cost) of the selected set of candidate codec modes.

In the following detailed description, many specific details are explained by way of example in order to provide a thorough understanding of the relevant teachings. Any variation, derivation and/or extension based on the teachings described herein are within the scope of protection of the present disclosure. In some cases, well-known methods, processes, components and/or circuits related to one or more example implementations disclosed herein may be described at a relatively high level without details to avoid unnecessarily obscuring aspects of the teachings of the present disclosure.

As video codecs improve, more codecs are developed. However, the codec gains of these different codecs are not additive. This is because (1) not all new codec modes are candidates for block modes, especially when considering syntax overhead, and (2) as the number of candidate modes for a block increases, longer codewords are required to indicate a codec mode from multiple candidate modes.

For example, after HEVC, new merge candidates (such as pairwise average merge candidates, HMVP merge candidates, etc.) are proposed to be added to the merge candidate list. The index of the best merge candidate is encoded/decoded to indicate the merge candidate selected by the current block. However, the number of merge candidates in the merge candidate list is limited to a predetermined number, so not all merge candidates can be added to the merge candidate list. As the number of merge candidates in the merge candidate list increases, the codeword length of the index of the best merge candidate also increases.

Some embodiments of the present disclosure provide a scheme for adaptively reordering candidate modes. According to the scheme, a video encoder/decoder calculates the cost of each candidate mode, which can be a merge candidate and/or a candidate mode of another tool. The video codec prioritizes the candidate modes based on the cost. (In some embodiments, candidate modes with smaller costs receive higher priorities. In some other embodiments, candidate modes with smaller costs receive lower priorities.) The candidate modes are then reordered based on the priority.

In some embodiments, the video codec uses a reduced, reordered set of candidate modes that includes only the top k candidate modes with higher priorities (k < the number of all possible candidate modes). As the number of candidate modes in the candidate mode set is reduced, the syntax used to indicate the selected candidate mode is also reduced. An index can be used to refer to the candidate mode selected after reordering, so that a smaller index value can refer to a candidate mode with a higher priority. (In other words, the index value refers to the index of the candidate mode in the candidate list, and after applying the reordering, the index value refers to the reordered index of the candidate mode.) In some embodiments, for candidate modes with higher priorities, shorter codewords are used for encoding/decoding. The candidate mode with the highest priority can be implicitly set as the codec mode for the current block.

In some embodiments, the video codec prioritizes candidate modes in a candidate mode set based on boundary matching. For each candidate mode, a boundary matching cost for encoding and decoding the current block using the candidate mode is calculated, and the priority of the candidate mode in the reordered candidate mode list is determined according to the boundary matching cost of the candidate mode.

Ⅰ , boundary matching cost

The boundary matching cost of a candidate model refers to the discontinuity measure (including top boundary matching and/or left boundary matching) between the current prediction (prediction sample or predictor of the current block) and the neighboring reconstruction (reconstruction samples in one or more neighboring blocks) generated from the candidate model. Top boundary matching refers to the comparison of the current top prediction sample with the neighboring top reconstruction samples, and left boundary matching refers to the comparison of the current left prediction sample with the neighboring left reconstruction samples.

Figure 1 shows the reconstructed neighboring samples and predicted samples used in boundary matching. As shown in the figure, pred _x,0 is the predicted sample along the top boundary, and rec _x,-1 is the reconstructed neighboring sample along the top boundary; where pred _0,y is the predicted sample along the left boundary, and rec _-1,y is the reconstructed neighboring sample along the left boundary.

In some embodiments, a predetermined subset of the current prediction is used to calculate the boundary matching cost. For example, n rows of the top boundary within the current block and/or m rows of the left boundary within the current block can be used. In some of these embodiments, _n2 rows of the top neighbor reconstruction and/or _m2 rows of the left neighbor reconstruction can be used for boundary matching.

The example boundary matching cost is calculated according to the following formula, where n = 2, m = 2, n ₂ = 2, and m ₂ = 2:

In this example, two rows of predicted samples and two rows of reconstructed neighboring samples along the top and left boundaries are used to compute the difference measure (or similarity measure). The weights (a, b, c, d, e, f, g, h, i, j, k, l) can be any positive integer, for example, a = 2, b = 1, c = 1, d = 2 , e = 1, f = 1, g = 2, h = 1, i = 1, j = 2, k = 1, l = 1. Another example of computing the boundary matching cost is as follows, where n = 2, m = 2, n ₂ = 1, m ₂ = 1:

The weights (a, b, c, g, h, i) can be any positive integer, for example a = 2, b = 1, c = 1, g = 2, h = 1, i = 1. Another example of calculating the boundary matching cost is as follows, where n = 1, m = 1, n ₂ = 2, m ₂ = 2:

The weights (d, e, f, j, k, l) can be any positive integer, for example, d = 2, e = 1, f = 1, j = 2, k = 1, l = 1. Another example of calculating the boundary matching cost is as follows, where n = 1, m = 1, n ₂ = 1, m ₂ = 1:

The weights (a, c, g, i) can be any positive integer, for example a = 1, c = 1, g = 1, i = 1. Another example of calculating the boundary matching cost is as follows, where n = 2, m = 1, n ₂ = 2, m ₂ = 1:

where the weights (a, b, c, d, e, f, g, i) can be any positive integer, for example, a = 2, b = 1, c = 1, d = 2, e = 1, f = 1, g = 1, i = 1. Another example of calculating the boundary matching cost is as follows, where n = 1, m = 2, n ₂ = 1, m ₂ = 2:

Wherein the weights (a, c, g, h, i, j, k, l) can be any positive integer, for example, a = 1, c = 1, g = 2, h = 1, i = 1, j = 2, k = 1, l = 1. In general, n or _n2 can be any positive integer such as 1, 2, 3, 4, etc., and m or _m2 can be any positive integer such as 1, 2, 3, 4, etc. In some embodiments, n and/or m vary with block width, height or area. For example, for larger blocks (area>threshold), m becomes larger, and the threshold can be 64, 128 or 256. When area>threshold, m can be increased from 1 to 2, or from 1 or 2 to 4. For another example, for taller blocks (height>threshold*width), m becomes larger and/or n becomes smaller, and the threshold can be 1, 2, or 4; when height>threshold*width, m may increase from 1 to 2, or from 1 or 2 to 4. For another example, for larger blocks (area>threshold), n becomes larger, and the threshold may be 64, 128, or 256; when area>threshold, n increases from 1 to 2, or from 1 or 2 to 4. For another example, for wider blocks (width>threshold*height), n becomes larger and/or m becomes smaller, and the threshold = 1, 2, or 4; when width>threshold*height, n increases from 1 to 2; when width>threshold*height, n increases from 1 or 2 to 4.

For example, n and/or m may be defined in a video codec standard or depend on a codec video syntax sent or parsed at a CU/CB, PU/PB, TU/TB, CTU/CTB, tile, slice, picture level, sps level and/or pps level.

In some embodiments, when the current block is at the top boundary within a CTU row, top boundary matching is not used and/or only left boundary matching is used. (Adjacent reconstructed samples at cross-CTU rows are not used.) In some embodiments, when the current block is at the left boundary within a CTU, left boundary matching is not used and/or only top boundary matching is used. In some embodiments, when the current block is taller (height>threshold*width), only left boundary matching is used. In some embodiments, when the current block is wider (width>threshold*height), only top boundary matching is used.

In some embodiments, the upper left neighboring reconstructed sample (reco _-1,1 ) can be used for boundary matching. For example, the boundary matching cost is added with the following term:

In some embodiments, when calculating the boundary matching cost, the residual can also be added to the current prediction (or the prediction sub-block of the current block) to reconstruct the samples of the current block, and the reconstructed samples of the current block are used to calculate the boundary matching cost. For example, the residual is generated by restoring the DC and/or all AC coefficients or any subset of the AC coefficients after the transformation process. The transformation process can use any predetermined transformation kernel for secondary transformation and/or primary transformation. For example, the transformation kernel used for secondary transformation refers to the Low Frequency Non-Separable Transform (LFNST) transformation kernel. For example, the transformation kernel used for primary transformation refers to DCT2 and/or any transformation kernel used for Multiple Transform Selection (MTS), such as DST7. For example, a transformation kernel refers to the actual transformation applied in the transformation module of the current block.

Ⅱ , as a candidate for merging with the candidate model

Some embodiments of the present disclosure provide a scheme for reordering and/or reducing merge candidates. In some embodiments, the index of the best merge candidate (index_best_merge) does not refer to the order of the merge candidates in the merge candidate list, but refers to the priority based on the boundary matching cost. In some embodiments, only the top k merge candidates (with higher priority) can be candidate modes.

For example, for a merge candidate list {cand0, cand1, cand2, cand3, cand4, cand5, ...}, the boundary matching cost of the merge candidates is calculated as {cost_cand0, cost_cand1, cost_cand2, ...}, such that cost_cand0 refers to the boundary matching cost of cand0, cost_cand1 refers to the boundary matching cost of cand1, cost_cand2 refers to the boundary matching cost of cand2, and so on. The video codec then reorders {cand0, cand1, cand2, ...} according to the boundary matching cost. For example, if cost_cand0 > cost_cand1 > cost_cand2 > cost_cand3 > cost_cand4 > cost_cand5, the reordered merge candidates become {cand5, cand4, cand3, cand2, cand1, cand0}, index_best_merge 0 can refer to cand5 (the merge candidate with the smallest cost is sent using the shortest codeword), index_best_merge 1 can refer to cand4, index_best_merge 2 can refer to cand3, and so on.

In some embodiments, the candidate pattern set includes only the top k merge candidates (sorted by boundary matching cost), rather than being determined by a constant (e.g., MaxNumMergeCand in VVC, which can be 6). For example, in some embodiments, k can be 4, and the signaling of each merge candidate can include: index_best_merge 0 refers to cand5 with codeword 0 (the merge candidate with the largest cost is sent using the shortest codeword); index_best_merge 1 refers to cand4 with codeword 10; index_best_merge 2 refers to cand3 with codeword 110; and index_best_merge 3 refers to cand2 with codeword 111. Otherwise, for example, if cost_cand0＜cost_cand1＜cost_cand2＜cost_cand3＜cost_cand4＜cost_cand5, the order of the merged candidates can be the same as the original list without reordering: index_best_merge 0 refers to cand0 with codeword 0; index_best_merge 1 refers to cand1 with codeword 10; index_best_merge 2 refers to cand2 with codeword 110; and index_best_merge 3 refers to cand3 with codeword 111, and so on.

In some embodiments, the merge candidate with the largest cost is sent using the shortest codeword. For another example, if cost_cand0 < cost_cand1 < cost_cand2 < cost_cand3 < cost_cand4 < cost_cand5, the reordered merge candidates become {cand5, cand4, cand3, cand2, cand1, cand0}, so that index_best_merge 0 refers to cand5 with codeword 0 (maximum cost using the shortest codeword); index_best_merge 1 refers to cand4 with codeword 10; and index_best_merge 2 refers to cand3 with codeword 110.

In some sub-embodiments, whether to use the reduced and reordered merge candidate list (reordered based on boundary matching cost and limited to k candidates) or the original merge candidate list (without reordering) is determined based on predetermined rules (e.g., implicitly based on block width, block height, or block area, or explicitly based on CU/CB, PU/PB, TU/TB, CTU/CTB, tile, slice level, picture level, SPS level, and/or PPS level). When the reduced and reordered merge candidate list is used, the entropy coding decoding context used for transmission may be different from that when the original merge candidate list is used.

In some embodiments, smaller values of the index_best_merge variable are encoded with a shorter codeword length. The index_best_merge variable can be encoded with a truncated unary codeword.

In some embodiments, the reordering is applied only to a subset of the merge candidate list. For example, the subset may refer to the original first n candidates cand0, cand1, cand2, so that index_best_merge 0/1/2 refers to the priority based on boundary matching, and index_best_merge 3/4/5 refers to the original cand 3/4/5. For another example, the subset refers to the original last n candidates cand3, cand4, cand5. Then index_best_merge 3/4/5 refers to the priority based on boundary matching, and index_best_merge 0/1/2 refers to the original cand 0/1/2. For another example, the subset may refer to the spatial merge candidate.

In some embodiments, the best merge candidate is inferred to be the merge candidate with the minimum boundary matching cost among all merge candidates. In some embodiments, the best merge candidate is inferred to be the merge candidate with the maximum boundary matching cost among all merge candidates. In some of these embodiments, index_best_merge is not sent/parsed by the encoder/decoder and can be inferred to be 0.

In some embodiments, the merge candidates are divided into several groups. The boundary matching cost of each group is calculated. The promising group is implicitly identified as the group with the highest priority. If more than one merge candidate is included in the identified promising group, a reduced merge index is sent/parsed to indicate the merge candidates from the promising group. Since the number of merge candidates in each group is less than the number of merge candidates in the original merge candidate list, the reduced merge index will occupy fewer codewords than the original merge index.

In some embodiments, if k groups are used and the number of merge candidates in the original merge candidate list is N, the number of merge candidates in each group is "N/k" and the grouping rule depends on the merge index and/or merge type. For example, merge candidates with the same "merge index % k" (remainder of the modulus operator) value are in the same group. For another example, merge candidates with the same "merge index / k" (quotient of the division operator) value are in the same group. For another example, spatial merge candidates are in the same group. For another example, k varies with block width, block height and/or block area. For another example, merge candidates with small motion difference are in the same group. Motion difference includes mv difference and/or reference image difference. An example of calculating the mv difference (denoted as mv_diff) between candidate 0 and candidate 1 is: mv_diff = |(mvx_cand0 –mvx_cand1)| + |(mvy_cand0 –mvy_cand1)|

If the reference images are identical and/or the mv difference is less than a predetermined threshold, the motion difference is small.

In some embodiments, when a group contains more than one merge candidate, the cost of the group (the representative cost of the group) is the average cost of the costs of all merge candidates in the group. In some sub-embodiments, when a group contains more than one merge candidate, the representative cost of the group is the average, maximum, or minimum cost of the costs from all merge candidates in the group.

In some embodiments, the merge candidates are divided into several subsets, and the selection of the subsets is explicitly sent to the decoder. In addition, the selection of the merge candidates in the merge candidate subset is performed by using the boundary matching cost. The subset partitioning rule may depend on the merge index and/or the merge type. For example, merge candidates with the same "merge index % k" value are in the same subset. For another example, merge candidates with the same "merge index / k" value are in the same subset (k may vary with block width, block height and/or block area.) For another example, spatial merge candidates are in different subsets. For another example, merge candidates with larger motion differences are in the same subset. Motion differences include mv differences and/or reference picture differences. An example of calculating the mv difference (denoted as mv_diff) between candidate 0 and candidate 1 is: mv_diff = |(mvx_cand0 –mvx_cand1)| + |(mvy_cand0 –mvy_cand1)|

If the reference images are different and/or the mv difference is greater than a predetermined threshold, the motion difference is large.

In some embodiments, the merge candidates of a CU include one or more of the following candidates: (1) spatial merge candidates or spatial MVPs from spatially neighboring CUs, (2) temporal MVPs from co-located CUs, (3) history-based MVPs from FIFO tables, (4) pairwise average MVPs, and/or (5) zero MV.

In some embodiments, the merge candidates in this section refer to merge candidates for combined inter and intra prediction (CIIP). The prediction samples within the current block are generated for CIIP processing. In some embodiments, the merge candidates in this section refer to merge candidates for sub-block merge candidates, such as affine merge candidates. The prediction samples within the current block are generated by affine processing, for example, block-based affine transformation motion compensation prediction.

a. Spatial Merge Candidates

At most four merge candidates are selected from the candidates located at positions surrounding the CU, as shown in Figure 2, which shows the positions of the spatial merge candidates. The derivation order is B ₀ , A ₀ , B ₁ , A ₁ , B ₂ . Position B ₂ is considered only when one or more CUs at positions B ₀ , A ₀ , B ₁ , A ₁ are not available (for example, because it belongs to another fragment or tile) or is encoded and decoded within the frame. After the candidate at position A ₁ is added, a redundancy check is performed on the addition of the remaining candidates to ensure that candidates with the same motion information are excluded from the list, thereby improving encoding and decoding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Candidates are added to the list only if the corresponding candidates used for redundancy verification do not have the same motion information. Only the following candidate pairs are considered: (A ₁ , B ₁ ), (A ₁ , A ₀ ), (A ₁ , B ₂ ), (B ₁ , B ₀ ), (B ₁ , B ₂ ).

b. Time merge candidates

Only one temporal merge candidate is added to the merge candidate list. Specifically, in the derivation of the temporal merge candidate, a scaled motion vector is derived based on a co-located CU, which belongs to a co-located reference picture. The reference picture list and reference index used to derive the co-located CU are explicitly sent in the slice header. As shown by the dotted line in Figure 3, the scaled motion vector of the temporal merge candidate is obtained, which shows that the motion vector of the temporal merge candidate is scaled. The scaled motion vector is scaled from the motion vector of the co-located CU using the picture order count (POC) distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. The position of the temporal merge candidate is selected between candidates C ₀ and C ₁ , as shown in Figure 4, which shows the candidate positions of the temporal merge candidate for the current block. If the CU at position C ₀ is not available, is intra-coded or is outside the current CTU row, then position C ₁ is used for the temporal merge candidate. Otherwise, position C ₀ is used to derive temporal merge candidates.

c. History-based Merger Candidates

History-based MVP (history-based motion vector prediction, HMVP for short) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previously encoded and decoded blocks is stored in a table and used as the MVP of the current CU. In the encoding/decoding process, a table with multiple HMVP candidates is maintained. When a new CTU row is encountered, the table will be reset (cleared). Whenever there is a non-sub-block inter-frame encoded and decoded CU, the associated motion information will be added to the last entry of the table as a new HMVP candidate.

The HMVP table size S is set to 6, which means that up to 5 history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a constrained first-in-first-out (FIFO) rule is applied, where a redundancy check is first applied to check if the same HMVP exists in the data table. If found, the same HMVP is removed from the table and all subsequent HMVP candidates are moved forward, and the same HMVP is inserted into the last entry of the table.

HMVP candidates can be used in the merge candidate list construction process. The latest few HMVP candidates in the table are checked in turn and inserted into the candidate list after the TMVP candidates. Redundancy checks are applied to multiple candidates from HMVP candidates to spatial or temporal merge candidates.

In some embodiments, in order to reduce the number of redundant verification operations, the last two entries in the table perform redundant verification on the _A1 and _B1 spatial candidates respectively. Once the total number of available merge candidates reaches the maximum number of allowed merge candidates minus 1, the HMVP merge candidate list construction process is terminated.

d. Pairwise average merge candidates

The pairwise average candidate is generated by averaging a predetermined candidate pair in the existing merge candidate list using the first two merge candidates. The first merge candidate is defined as p0Cand and the second merge candidate can be defined as p1Cand. The average motion vector is calculated based on the availability of the motion vectors of p0Cand and p1Cand for each reference list respectively. If two motion vectors are available in one list, even if the two motion vectors point to different reference pictures, they are averaged, and their reference pictures are set to the reference pictures belonging to or used for one of p0Cand and p1Cand (e.g., set to the reference picture of p0Cand or set to the reference picture of p1Cand)"; if only one motion vector is available, the merge candidate corresponding to that motion vector is used directly; if no motion vector is available, this list is kept invalid. In addition, if the half-pel inerpolation filter index of p0Cand and p1Cand is index), it is set to 0. In some embodiments, when the merge list is not full, after the pairwise average merge candidates are added, the last zero MVP is inserted until the maximum number of merge candidates is encountered.

e. Combined inter- and intra-frame prediction ( Combined inter and intra prediction , abbreviation CIIP ）

When the CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if the CU width and the CU height are both less than 128 luma samples, an additional flag may be sent to indicate whether the combined inter-frame/intra-frame prediction (CIIP) mode is applied to the current CU. The CIIP prediction combines the inter-frame prediction signal with the intra-frame prediction signal. The inter-frame prediction signal P _inter in the CIIP mode is derived using the same inter-frame prediction processing as applied to the conventional merge mode; and the intra-frame prediction signal P _intra is derived according to the conventional intra-frame prediction processing, and the one or more intra-frame prediction modes used are the planar mode or one or more intra-frame prediction modes derived from a predetermined mechanism. For example, in the following, the predetermined mechanism is based on the neighboring reference areas (templates) of the current block. The intra-frame prediction mode of the CU is implicitly derived from the neighboring templates at the encoder and decoder, rather than being sent to the decoder as exact intra-frame prediction mode bits. A prediction sample of the template is generated for each template reference sample of the candidate mode. The cost is calculated as the SATD between the predicted sample and the reconstructed sample of the template. The intra-frame prediction mode with the minimum cost and/or some intra-frame prediction modes with smaller costs are selected and used for intra-frame prediction of the CU. The candidate modes can be all MPMs and/or any subset of MPMs, such as 67 intra-frame prediction modes in VVC or extended to 131 intra-frame prediction modes. The intra-frame and inter-frame prediction signals are combined using a weighted average, where the weights are calculated based on the coding and decoding modes of the top and left neighboring blocks. The CIIP prediction P _CIIP is constructed as follows: (wt is the weight value)

f. Affine merge candidates

Objects in the video may have different types of motion, including translation, zooming in/out, rotation, perspective, and other irregular motions. In some embodiments, block-based affine transformation motion compensation prediction is used to solve these different types of motions. VVC provides block-based affine transformation motion compensation prediction. Specifically, the affine motion field of the current block can be expressed with motion information of two control points (e.g., at the upper right and upper left corners of the block) (4 parameters) or motion information of three control points (e.g., at the upper right, upper left, and lower left corners of the block) (6 parameters). For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

Where (mv _0x ,mv _0y ) is the motion vector of the upper left control point, (mv _1x ,mv _1y ) is the motion vector of the upper right control point, and (mv _2x ,mv _2y ) is the motion vector of the lower left control point.

Affine merge mode, or AF_MERGE mode, can be applied to CUs with width and height greater than or equal to 8. In this mode, the motion vector at the control points (CPMV) of the current CU is generated based on the motion information of the spatially neighboring CUs. There can be up to five CPMVP candidates, and so one is sent to indicate the one to be used for the current CU. The following three types of CPMV candidates are used to form the affine merge candidate list: (1) inherited affine merge candidates, inferred from the CPMV of neighboring CUs; (2) constructed affine merge candidate CPMVPs, which are derived using the translation MV of neighboring CUs; (3) zero MV.

In VVC, there are at most two inherited affine candidates, which come from the affine motion models of neighboring blocks, one from the left neighboring CU (left predictor) and one from the top neighboring CU (top predictor). For the left predictor, the scanning order is A0->A1, and for the top predictor, the scanning order is B0->B1->B2. Only the first inherited candidate on each side is selected. No pruning check is performed between two inherited candidates. When the adjacent affine CU is identified, its control point motion vectors are used to derive the CPMVP candidate in the affine merge list of the current CU. When the adjacent affine CU is encoded and decoded using the affine mode with 6 parameters, the three CPMVs of the current CU are calculated based on the motion vectors of the upper left corner, upper right corner, and lower left corner of the adjacent affine CU. When the adjacent affine CU is encoded and decoded using the affine mode with 4 parameters, the two CPMVs of the current CU are calculated based on the motion vectors of the upper left corner and upper right corner of the adjacent affine CU.

The constructed affine candidates are constructed by combining the neighboring translation motion information of each control point. The motion information of the control points comes from the spatial neighbors (A0, A1, A2, B0, B1, B2, B3) and temporal neighbors of the current block. CPMV _k (k=1, 2, 3, 4) represents the MV of the kth control point. For CPMV1, B2->B3->A2 blocks are checked and the MV of the first available block is used. For CPMV ₂ , B1->B0 blocks are checked, and for CPMV3, A1->A0 blocks are checked. If TMVP is available, it is used as CPMV4.

After obtaining the MVs of the four control points, an affine merge candidate is constructed based on this motion information. The following combinations of control point MVs are used for construction in sequence: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2 }, {CPMV1, CPMV3}. A combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and a combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid motion scaling processing, if the reference indices of the control points are different, the relevant combination of control point MVs is discarded. After the inherited affine merge candidates and constructed affine merge candidates are checked, if the candidate list is still not full, a zero MV is inserted at the end of the list.

III , MMVD As a candidate model

Merge Mode with Motion vector Difference (MMVD) is a new codec for the Versatile Video Coding (VVC) standard. Unlike the regular merge mode, where the implicitly derived motion information is directly used to generate prediction samples for the current CU, in MMVD the derived motion information is further refined by motion vector difference (MVD). MMVD also expands the candidate list for the merge mode by adding additional MMVD candidates based on a predetermined offset (also called MMVD offset).

After sending the skip flag and merge flag to specify whether the MMVD mode is used for the CU, the MMVD flag can be sent. If the MMVD mode is used, the selected merge candidate will be refined by the MVD information. The MVD information also includes the merge candidate flag, a distance index for specifying the magnitude of motion, and an index for indicating the direction of motion. The merge candidate flag is sent to specify which of the first two merge candidates is to be used as the starting MV.

The distance index is used to specify the motion magnitude information by indicating a predetermined offset from the starting MV. The offset can be added to the horizontal component or the vertical component of the starting MV. An example mapping from the distance index to the predetermined offset is specified in the following Table III-1: Table III-1 . Distance Index Distance Index 0 1 2 3 4 5 6 7 Offset ( in luminance samples ) 1/4 1/2 1 2 4 8 16 32  

The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate one of the four directions shown in Table III-2. Table III-2. MV offset symbols specified by the direction index Distance Index 00 01 10 11 x- axis + – N/A N/A y- axis N/A N/A + –  

It should be noted that the meaning of the MVD flag may be different depending on the information of the starting MV. When the starting MV is a uni-prediction MV or a bi-prediction MV, both lists point to the same side of the current picture (i.e., the picture order counts or POCs of the two reference pictures are both greater than the POC of the current picture, or both are less than the POC of the current picture), and the symbols in Table III-2 specify the signs of the MV offsets added to the starting MV. When the starting MV is a bi-prediction MV, the two MVs point to different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture), and each symbol in Table III-2 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the symbols of the list 1 MV have opposite values. In some embodiments, the predetermined offset of the MMVD candidate (MmvdOffset) is derived from or expressed as a distance value (MmvdDistance) and a direction sign (MmvdSign or MmvdDirection).

FIG. 5 conceptually illustrates MMVD candidates and their corresponding offsets. The figure shows a merge candidate 510 as a starting MV and several MMVD candidates in the vertical and horizontal directions. Each MMVD candidate is derived by applying an offset to the starting MV 510. For example, MMVD candidate 522 is derived by adding an offset of 2 to the horizontal component of merge candidate 510, and MMVD candidate 524 is derived by adding an offset of -1 to the vertical component of merge candidate 510. MMVD candidates with an offset in the horizontal direction, such as MMVD candidate 522, are referred to as horizontal MMVD candidates. MMVD candidates with an offset in the vertical direction, such as MMVD candidate 524, are referred to as vertical MMVD candidates.

In some embodiments, a candidate mode reduction/reordering scheme is used to reorder MMVD candidates to improve specific syntax elements. Below is the syntax table of MMVD in the VVC standard. if( mmvd_merge_flag[ x0 ][ y0 ] = = 1 ) { if( MaxNumMergeCand > 1 ) mmvd_cand_flag [ x0 ][ y0 ] ae(v) mmvd_distance_idx [x0][y0] ae(v) mmvd_direction_idx [x0][y0] ae(v)

The syntax element mmvd_cand_flag [x0][y0] specifies whether the first (0) or second (1) candidate in the merge candidate list with the motion vector difference derived from the syntax elements mmvd_distance_idx[x0][y0] and mmvd_direction_idx is used. The array indices x0, y0 specify the position (x0, y0) of the top-left luma sample in the considered codec block relative to the top-left luma sample in the picture. When mmvd_cand_flag[x0][y0] is not present, it is inferred to be equal to 0.

The syntax element mmvd_distance_idx [x0][y0] specifies the index used to derive the variable MmvdDistance[x0][y0] as described below. The array index x0, y0 specifies the position (x0, y0) of the top-left luma sample in the codec block under consideration relative to the top-left luma sample in the picture. mmvd_distance_idx[ x0 ][ y0 ] MmvdDistance[ x0 ][ y0 ] ph_mmvd_fullpel_only_flag = = 0 ph_mmvd_fullpel_only_flag = = 1 0 1 4 1 2 8 2 4 16 3 8 32 4 16 64 5 32 128 6 64 256 7 128 512

The syntax element mmvd_direction_idx[x0][y0] specifies the index used to derive the variable MmvdSign[x0][y0] as described below. The array index x0, y0 specifies the position (x0, y0) of the top-left luma sample relative to the top-left luma sample of the picture in the codec block under consideration. mmvd_direction_idx[x0][y0] MmvdSign[ x0 ][ y0 ][ 0 ] MmvdSign[ x0 ][ y0 ][ 1 ] 0 +1 0 1 −1 0 2 0 +1 3 0 −1

In some embodiments, the syntax element mmvd_cand_flag can be improved by using a reduced/reordered candidate mode list. Specifically, a boundary matching cost is calculated for each MMVD mode, including MMVD mode 0, where the MMVD with the first candidate is in the merged candidate list; and MMVD mode 1, where the MMVD with the second candidate is in the merged candidate list. In some embodiments, after reordering according to cost, if the cost of MMVD mode 0 > the cost of MMVD mode 1, then mmvd_cand_flag equals 0 to indicate MMVD mode 1 and mmvd_cand_flag equals 1 to indicate MMVD mode 0. Optionally, mmvd_cand_flag is implicit, and the first or second candidate (with the smallest cost) in the merged candidate list is used for the MMVD.

In some embodiments, after reordering according to boundary matching cost, if the cost of MMVD mode 0 < the cost of MMVD mode 1, mmvd_cand_flag equals 0 to indicate MMVD mode 1 and mmvd_cand_flag equals 1 to indicate MMVD mode 0. Alternatively, mmvd_cand_flag is implicit and the first or second candidate (with the largest cost) in the merge candidate list is used for MMVD.

In some embodiments, the sending of the syntax elements mmvd_cand_flag, mmvd_distance_idx, and mmvd_direction_idx can be improved by using a reduced/reordered list of candidate modes. A joint indication (denoted as MMVD_joint_idx) is sent/parsed/assigned to specify a selected combination of mmvd_cand_flag, mmvd_distance_idx, and mmvd_direction_idx. The value of MMVD_joint_idx ranges from 0 to the value calculated by "MMVD candidate number" (e.g., 2) * "MMVD distance number" (e.g., 8) * "MMVD direction number" (e.g., 4) minus one (e.g., 63).

In some embodiments, a boundary matching cost is calculated for each MMVD combination. After reordering different MMVD combinations according to cost, MMVD_joint_idx can be used to select an MMVD combination based on the reordering. For example, in some embodiments, if the cost of MMVD combination 0 > the cost of MMVD combination 1 > ..., then MMVD_joint_idx equals 0 to represent MMVD combination 63, and MMVD_joint_idx equals 63 to represent MMVD combination 0. In some embodiments, MMVD_joint_idx is implicit, and the MMVD combination with the smallest cost is used for MMVD. In some embodiments, the number of MMVD combinations is reduced and MMVD combinations with smaller costs are retained in the candidate pattern set. The codewords used to send/parse MMVD_joint_idx are thereby reduced.

In some embodiments, after reordering different MMVD combinations according to cost, MMVD_joint_idx can be used to select an MMVD combination based on the reordering. In some embodiments, if the cost of MMVD combination 0 < the cost of MMVD combination 1 < ..., then MMVD_joint_idx equal to 0 refers to MMVD combination 63, and MMVD_joint_idx equal to 63 refers to MMVD combination 0. In some embodiments, MMVD_joint_idx is implicit and the MMVD combination with the largest cost is used for MMVD. In some embodiments, the number of MMVD combinations is reduced, and the MMVD combinations with larger costs are retained in the candidate mode set. The codewords used to send/parse MMVD_joint_idx are thereby reduced. A similar approach can be used to improve the sending of mmvd_distance_idx and/or mmvd_direction_idx.

IV. BCW Weights as candidate models

Bi-prediction with CU-level Weight (BCW) is a coding tool for enhancing bi-directional prediction. BCW allows different weights to be applied to L0 prediction and L1 prediction, which are then combined to generate the bi-directional prediction of the CU. For a CU to be coded and decoded by BCW, a weighting parameter w is sent for L0 and L1 predictions so that the bi-directional prediction result Pbi-pred is calculated based on w according to the following formula:

P0 represents the pixel value predicted by L0 MV (or L0 prediction). P1 represents the pixel value predicted by L1 MV (or L1 prediction). P _bi-pred is the weighted average of P0 and P1 according to w. For low-latency pictures, that is, pictures using reference frames with small picture order counts (POCs), possible values of w include {-2, 3, 4, 5, 10}, which are also referred to as BCW candidate weights. For non-low-latency pictures, possible values of w (BCW candidate weights) include {3, 4, 5}. In some embodiments, in order to find the best w for encoding and decoding the current CU, instead of searching for all possible values of w for all candidate bi-prediction MV positions, the LC-RDO stage can adopt an interlaced search mode to search for the optimal value of the BCW weighting parameter w. More weights can be supported as shown below. For example, for merge mode, the weights are expanded from {-2, 3, 4, 5, 10} to {-4, -3, -2, -1, 1, 2, 3, 4, 5, 6, 7 , 9, 10, 11, 12} or any subset above. When negative bidirectional prediction weights are not supported, the weights for merge mode are expanded from {-2, 3, 4, 5, 10} to {1, 2, 3, 4, 5, 6, 7}. In addition, the negative bidirectional prediction weights for non-merge mode are replaced with positive weights, that is, the weights {-2, 10} are replaced with {1, 7}.

In some embodiments, the proposed candidate mode reduction/reordering scheme is used to reorder BCW candidates to improve the transmission of specific syntax elements, such as bcw_idx. Below is the syntax table of BCW in the VVC standard. if( sps_bcw_enabled_flag && inter_pred_idc[ x0 ][ y0 ] = = PRED_BI && luma_weight_l0_flag[ ref_idx_l0 [ x0 ][ y0 ] ] = = 0 && luma_weight_l1_flag[ ref_idx_l1 [ x0 ][ y0 ] ] = = 0 && chroma_weight_l0_flag[ ref_idx_l0 [ x0 ][ y0 ] ] = = 0 && chroma_weight_l1_flag[ ref_idx_l1 [ x0 ][ y0 ] ] = = 0 && cbWidth * cbHeight ＞= 256 ) bcw_idx [ x0 ][ y0 ] ae(v)

The syntax element bcw_idx [x0][y0] specifies the weight index for bidirectional prediction with CU weights. The array index x0, y0 specifies the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture. For each bidirectionally predicted CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is sent after the motion vector difference; 2) for merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. Syntax structure Syntax elements Binarization handle Input parameters bcw_idx[ ][ ] TR cMax = NoBackwardPredFlag ? 4: 2, cRiceParam = 0

In order to reduce and reorder the BCW candidates, the boundary matching cost of each BCW candidate weight is calculated, and the different BCW candidate weights are reordered according to the cost. In one embodiment, the cost of the predetermined candidate can be reduced. For example, the predetermined candidate refers to a candidate inferred from one or more adjacent blocks. In another embodiment, the candidate weights (only) include two adjacent bidirectional prediction weights (i.e., ±1) and inherited (inferred) bidirectional prediction weights. In some embodiments, bcw_idx equals 0 to represent the BCW candidate weight with the smallest cost, and bcw_idx equals 4 to represent the BCW candidate weight with the largest cost. In some embodiments, bcw_idx is implicit, and the BCW candidate weight with the smallest cost is used. In some embodiments, bcw_idx equals 0 to indicate the BCW candidate weight with the largest cost, and bcw_idx equals 4 to indicate the BCW candidate weight with the smallest cost. In some embodiments, bcw_idx is implicit, and the BCW candidate weight with the largest cost is used.

In some embodiments, only the first k BCW weights (with higher priority due to cost) can be used as candidate weights in the candidate pattern set (the original number of BCW weights is 5). In some embodiments, the number of BCW weights in the candidate pattern set is k (e.g., 3), and each BCW weight is sent as follows: bcw_idx 0 refers to the BCW candidate with the highest priority using codeword 0; bcw_idx 1 refers to the BCW candidate with the second highest priority using codeword 10; bcw_idx 2 refers to the BCW candidate with the third highest priority using codeword 11.

In some sub-embodiments, whether to use the reduced and reordered BCW candidates (reordered based on boundary matching cost and limited to k candidates) or the original BCW candidates (no reordering) is based on predetermined rules (e.g., implicitly depending on block width, block height, or block area, or explicitly depending on one or more flags of CU/CB, PU/PB, TU/TB, CTU/CTB, tile, slice level, picture level, SPS level, and/or PPS level.) When a reduced and reordered merge candidate list is used, the entropy encoding decoding context used for sending may be different from using the original merge candidate list.

In some embodiments, the BCW weights are divided into several groups. Next, the boundary matching cost of each group is calculated. Then, a promising group is implicitly selected as the group with the highest priority. If more than one BCW weight is included in the promising group, a reduced BCW weight index is sent/parsed to indicate the BCW weights from the promising group. Since the number of BCW weights in each group is less than the number of BCW weights in the original BCW weight set, the reduced BCW weight index may occupy fewer codewords than the original BCW weight index.

In some embodiments, if k groups are used and the number of BCW weights in the original BCW weight set is N, the number of BCW weights in each group is "N/k", and the grouping rules depend on the BCW weight index and/or the BCW weight value. For example, BCW weights with the same value of "BCW weight index %k" belong to the same group. For another example, BCW weights with the same value of "BCW weight index/k" belong to the same group. For another example, BCW weights with similar values are in the same group (e.g., BCW weights with negative values are in one group and/or BCW weights with positive values are in another group). In some embodiments, k (the number of candidates in the group) varies with block width, block height and/or block area.

In some embodiments, when a group contains more than one BCW weight, the cost of the group is the average cost of the costs from all BCW weights in the group. In some embodiments, when a group contains more than one BCW weight, the cost of the group is the average, maximum, or minimum cost of the costs from all BCW weights in the group. In some embodiments, the BCW weights are divided into several subsets, and the selection of the subset is explicitly sent to the decoder. And the selection of the subset is performed by using the boundary matching cost.

In some embodiments, the BCW weights are divided into multiple subsets. In some embodiments, BCW weights with similar values are divided into different subsets so that the BCW weights of the same subset have a larger difference from each other. For example, in some embodiments, weights with negative values can be divided into different groups. For another example, in some embodiments, weights with a weight difference greater than a predetermined threshold are divided into the same group. The predetermined threshold can be fixed or explicitly determined. This may help improve the hit rate of boundary matching.

Ⅴ , linear model as a candidate model

The Cross Component Linear Model (CCLM) or Linear Model (LM) mode is a cross-component prediction mode in which the chrominance components of a block are predicted from the co-located reconstructed luma samples using a linear model. The parameters of the linear model (e.g., scale and offset) are derived from the reconstructed luma and chroma samples adjacent to the block. For example, in VVC, the CCLM mode predicts chroma samples from reconstructed luma samples using inter-channel dependencies. The prediction is performed using a linear model of the following form:

represents the predicted chroma samples in a CU (or the predicted chroma samples of the current CU), and Represents the downsampled reconstructed luma samples of the same CU (or the corresponding reconstructed luma samples of the current CU).

CCLM model parameters (scaling parameter) and (Offset parameters) are derived based on up to four adjacent chrominance samples and their corresponding downsampled luma samples. In LM_A mode (also denoted as LM-T mode or INTRA_T_CCLM), only the upper (also called top) adjacent samples are used to calculate the linear model coefficients. In LM_L mode (also denoted as LM-L mode or INTRA_L_CCLM), only the left samples are used to calculate the linear model coefficients. In LM-LA mode (also denoted as LM-LT mode or INTRA_LT_CCLM), both the left and upper samples are used to calculate the linear model coefficients. The proposed method can be applied to any cross-component mode, not limited to LM mode. For example, the cross-component mode uses information of the first color component (e.g., Y) to predict the second/third color component (e.g., Cb/Cr).

In some embodiments, the proposed candidate mode reduction/reordering scheme is used to reorder LM candidates to improve the signaling of specific syntax, such as cclm_mode_idx. Below is the syntax table of LM in the VVC standard. if( cclm_mode_flag ) cclm_mode_idx ae(v)

The syntax element ccm_mode_idx specifies which of the LM candidate modes (e.g., INTRA_LT_CCLM, INTRA_L_CCLM, and INTRA_T_CCLM) is applied. For example, boundary matching costs are calculated for different LM candidate modes. After reordering the different LM candidate modes according to the boundary matching costs, ccm_mode_idx can be used to select a candidate LM candidate mode based on the reordering. In some embodiments, ccm_mode_idx equal to 0 refers to the LM candidate mode with the smallest cost, and ccm_mode_idx equal to 2 refers to the LM candidate mode with the largest cost. In some embodiments, ccm_mode_idx is implicit and the LM candidate mode with the smallest cost is used.

In some embodiments, only the first k LM candidate modes (with higher priority due to cost) can be candidate LM modes (the original number of LM candidate modes is 3). In some embodiments, k=2 and the LM candidate modes are sent as follows: ccm_mode_idx 0 refers to the candidate LM mode with the smallest cost using codeword 0; ccm_mode_idx 1 refers to the candidate LM mode with the second smallest cost using codeword 1.

In some embodiments, after reordering by cost, when k=2, ccm_mode_idx equal to 0 indicates the LM candidate mode with the largest cost, and ccm_mode_idx equal to 1 indicates the LM candidate mode with the smallest cost. In some embodiments, ccm_mode_idx is implicit and the LM candidate mode with the largest cost is used. In some of these embodiments, ccm_mode_idx 0 refers to the candidate LM mode with the largest cost using codeword 0; ccm_mode_idx 1 refers to the candidate LM mode with the second largest cost using codeword 1.

In addition to applying the reduced/reordered candidate mode list to the traditional CCLM modes where the chrominance components (Cr or Cb) are predicted from the luma components, the reduced/reordered candidate mode list can also be used in other variants of CCLM. Here variants of CCLM mean that when the block indication refers to using one of the cross-component modes of the current block (e.g. CCLM_LT, MMLM_LT, CCLM_L, CCLM_T, MMLM_L, MMLM_T and/or intra-frame prediction modes, not one of the traditional DC, planar and angular modes), some alternative modes can be selected.

The following is an example of using the convolutional cross-component mode (CCCM) as an optional mode. When this optional mode is applied to the current block, the cross-component information of the usage model (including nonlinear terms) is used to generate the chrominance prediction. The optional mode can follow the template selection of CCLM, so the CCCM family includes CCCM_LT CCCM_L and/or CCCM_T.

In some embodiments, the cost of a predetermined candidate can be reduced. A predetermined candidate can refer to a candidate inferred from one or more neighboring blocks and/or popular or common patterns such as CCLM_LT, MMLM_LT, CCCM_LT. In some embodiments, the candidate pattern (only) includes relevant popular patterns and popular patterns. In one example, the relevant popular patterns are CCCM_L and/or CCCM_T, and the popular pattern is CCCM_LT. For example, the relevant popular patterns are CCLM_L and/or CCLM_T, and the popular pattern is CCLM_LT. In another example, the relevant popular patterns are MMLM_L and/or MMLM_T, and the popular pattern is MMLM_LT. In another example, the popular pattern indicates whether the reference used is from L, T or LT. If the reference used is from LT, the relevant prevalence patterns are all variants or a subset of variants of LT, such as CCLM_LT, MMLM_LT, and CCCM_LT.

In some embodiments, the candidate mode corresponds to a linear model derived by using adjacent reconstructed samples of Cb and Cr as input X and Y (i.e., using reconstructed Cb samples to derive or predict Cr samples and vice versa). In some embodiments, the candidate mode corresponds to a linear model derived from multiple co-located luminance blocks for predicting or deriving chrominance components. In some embodiments, the candidate modes in the reduced/reordered candidate list may correspond to a multi-model linear model (MMLM) mode, so that different linear models may be selected from multiple linear models to predict chrominance components of different regions or different pixel groups. Similar to CCLM, MMLM may have MMLM_LT, MMLM_L and/or MMLM_T.

The LM candidate modes in the reduced/reordered candidate mode list may also include any LM extension/variant, so that the number of candidate modes in the list may also increase. As the number of LM candidate modes increases, the coding performance improvement using the reduced/reordered candidate mode list based on the boundary matching cost becomes more significant.

VI. MHP As a candidate model

For blocks to which multi-hypothesis prediction (MHP) is applied, one or more prediction hypotheses (prediction signals) are combined with the existing prediction hypotheses to form the final (result) prediction for the current block. MHP can be applied to many inter-frame modes, such as merging, sub-block merging, inter-frame AMVP, and/or affine. The final prediction can be repeatedly accumulated with each additional hypothesis of the prediction signal. An example of accumulation is shown below:

The predicted signal is taken as the last (Right now, Has the largest index ). P ₀ is the first (existing) prediction for the current block. For example, if MHP is applied to merge candidates, then p ₀ is indicated by the existing merge index. Additional predictions are denoted h, and will be further combined with previously accumulated predictions by weights α. Therefore, for each additional hypothesis for prediction, a weighting index is sent/parsed to indicate the weighting, and/or for each hypothesis for prediction, an interval index is sent/parsed to indicate the motion candidate (used to generate prediction sample hypotheses for this).

A proposed re-ranking scheme is needed to make the sending of MHP weights and/or the sending of motion candidates more efficient. In some embodiments, a margin matching cost for each MHP candidate weight is calculated for each additional prediction hypothesis. For example, assume that the prediction hypothesis has two candidate weights: Step 0: For the predicted additional hypothesis, cost_w0 and cost_w1 are calculated as the costs of the first and second candidate weights, respectively; Step 1: For the prediction hypothesis, the candidate weights are re-ranked according to the cost.

In some sub-embodiments, candidate weights with smaller costs are given higher priority. If cost_w0 > cost_w1, weight index 0 refers to w1 and weight index 1 refers to w0. Otherwise, reordering is not used, and weight indices 0 and 1 refer to the original w0 and w1. In some sub-embodiments, candidate weights with larger costs are given higher priority. If cost_w0 < cost_w1, weight index 0 refers to w1 and weight index 1 refers to w0. Otherwise, reordering is not used, and weight indices 0 and 1 refer to the original w0 and w1.

In some embodiments, the candidate weight with the smallest cost is used for the current additional prediction hypothesis. In this case, the weight index of the current additional prediction hypothesis is inferred. In some embodiments, the weight of each hypothesis or any subset of hypotheses is implicitly set according to the cost. One possible way is that the weight is a scaled value of the cost or the inverse of the scaled value of the cost. (For example, if cost = 2, then the inverse of cost = 1/2.) In some sub-embodiments, the candidate weight with the largest cost is used for the current additional prediction hypothesis. In this case, the weight index of the current additional prediction hypothesis is inferred.

Steps 0 and 1 can be repeated for each additional prediction hypothesis, and the meaning of each weight index for each additional prediction hypothesis is obtained. In some embodiments, assume that there are t candidate weights for the prediction hypothesis (t can be any positive integer. In the following example, t = 5 is taken.) Step 0: For the additional prediction hypothesis, the cost of each candidate weight is calculated separately. Step 1: For the prediction hypothesis, the candidate weights are reordered according to the cost.

For example, a candidate weight with a smaller cost obtains a higher priority. For another example, a candidate weight with a larger cost obtains a higher priority. Only the first k candidate weights (with high priority) can be used as candidate weights. The candidate mode set only includes the first k candidate weights. In some embodiments, the number of candidate weights in the candidate mode set is k (for example, 2) and each candidate weight is sent as follows: weight_idx 0 refers to the candidate with the highest priority and codeword 0; weight_idx 1 refers to the candidate with the second highest priority and codeword 1. Steps 0 and 1 can be repeated for each additional prediction hypothesis, and the meaning of each weight index is obtained for each additional prediction hypothesis.

In some embodiments, the MHP weights are divided into several groups. Next, the boundary matching cost of each group is calculated. The promising group is then implicitly determined as the group with the highest priority. If more than one MHP weight is included in the promising group, a reduced MHP weight index is sent/parsed to indicate the MHP weights from the promising group. Since the number of MHP weights in each group is less than the number of MHP weights in the original MHP weight set, the reduced MHP weight index should occupy fewer codewords than the original MHP weight index.

In some embodiments, if k groups are used and the number of MHP weights in the original MHP weight set is N, the number of MHP weights in each group is "N/k" and the grouping rule depends on the MHP weight index and/or the MHP weight value. For example, MHP weights with the same "MHP weight index % k" value belong to the same group. For another example, MHP weights with the same "MHP weight index/k" belong to the same group. For another example, MHP weights with similar values are in the same group (e.g., MHP weights with negative values are in one group and/or MHP weights with positive values are in another group). For another example, k varies with block width, block height and/or block area. In some embodiments, when a group contains more than one merge candidate, the cost of the group is the average cost from the costs of all merge candidates in the group. In some embodiments, when a group contains more than one merge candidate, the cost of the group is the average, maximum, or minimum cost of all the merge candidates in the group.

In some embodiments, the MHP weights are divided into several subsets, and the subset selection is explicitly sent to the decoder. And the selection in the subset is performed by using the boundary matching cost. In some embodiments, weights with large different values from each other will be divided into the same subset. That is, weights with similar values will be divided into different subsets. This may help to improve the hit rate of boundary matching. For example, weights with negative values are divided into different groups. For another example, those weights whose weight difference is greater than a predetermined threshold are divided into the same group. The predetermined threshold can be fixed or explicitly determined.

In some embodiments, for each hypothesis, the current prediction of the candidate weights (used to compute the margin matching cost) is a combined prediction (a weighted average of the predictions from the motion candidates and the previously accumulated predictions). (The motion candidates are indicated by the motion index sent/resolved.) An example is shown in the figure below. When the proposed method is applied to reorder the sending of candidate weights for h2, h2 has 4 candidate weights (including candidates 0 to 3). The cost of candidate n is calculated as a weighted average of the previously accumulated predictions and the predictions for h2 (the weights are from candidate n).

When re-ranking is applied to the candidate weights for h1, there are 4 candidate weights (including candidates 0 to 3): for h1, the current prediction of candidate 0 = the weighted average of the prediction of p0 and the prediction of h1, with the weight coming from candidate 0; the current prediction of candidate 1 = the weighted average of the prediction of p0 and the prediction of h1, with the weight coming from candidate 1.

In some embodiments, a boundary matching cost is calculated for each MHP motion candidate. The following sub-embodiments take MHP as a merging method as an example. (MHP can be applied to other inter-frame modes, such as inter-frame AMVP and/or affine, and when inter-frame AMVP or affine is used, "merge" in the following examples is replaced with the naming of the inter-frame mode.) In some embodiments, a boundary matching cost is calculated for each motion candidate. For example, the candidate mode includes candidates 0 to 4. Initially, index 0 (shorter codeword) refers to candidate 0, and index 4 (longer codeword) refers to candidate 4. Using the proposed method, the meaning of the index follows the priority order. If the priority order (based on the boundary matching cost) specifies that candidate 4 has the highest priority, index 0 is mapped to candidate 4.

In some embodiments, a boundary matching cost is calculated for each motion candidate. For example, the candidate pattern includes candidates 0 to 4. Initially, index 0 (shorter codeword) refers to candidate 0 and index 4 (longer codeword) refers to candidate 4. Using the proposed method, the meaning of the index follows the priority order. If the priority order (based on the boundary matching cost) specifies that candidate 4 has the highest priority, the index is not sent/parsed, and the selected motion candidate is inferred to be the motion candidate with the highest priority.

In some embodiments, for each hypothesis, the current prediction of a motion candidate (used to compute the boundary matching cost) is the motion compensation result produced by that motion candidate.

An example combining the above two sub-embodiments is shown below. For example, if the number of additional hypotheses is equal to 2, the top three motion candidates with higher priorities are used to form the result prediction of the current MHP block. For another example, if the number of additional hypotheses is equal to 2, the existing hypotheses are retained, and the two motion candidates with higher priorities are used to form the prediction of the additional hypotheses, and the resulting prediction is formed by the existing hypotheses and the additional hypotheses.

In some embodiments, the current prediction for the motion candidate (used to compute the margin matching cost) is a combined prediction (a weighted average of the prediction from the motion candidate and the existing prediction (p0)). (The weights are indicated by the weight index sent/parsed.)

In some embodiments, for each hypothesis, the current prediction of the motion candidate (used to calculate the margin matching cost) is a combined prediction (a weighted average of the prediction from the motion candidate and the previously accumulated prediction). (The weights are indicated by the weight index of the send/parse.) An example is shown in the figure below. When the proposed method is applied to reorder the sending of motion candidates for h2, h2 has 4 motion candidates (including candidates 0 to 3). The cost of candidate n is calculated as the weighted average of the prediction of p0, the prediction of h1, and the prediction of candidate n. In other words, for h2: the current prediction of candidate 0 = the weighted average of the prediction of h1 and the prediction from candidate 0; the current prediction of candidate 1 = the weighted average of the prediction of h1 and the prediction from candidate 1.

When the proposed method is applied to reorder the sending of motion candidates for h1, h1 has 4 motion candidates (including candidates 0 to 3): the current prediction for candidate 0 = the weighted average of the prediction for p0 and the prediction from candidate 0; the current prediction for candidate 1 = the weighted average of the prediction for p0 and the prediction from candidate 1.

In some embodiments, a boundary matching cost is calculated for each MHP combination (motion candidate and weight) for each prediction hypothesis. The following sub-embodiments take MHP as an example of a merge mode. (MHP can be applied to other inter-frame modes, such as inter-frame AMVP and/or affine, and when inter-frame AMVP or affine is used, "merge" in the following examples will be replaced by the name of the inter-frame mode.)

In some embodiments, a combination refers to a motion candidate and a weight. If there are m motion candidates and each motion candidate has n weights, the number of combinations is m*n. In some sub-embodiments, the current prediction of the combination (used to calculate the boundary matching cost) is the combined prediction (the weighted average of the predictions from the motion candidates and the existing prediction (p0)). Using this method, the merge index and the weights representing the additional hypotheses for the prediction are jointly determined. For example, the combination with the highest priority is the selected MHP combination. (There is no need to send/parse the merge index and weights to indicate the additional hypotheses for the prediction). For another example, the joint index is sent/parsed to determine the MHP combination. This embodiment can fix the number of additional hypotheses.

In some embodiments, the boundary matching cost of each MHP motion candidate is calculated. Take the MHP in merge mode as an example. The merge index (used to indicate the motion candidate for each hypothesis) can be inferred. For example, based on the order of the merge candidates in the merge candidate list, hypothesis 0 is merge candidate 0, hypothesis 1 is merge candidate 1, and so on. For another example, based on the cost, the merge candidate with a smaller cost is used first. For another example, it depends on the predetermined number of merge candidates. If the number of hypotheses is 4, the 4 merge candidates in the merge candidate list are used as motion candidates for each hypothesis. The first 4 merge candidates are used, and any 4 merge candidates from the merge candidate list are used. The weights (used to combine forecast hypotheses) are implicit and depend on the cost. The MHP result forecast is formed as follows. (weight0)*(hypothesis0) + (weight1)*(hypothesis1) + (weight2)*(hypothesis2)+…

In some embodiments, a fixed number of prediction hypotheses are used. That is, a fixed number of hypotheses are mixed and matching costs are implicitly used as weights. In some embodiments, motion candidates (or hypotheses) with higher priorities are weighted more heavily than motion candidates with lower priorities. In some embodiments, the current prediction of a motion candidate (used to calculate the boundary matching cost) is the motion compensation result generated by the motion candidate. In some embodiments, the first n motion candidates in the merged candidate list are used to generate prediction hypotheses. Using this proposed method, no merge index is sent to the MHP. In some embodiments, all motion candidates in the merged candidate list are used to generate prediction hypotheses. (The weight can determine whether a motion candidate is used at all. If its weight is zero, this motion candidate is not actually used.) Using this proposed method, no merge index is sent to the MHP. In some embodiments, the weight of a motion candidate with a higher priority is greater than the weight of a motion candidate with a lower priority. For example, the weight follows the cost ratio of different motion candidates. If there are two motion candidates and cost_cand0 = 2* cost_cand1, weight_cand0 = 2*weight_cand1 or weight_cand0 = 1/2*weight_cand1.

For another example, the cost of each motion candidate is first normalized to the interval [MIN_VALUE, MAX_VALUE]. MAX_VALUE is predetermined, such as the number of prediction hypotheses. MIN_VALUE is predetermined, such as 0. For example, (MAX_VALUE-normalized cost) can be the weight of the motion candidate or the normalized cost can be the weight of the motion candidate.

Another example is that the weight is a scaled value of the cost or a scaled value of the inverse of the cost. (For example, if cost = 2, then the inverse of cost = ½.) The scaled value represents the scale factor * the original value. If the scale factor = 1, then scale.

In some embodiments, the weights and merge indexes are implicit to the proposed method. The generation of the method's predictions can refer to any other method proposed in the present invention. In some embodiments, the proposed scheme is applied to a subset of all additional prediction hypotheses. (That is, the above steps 0 and 1 are repeated for all subsets of additional prediction hypotheses.) For example, only the candidate weights of the first additional prediction hypothesis (combined with the existing prediction hypothesis) are reordered using the proposed scheme. In some embodiments, the subset is predetermined in the video coding standard. In an embodiment, the subset depends on the current block width, height, or area. For example, for blocks with a block area greater than (or less than) a threshold, the subset includes more prediction hypotheses.

In some embodiments, the reordered results from the subset can be reused for the remaining additional prediction hypotheses. For example, based on the reordered results of the first prediction hypothesis, weight indices 0 and 1 refer to w1 and w0, respectively. For the following prediction hypotheses, weight indices 0 and 1 also refer to w1 and w0, respectively.

In some embodiments, the proposed scheme is applied to a subset of all candidate weights for additional prediction hypotheses. That is, the above steps 0 and 1 are repeated for a subset of all candidate weights for additional prediction hypotheses. Take the number of candidate weights (for additional prediction hypotheses) equal to 4 as an example. For additional prediction hypotheses, only the first (or last) two candidate weights are reordered using the proposed scheme. In some embodiments, the subset is predetermined in the video coding standard. In some embodiments, the subset depends on the current block width, height or area. For example, for blocks with a block area greater than (or less than) a threshold, the subset includes more candidate weights.

In some embodiments, the prediction hypothesis can be a prediction signal from a one-way prediction or a two-way prediction motion compensation result. The reordering schemes of different proposed tools (not limited to those in the following examples) can be unified. For example, the proposed reordering schemes for MHP, LM, BCW, MMVD and/or merge candidates are unified with the same rules for calculating the boundary matching cost.

The methods proposed in the present invention can be enabled and/or disabled according to implicit rules (e.g., block width, height or area) or according to explicit rules (e.g., based on blocks, tiles, fragments, pictures, SPS or PPS levels). For example, when the block area is smaller than a threshold, the proposed reordering is applied. Any combination of the methods proposed in the present invention can be applied.

Any of the aforementioned proposed methods can be implemented in an encoder and/or a decoder. For example, any of the proposed methods can be implemented in an intra-frame/inter-frame encoding and decoding module of an encoder, a motion compensation module, or a merge candidate derivation module of a decoder. Alternatively, any of the proposed methods can be implemented as a circuit coupled to an intra-frame/inter-frame encoding module of an encoder and/or a motion compensation module or a merge candidate derivation module of a decoder.

VII , Sample Video Encoder

FIG6 illustrates an example video encoder 600 that may implement candidate codec mode selection based on boundary matching cost. As shown, the video encoder 600 receives an input video signal from a video source 605 and encodes the signal into a bit stream 695. The video encoder 600 has several components or modules for encoding a signal from a video source 605, including at least some components selected from the following: a transform module 610, a quantization module 611, an inverse quantization module 614, an inverse transform module 615, an intra-frame estimation module 620, an intra-frame prediction module 625, a motion compensation module 630, a motion estimation module 635, a loop filter 645, a reconstructed picture buffer 650, an MV buffer 665, an MV prediction module 675, and an entropy encoder 690. The motion compensation module 630 and the motion estimation module 635 are part of the inter-frame prediction module 640.

In some embodiments, modules 610-690 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 610-690 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 610-690 are shown as separate modules, some modules may be combined into a single module.

The video source 605 provides an uncompressed raw video signal that presents pixel data for each video frame. The subtractor 608 calculates the difference between the raw video pixel data of the video source 605 and the predicted pixel data 613 from the motion compensation module 630 or the intra-frame prediction module 625. The transform module 610 converts the difference (or residual pixel data or residual signal 608) into a transform coefficient (e.g., by performing a discrete cosine transform, or DCT). The quantization module 611 quantizes the transform coefficient into quantized data (or quantized coefficient) 612, which is encoded into a bit stream 695 by the entropy encoder 690.

The inverse quantization module 614 inversely quantizes the quantized data (or quantized coefficients) 612 to obtain transformation coefficients, and the inverse transformation module 615 inversely transforms the transformation coefficients to generate reconstruction residues 619. The reconstruction residues 619 are added to the predicted pixel data 613 to generate reconstructed pixel data 617. In some embodiments, the reconstructed pixel data 617 is temporarily stored in a row buffer (not shown) for intra-picture prediction and spatial MV prediction. The reconstructed pixels are filtered by a loop filter 645 and stored in a reconstructed picture buffer 650. In some embodiments, the reconstructed picture buffer 650 is a memory outside the video encoder 600. In some embodiments, the reconstructed picture buffer 650 is a memory inside the video encoder 600.

The intra-frame estimation module 620 performs intra-frame prediction based on the reconstructed pixel data 617 to generate intra-frame prediction data. The intra-frame prediction data is provided to the entropy encoder 690 to be encoded into a bit stream 695. The intra-frame prediction data is also used by the intra-frame prediction module 625 to generate the predicted pixel data 613.

The motion estimation module 635 performs inter-frame prediction by generating MVs to refer to pixel data of previously decoded frames stored in the reconstructed picture buffer 650. These MVs are provided to the motion compensation module 630 to generate predicted pixel data.

Instead of encoding the complete actual MV in the bitstream, the video encoder 600 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 695.

Based on the reference MV generated for encoding the previous video frame, i.e., the motion compensation MV for performing motion compensation, the MV prediction module 675 generates a predicted MV. The MV prediction module 675 obtains the reference MV from the previous video frame from the MV buffer 665. The video encoder 600 stores the MV generated for the current video frame in the MV buffer 665 as a reference MV for generating the predicted MV.

The MV prediction module 675 uses the reference MV to create a predicted MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The entropy encoder 690 encodes the difference (residual motion data) between the predicted MV and the motion compensation MV (MC MV) of the current frame into the bit stream 695.

The entropy encoder 690 encodes various parameters and data into a bit stream 695 by using entropy coding and decoding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding. The entropy encoder 690 encodes various header elements, flags, along with quantized transform coefficients 612 and residual motion data as syntax elements into the bit stream 695. The bit stream 695 is in turn stored in a storage device or transmitted to a decoder via a communication medium (e.g., a network).

The in-loop filter 645 performs a filtering or smoothing operation on the reconstructed pixel data 617 to reduce encoding and decoding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

FIG7 illustrates a portion of the video encoder 600 that implements candidate codec mode selection based on boundary matching costs. Specifically, the figure illustrates components of the inter-frame prediction module 640 of the video encoder 600 that calculates boundary matching costs for all or any subset of different candidate codec modes and selects a candidate codec mode based on the group assignment and the calculated costs.

The inter-frame prediction module 640 includes various boundary prediction modules 710. These boundary prediction modules 710 generate prediction samples 715 of the current block along the boundaries of the current block for various candidate codec modes. The boundary matching cost calculator 730 compares the boundary prediction sub-samples 715 of each candidate codec mode with the neighboring samples 725 of the current block (obtained from the reconstructed picture buffer 650) to calculate the boundary matching cost of each candidate codec mode or any subset of the candidate codec modes. Part I above describes the calculation of the boundary matching cost based on the neighboring samples and the boundary prediction samples.

The group allocation and selection module 740 allocates different candidate codec modes to different groups and selects one of the groups. The candidate selection module 750 then selects a candidate codec mode from the selected group. The inter-frame prediction module 640 then uses the selected candidate codec mode to perform motion compensation. In some embodiments, the group allocation module 740 allocates a certain number of candidate codec modes with the lowest boundary matching cost to form a minimum cost group, and the candidate selection module 750 selects the candidate codec mode from the minimum cost group. In some embodiments, the group allocation module 740 allocates a certain number of candidate codec modes to form groups with predetermined rules (which may or may not depend on cost), and the candidate selection module 750 selects a candidate codec mode from the group based on cost. For example, the selected candidate codec mode is the candidate mode with the highest priority in the group.

In some embodiments, the identity of the selected group is sent to the entropy encoder 690 to be sent in the bitstream 695. In some embodiments, the identity of the selected group is implicitly determined and not sent in the bitstream. In some embodiments, the selection of the group is determined based on the calculated boundary matching costs of different groups, for example, the group selection module 740 can select the group with the lowest representative cost.

In some embodiments, the identity of the selected candidate codec mode within the selected group is provided to the entropy encoder 690 for transmission in the bitstream 695. In some embodiments, based on the calculated boundary matching costs, the candidate codec modes within the group are reordered so that the lowest cost (or highest cost) candidate will be transmitted using the shortest codeword. In some embodiments, the identity of the selected candidate codec mode will be determined implicitly and not transmitted in the bitstream, for example, by selecting the candidate codec mode within the group with the lowest boundary matching cost. The selection of a set of candidate codec modes and the selection of a candidate codec mode from the set are described in Sections II-VI above.

The boundary prediction module 710 includes modules for various different candidate codec modes. These candidate codec modes can use the luminance and chrominance samples stored in the reconstruction buffer 650, the motion information from the MV buffer 665 and/or the incoming luminance and chrominance samples from the video source 605 to generate boundary prediction samples 715 (e.g., by deriving merge candidates, deriving MMVD candidates, using motion information to obtain reference samples, generating linear models, etc.) In some embodiments, the various candidate codec modes include those candidate codec modes corresponding to the various merge candidates of the current block. Various types of merge candidates (e.g., spatial, temporal, affine, CIIP, etc.) as candidate codec modes are described in Part II above. Different MMVD distances and directions as candidate codec modes are described in Section III above. Different BCW weights as candidate codec modes are described in Section IV above. Various types of linear models (e.g., LM-L, LM-T, LM-LT) as candidate codec modes are described in Section V above.

FIG. 8 conceptually illustrates a process 800 for selecting a candidate encoding/decoding mode to encode a current block using a boundary matching cost. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the encoder 600 perform the process 800 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the encoder 600 performs the process 800.

The encoder receives (at block 810) data for a block of pixels for a current block in a current picture of a video to be encoded. The encoder identifies (at block 820) a plurality of candidate codec modes applicable to the current block. In some embodiments, the plurality of candidate codec modes include merge candidates for the current block. The merge candidates for the current block may include (i) merge candidates using CIIP and/or (ii) merge candidates using affine transform motion compensation prediction. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different MMVD combinations of distance and offset for refining motion information. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different linear models for deriving a predictor of chrominance samples of the current block based on luma samples of the current block. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different candidate BCW weights for combining inter-frame predictions of different directions.

The encoder identifies (at block 830) a first candidate codec mode set as a subset of the plurality of candidate codec modes such that a number of codec modes in the first candidate codec mode set is less than a number of candidate codec modes in the plurality of candidate codec modes.

In some embodiments, the first candidate codec mode group identifies the highest priority candidate codec mode based on the cost of multiple candidate codec modes. The encoder can index the candidate codec modes according to the priority of the candidate codec modes or assign codewords to the candidate codec modes in the identified candidate codec mode group. In some embodiments, the cost of the candidate codec mode is a boundary matching cost calculated by comparing (i) reconstructed samples adjacent to the current block with (ii) predicted samples of the current block along the boundary of the current block generated according to the candidate codec mode. Section I above describes calculating the boundary matching cost based on adjacent samples and boundary prediction samples.

In some embodiments, each of a plurality of candidate codec modes is assigned to one of a plurality of candidate codec mode groups. For example, in some embodiments, each of a plurality of candidate codec modes is associated with an original candidate index, wherein each candidate codec mode is assigned to one of K groups of candidate codec modes based on the result of the original index modulo K or the result of the original index divided by K. In some embodiments, candidate codec modes corresponding to spatial merge candidates are assigned to the same candidate codec mode group. In some embodiments, candidate codec modes corresponding to spatial merge candidates are assigned to different candidate codec mode groups. In some embodiments, candidate codec modes for merge candidates having a motion difference less than a threshold are assigned to the same group. In some embodiments, candidate codec modes of merged candidates having motion differences greater than a threshold are assigned to the same group. In some embodiments, the encoder identifies a candidate codec mode group having the lowest representative cost among multiple groups of candidate codec modes, and sends an index used to select a candidate codec mode from the identified candidate codec mode group. The representative cost of the identified group can be an average, maximum, or minimum value of the costs (e.g., boundary matching costs) of the candidate codec modes of the identified group. In some embodiments, the encoder sends an index for selecting a candidate codec mode group, and identifies a candidate codec mode from the selected candidate codec mode group based on the cost (e.g., boundary matching cost) of the selected candidate codec mode group. The selection of a set of candidate codec modes and the selection of a candidate codec mode from the set are described in Sections II-VI above.

The encoder selects (at block 840) a candidate codec mode in the first set of candidate codec modes. The candidate codec mode selected in the first set of candidate codec modes may be selected based on cost.

The encoder encodes the current block using the selected candidate codec mode (at block 850). Specifically, the encoder constructs a predictor of the current block according to the selected candidate codec mode, and encodes the current block using the predictor.

VIII . Sample Video Decoder

In some embodiments, an encoder may send (or generate) one or more syntax elements in a bitstream so that a decoder may parse the one or more syntax elements from the bitstream.

FIG. 9 shows an example video decoder 900 that can implement the candidate decoding mode selection based on the boundary matching cost. As shown in the figure, the video decoder 900 is an image decoding or video decoding circuit that receives a bit stream 995 and decodes the content of the bit stream into pixel data of a video frame for display. The video decoder 900 has several components or modules for decoding the bit stream 995, including some components selected from the following: an inverse quantization module 911, an inverse transform module 910, an intra-frame prediction module 925, a motion compensation module 930, an intra-loop filter 945, a decoded picture buffer 950, an MV buffer 965, an MV prediction module 975 and a parser 990. The motion compensation module 930 is part of the frame prediction module 940.

In some embodiments, modules 910-990 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 910-990 are hardware circuit modules implemented by one or more ICs of an electronic device. Although modules 910-990 are shown as separate modules, some modules may be combined into a single module.

The parser 990 (or entropy decoder) receives the bitstream 995 and performs initial parsing according to the syntax defined by the video codec or image codec standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 912. The parser 990 uses, for example, context-adaptive binary arithmetic coding (CABAC) or Huffman coding.

The inverse quantization module 911 inversely quantizes the quantized data (or quantized coefficients) 912 to obtain the transformed coefficients, and the inverse transform module 910 inversely transforms the transformed coefficients 916 to obtain the reconstructed residual signal 919. The reconstructed residual signal 919 is added to the predicted pixel data 913 from the intra-frame prediction module 925 or the motion compensation module 930 to generate the decoded pixel data 917 together. The decoded pixel data is filtered by the in-loop filter 945 and stored in the decoded picture buffer 950. In some embodiments, the decoded picture buffer 950 is a memory outside the video decoder 900. In some embodiments, the decoded picture buffer 950 is a memory inside the video decoder 900.

The intra prediction module 925 receives intra prediction data from the bitstream 995 and generates prediction pixel data 913 from the decoded pixel data 917 stored in the decoded picture buffer 950. In some embodiments, the decoded pixel data 917 is also stored in a row buffer (not shown) for intra prediction and spatial MV prediction.

In some embodiments, the contents of the decoded picture buffer 950 are used for display. The display device 955 obtains the contents of the decoded picture buffer 950 for direct display, or obtains the contents of the decoded picture buffer to the display buffer. In some embodiments, the display device receives pixel values from the decoded picture buffer 950 by pixel transmission.

The motion compensation module 930 generates predicted pixel data 913 from the decoded pixel data 917 stored in the decoded picture buffer 950 according to the motion compensation MV (MC MV). These motion compensation MVs are decoded by adding the residual motion data received from the bitstream 995 to the predicted MV received from the MV prediction module 975.

The MV prediction module 975 generates a predicted MV based on a reference MV generated for decoding a previous video frame, for example, a motion compensation MV for performing motion compensation. The MV prediction module 975 obtains the reference MV of the previous video frame from the MV buffer 965. The video decoder 900 stores the motion compensation MV generated for decoding the current video frame in the MV buffer 965 as a reference MV for generating the predicted MV.

The in-loop filter 945 performs a filtering or smoothing operation on the decoded pixel data 917 to reduce coding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

FIG. 10 illustrates a portion of the video decoder 900 that implements candidate codec mode selection based on boundary matching costs. Specifically, the figure illustrates components of the inter-frame prediction module 940 of the video decoder 900 that calculates boundary matching costs for all or any subset of different candidate codec modes and selects a candidate codec mode based on the group assignment and the calculated costs.

The inter-frame prediction module 940 includes various boundary prediction modules 1010. These boundary prediction modules 1010 generate prediction samples 1015 of the current block along the boundaries of the current block for various candidate codec modes. The boundary matching cost calculator 1030 compares the boundary predictor samples 1015 of each candidate codec mode with neighboring samples 1025 of the current block (obtained from the reconstructed picture buffer 950) to calculate the boundary matching cost for each candidate codec mode or any subset of the candidate codec modes. Section I above describes the calculation of the boundary matching cost based on neighboring samples and boundary prediction samples.

The group allocation and selection module 1040 allocates different candidate codec modes to different groups and selects one of the groups. Then, the candidate selection module 1050 selects a candidate codec mode from the selected group. Then, the frame prediction module 940 uses the selected candidate codec mode to perform motion compensation. In some embodiments, the group allocation module 1040 allocates a certain number of candidate codec modes with the lowest boundary matching cost to form a minimum cost group, and the candidate selection module 1050 selects the candidate codec mode from the minimum cost group. In some embodiments, the group allocation module 1040 allocates a certain number of candidate codec modes to form a group with predetermined rules (which may or may not depend on the cost), and the candidate selection module 1050 selects the candidate codec mode from the group according to the cost. For example, the candidate codec mode selected is the candidate mode with the highest priority in the group.

In some embodiments, the entropy decoder 990 parses the bitstream 995 to obtain the identification of the selected group, and provides the identification of the selected group to the group allocation and selection module 1040. In some embodiments, the identification of the selected group is implicitly determined and is not sent in the bitstream. In some embodiments, the selection of the group is determined based on the calculated boundary matching costs of different groups, for example, the group selection module 1040 can select the group with the lowest representative cost. In some embodiments, the entropy decoder 990 parses the bitstream 995 to obtain the identification of the selected candidate codec mode within the selected group, and provides the identification of the selected candidate codec mode to the candidate selection module 1050.

In some embodiments, the candidate codec modes within a group are reordered based on the calculated boundary matching costs so that the lowest cost (or highest cost) candidate will be sent using the shortest codeword. In some embodiments, the identity of the selected candidate codec mode will be determined implicitly and not sent in the bitstream, for example, by selecting the candidate codec mode within the group with the lowest boundary matching cost. The selection of a set of candidate codec modes and the selection of a candidate codec mode from the set are described in Sections II-VI above.

Boundary prediction module 1010 includes modules for various different candidate codec modes. These candidate codec modes can use the luminance and chrominance samples stored in reconstruction buffer 950 and the motion information from MV buffer 965 to generate boundary prediction samples 1015 (e.g., by deriving merge candidates, deriving MMVD candidates, obtaining reference samples using motion information, generating linear models, etc.) In some embodiments, candidate codec modes include those candidate codec modes corresponding to various merge candidates of the current block. Various types of merge candidates (e.g., space, time, affine, CIIP, etc.) as candidate codec modes are described in Part II above. Different MMVD distances and directions as candidate codec modes are described in Part III above. Different BCW weights as candidate codec modes are described in Section IV above. Various types of linear models (e.g., LM-L, LM-T, LM-LT) as candidate codec modes are described in Section V above.

FIG. 11 conceptually illustrates a process 1100 of using a boundary matching cost to select a candidate encoding/decoding mode to decode the current block. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 900 perform the process 1100 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the decoder 900 performs the process 1100.

The decoder receives (at block 1110) data for a block of pixels to be decoded as a current block in a current picture of a video. The decoder identifies (at block 1120) a plurality of candidate codec modes applicable to the current block. In some embodiments, the plurality of candidate codec modes include merge candidates for the current block. The merge candidates for the current block may include (i) merge candidates using combined inter- and intra-frame prediction (CIIP) and/or (ii) merge candidates using affine transform motion compensation prediction. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different MMVD combinations of distance and offset for refining motion information. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different linear models (LMs) for deriving a predictor of chrominance samples of the current block based on luma samples of the current block. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different candidate BCW weights for combining inter-frame predictions of different directions.

The decoder identifies (at block 1130) a first candidate codec mode set that is a subset of the plurality of candidate codec modes. The number of candidate codec modes in the first candidate codec mode set is less than the number of candidate codec modes in the plurality of candidate codec modes. (That is, the first candidate codec mode set is a subset of the plurality of candidate codec modes.)

In some embodiments, the first candidate codec mode group identifies the highest priority candidate codec mode based on the cost of multiple candidate codec modes. The decoder can index the candidate codec modes in the identified candidate codec mode group or assign codewords to the candidate codec modes according to the priority of the candidate codec modes. In some embodiments, the cost of the candidate codec mode is a boundary matching cost calculated by comparing (i) reconstructed samples adjacent to the current block with (ii) predicted samples of the current block along the boundary of the current block generated according to the candidate codec mode. Section I above describes the calculation of the boundary matching cost based on adjacent samples and boundary prediction samples.

In some embodiments, each of a plurality of candidate codec modes is assigned to one of a plurality of candidate codec mode groups. For example, in some embodiments, each of a plurality of candidate codec modes is associated with an original candidate index, wherein each candidate codec mode is assigned to one of K groups of candidate codec modes based on the result of the original index modulo K or the result of the original index divided by K. In some embodiments, candidate codec modes corresponding to spatial merge candidates are assigned to the same candidate codec mode group. In some embodiments, candidate codec modes corresponding to spatial merge candidates are assigned to different candidate codec mode groups. In some embodiments, candidate codec modes for merge candidates having a motion difference less than a threshold are assigned to the same group. In some embodiments, candidate codec modes of merged candidates having motion differences greater than a threshold are assigned to the same group. In some embodiments, a decoder identifies a candidate codec mode group having a lowest representative cost among multiple groups of candidate codec modes, and sends an index for selecting a candidate codec mode from the identified candidate codec mode group. The representative cost of the identified group may be an average, maximum, or minimum value of the costs (e.g., boundary matching costs) of the candidate codec modes of the identified group. In some embodiments, a decoder sends an index for selecting a candidate codec mode group, and identifies a candidate codec mode from the selected candidate codec mode group based on the cost (e.g., boundary matching cost) of the selected candidate codec mode group. The selection of a set of candidate codec modes and the selection of a candidate codec mode from the set are described in Sections II-VI above.

The decoder selects (at block 1140) a candidate codec mode in the first set of candidate codec modes. The candidate codec mode selected in the first set of candidate codec modes may be selected based on cost.

The decoder decodes (at block 1150) the current block using the selected candidate codec mode to reconstruct the current block. Specifically, the decoder constructs a predictor of the current block according to the selected candidate codec mode, and uses the predictor to reconstruct the current block. The decoder can then provide the reconstructed current block for display as part of a reconstructed current picture.

VIII . Example electronic system

Many of the above features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, compact disc read-only memory (CD-ROM), flash memory drives, random-access memory (RAM) chips, hard disk drives, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. Computer-readable media does not include carrier waves and electronic signals transmitted over wireless or wired connections.

In this specification, the term "software" is intended to include firmware residing in read-only memory or applications stored in magnetic memory that can be read into memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while retaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement the software inventions described herein are within the scope of this disclosure. In some embodiments, the software program, when installed to run on one or more electronic systems, defines one or more specific machine implementations that process and execute the operations of the software program.

FIG. 12 conceptually illustrates an electronic system 1200 for implementing some embodiments of the present disclosure. The electronic system 1200 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1200 includes a bus 1205, a processing unit 1210, a graphics processing unit (GPU) 1215, a system memory 1220, a network 1225, a read-only memory 1230, a permanent storage device 1235, an input device 1240, and an output device 1245.

Buses 1205 collectively represent all system, peripheral, and chipset buses that communicatively couple the numerous internal devices of electronic system 1200. For example, bus 1205 communicatively couples processing unit 1210 with GPU 1215, read-only memory 1230, system memory 1220, and permanent storage device 1235.

The processing unit 1210 obtains instructions to be executed and data to be processed from these various memory units in order to perform the processing disclosed herein. In different embodiments, the processing unit can be a single processor or a multi-core processor. Some instructions are passed to and executed by the GPU 1215. The GPU 1215 can offload various calculations or supplement the image processing provided by the processing unit 1210.

Read-only-memory (ROM) 1230 stores static data and instructions used by processing unit 1210 and other modules of the electronic system. On the other hand, permanent storage device 1235 is a read-write storage device. This device is a non-volatile storage unit that stores instructions and data even when the electronic system 1200 is turned off. Some embodiments of the present disclosure use a large-capacity memory device (such as a disk or optical disk and its corresponding disk drive) as permanent storage device 1235.

Other embodiments use unloadable storage devices (e.g., floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like permanent storage device 1235, system memory 1220 is a read-write memory device. However, unlike permanent storage device 1235, system memory 1220 is a volatile read-write memory, such as random access memory. System memory 1220 stores some instructions and data used by the processor when running. In some embodiments, processing according to the present disclosure is stored in system memory 1220, permanent storage device 1235 and/or read-only memory 1230. For example, according to some embodiments of the present disclosure, various memory units include instructions for processing multimedia clips. From these various memory units, processing unit 1210 obtains instructions to be executed and data to be processed in order to perform the processing of some embodiments.

Bus 1205 is also connected to input devices 1240 and output devices 1245. Input devices 1240 enable a user to communicate information and select commands to an electronic system. Input devices 1240 include alphanumeric keyboards and pointing devices (also known as "cursor control devices"), cameras (e.g., webcams), microphones or similar devices for receiving voice commands, etc. Output devices 1245 display images or output data generated by the electronic system. Output devices 1245 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD), as well as speakers or similar audio output devices. Some embodiments include devices that function as both input and output devices, such as a touch screen.

Finally, as shown in FIG. 12 , bus 1205 also couples electronic system 1200 to a network 1225 via a network interface card (not shown). In this manner, the computer can be part of a computer network, such as a local area network (“LAN”), a wide area network (“WAN”), or an intranet, or a network of multiple networks, such as the Internet. Any or all of the components of electronic system 1200 may be used in conjunction with the present disclosure.

Some embodiments include electronic components, such as microprocessors, storage devices, and memory, which store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, double-layer DVD-ROM), various recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD Card, Mini SD Card, Micro SD Card, etc.), magnetic and/or solid state hard disk drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer codes include machine codes such as those generated by a compiler, and documents including high-level codes that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

Although the above discussion primarily involves microprocessors or multi-core processors executing software, many of the above features and applications are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

As used in this specification and any claims hereof, the terms "computer," "server," "processor," and "memory" refer to electronic or other technological devices. These terms do not include people or groups of people. For the purposes of this specification, the terms display or display refer to displaying on an electronic device. As used in this specification and any claims hereof, the terms "computer-readable medium," "computer-readable medium," and "machine-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient signals.

Although the present disclosure has been described with reference to many specific details, a person of ordinary skill in the art will recognize that the present disclosure may be implemented in other specific forms without departing from the spirit of the present disclosure. In addition, several figures (including Figures 8 and 11) conceptually illustrate the processing. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the processing may be implemented using multiple sub-processes or as part of a larger macro-process. Therefore, a person of ordinary skill in the art will understand that the present disclosure is not limited by the foregoing illustrative details, but is limited by the scope of the attached patent application. Supplementary Notes

The subject matter described herein sometimes represents different components, which are contained in or connected to other different components. It is understood that the described structure is only an example, and can actually be implemented by many other structures to achieve the same function. Conceptually, any arrangement of components that achieve the same function is actually "associated" in order to achieve the desired function. Therefore, regardless of the structure or intermediate components, any two components combined to achieve a specific function are considered to be "interrelated" to achieve the desired function. Similarly, any two associated components are considered to be "operably connected" or "operably coupled" to each other to achieve a specific function. Any two components that can be associated with each other are also considered to be "operably coupled" to each other to achieve a specific function. Any two components that can be associated with each other are also considered to be "operably coupled" to each other to achieve a specific function. Specific examples of operable connections include, but are not limited to, physically mateable and/or physically interacting components, and/or wirelessly interactable and/or wirelessly interacting components, and/or logically interacting and/or logically interactable components.

In addition, with respect to the use of substantially any plural and/or singular terms, those of ordinary skill in the art can convert from the plural to the singular and/or from the singular to the plural according to context and/or application. For clarity, the present invention expressly describes different singular/plural arrangements.

In addition, it is understood by those of ordinary skill in the art that, in general, the terms used in the present invention, especially in the claims, such as the subject matter of the claims, are generally used as "open" terms, for example, "including" should be interpreted as "including but not limited to", "having" should be interpreted as "at least having", "including" should be interpreted as "including but not limited to", etc. It is further understood by those of ordinary skill in the art that if a specific number of claims are intended to be introduced, it will be clearly indicated in the claims, and, if there is no such content, it will not be displayed. For example, to help understanding, the claims below may contain the phrases "at least one" and "one or more" to introduce the claims. However, the use of these phrases should not be construed as implying that the use of the indefinite article "a" or "an" to introduce claim content is limiting to any particular claim. Even when the same claim includes the introductory phrases "one or more" or "at least one," the indefinite article, such as "an" or "an," should be interpreted to mean at least one or more, as is the case with the use of the explicit description to introduce the claim. Furthermore, even when an introductory phrase explicitly refers to a specific number, one of ordinary skill in the art would recognize that such a phrase should be interpreted to mean the number being referred to, e.g., "two references" without other modifications means at least two references, or two or more references. In addition, when using expressions similar to "at least one of A, B, and C", it is usually expressed in such a way that a person of ordinary skill in the art can understand the expression, for example, "a system includes at least one of A, B, and C" will include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and/or a system having A, B, and C, etc. A person of ordinary skill in the art will further understand that any separated words and/or phrases represented by two or more alternative terms, whether in the specification, the scope of the patent application, or the drawings, should be understood to include the possibility of one of these terms, one of them, or both of these terms. For example, "A or B" should be understood as the possibility of "A", or "B", or "A and B".

As can be seen from the foregoing, various embodiments have been described for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the invention. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the true scope and application are indicated by the scope of the patent application.

510: Merge candidate 522: MMVD candidate 524: MMVD candidate 600: Video encoder 605: Video source 608: Subtractor 610: Transform module 611: Quantization module 612: Quantization transform coefficient 613: Pixel data 614: Inverse quantization module 615: Inverse transform module 616: Transform coefficient 617: Reconstructed pixel data 619: Reconstruction residual 620: Intra-frame estimation module 625: Intra-frame prediction module 630: Motion compensation module 635: Motion estimation module 640: Inter-frame prediction module 645: Loop filter 650: reconstructed image buffer 665: MV buffer 675: MV prediction module 690: entropy encoder 695: bit stream 710: boundary prediction module 715: boundary prediction sample 725: neighboring sample 730: boundary matching cost calculator 740: group selection module 750: candidate selection module 800: processing 810: step 820: step 830: step 840: step 850: step 900: video decoder 910: inverse transform module 911: inverse quantization module 912: quantized data 913: Prediction pixel data 916: Transformation coefficient 917: Decoded pixel data 919: Reconstructed residual signal 925: Intra-frame prediction module 930: Motion compensation module 940: Inter-frame prediction module 945: Loop filter 950: Decoded picture buffer 955: Display device 965: MV buffer 975: MV prediction module 990: Parser 895: Bit stream 1010: Boundary prediction module 1015: Boundary prediction sample 1020: Neighbor acquisition 1025: Neighbor sample 1030: Boundary matching cost calculator 1040: Group selection module 1050: Candidate selection module 1100: Processing 1110: Step 1120: Step 1130: Step 1140: Step 1150: Step 1200: Electronic system 1205: Bus 1210: Processing unit 1215: GPU 1220: System memory 1225: Network 1230: Read-only memory 1235: Permanent storage device 1240: Input device

The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated into and constitute a part of the present disclosure. The accompanying drawings illustrate the implementation of the present disclosure and are used together with the description to explain the principles of the present disclosure. It is worth noting that the accompanying drawings are not necessarily drawn to scale, because in order to clearly illustrate the concepts of the present disclosure, some components may be shown as being out of proportion to the size in the actual implementation. Figure 1 shows the reconstructed neighboring samples and predicted samples of the current block for boundary matching. Figure 2 shows the location of the spatial merge candidate. Figure 3 shows the motion vector scaling of the temporal merge candidate. Figure 4 shows the candidate location of the temporal merge candidate of the current block. FIG. 5 conceptually illustrates Merge Mode with Motion Vector Difference (MMVD) candidates and their corresponding offsets. FIG. 6 illustrates an example video encoder that can implement candidate codec mode selection based on boundary matching cost. FIG. 7 illustrates a portion of a video encoder that can implement candidate codec mode selection based on boundary matching cost. FIG. 8 conceptually illustrates the process of selecting a candidate codec mode for encoding a current block using boundary matching cost. FIG. 9 illustrates an example video decoder that can implement candidate codec mode selection based on boundary matching cost. FIG. 10 illustrates a portion of a video decoder that can implement candidate codec mode selection based on boundary matching cost. FIG. 11 conceptually illustrates the process of using a boundary matching cost to select a candidate encoding/decoding mode for decoding a current block. FIG. 12 conceptually illustrates an electronic system for implementing some embodiments of the present disclosure.

1100: Processing 1110: Step 1120: Step 1130: Step 1140: Step 1150: Step

Claims (16)

A video encoding and decoding method, comprising: receiving data of a pixel block, the data of the pixel block to be encoded or decoded into a current block of a current picture of a video; identifying multiple candidate encoding and decoding modes applicable to the current block; identifying a first candidate encoding and decoding mode group, the first candidate encoding and decoding mode group is a subset of the waiting encoding and decoding mode, wherein the number of candidate encoding and decoding modes in the first candidate encoding and decoding mode group is less than the number of candidate encoding and decoding modes in the waiting encoding and decoding mode; Calculating a boundary matching cost for each candidate codec mode of the first candidate codec mode set, the boundary matching cost being calculated by comparing (i) a plurality of reconstructed samples adjacent to the current block and (ii) a plurality of predicted samples of the current block along the boundary of the current block, wherein the predicted samples are generated according to a candidate codec mode in the first candidate codec mode set; Indexing the identified plurality of candidate codec modes in the first candidate codec mode set according to the boundary matching costs; implicitly selecting a candidate codec candidate mode from the first candidate codec mode set based on the index; or explicitly selecting the candidate codec candidate mode from the first candidate codec mode set based on a plurality of assigned codewords, wherein the codewords are assigned to a plurality of candidate codec modes in the identified candidate codec mode set according to a priority of the candidate codec modes, and the shortest codeword corresponds to the candidate with the minimum cost; and encoding or decoding the current block using the selected candidate codec mode. A video coding method as described in claim 1, wherein each candidate coding mode in the waiting coding mode is assigned to a candidate coding mode group in a plurality of candidate coding mode groups. A video coding method as described in claim 2, wherein each candidate coding mode in the waiting coding mode is associated with an original candidate index, wherein each candidate coding mode is assigned to a candidate coding mode group in K candidate coding mode groups based on a result of an original index modulo K or a result of an original index divided by K. A video coding and decoding method as described in claim 2, wherein the waiting selected coding and decoding modes corresponding to multiple spatial merging candidates are assigned to the same candidate coding and decoding mode group. A video coding method as described in claim 2, wherein the waiting selected coding modes corresponding to multiple spatial merging candidates are assigned to different candidate coding mode groups. A video coding and decoding method as described in claim 2, wherein the multiple merged candidate waiting selected coding and decoding modes whose motion difference is less than a threshold are assigned to the same candidate coding and decoding mode group. A video coding and decoding method as described in claim 2, wherein the multiple merged candidate waiting selected coding and decoding modes whose motion difference is greater than a threshold are assigned to the same candidate coding and decoding mode group. The video encoding and decoding method as described in claim 2, further comprising: Identifying a candidate encoding and decoding mode group with the lowest representative cost in the waiting encoding and decoding mode group; and Sending or receiving an index, which is used to select a candidate encoding and decoding mode from the identified candidate encoding and decoding mode group. A video coding method as described in claim 8, wherein the representative cost of the identified candidate coding mode set is the average, maximum or minimum cost of the waiting selected coding mode of the identified candidate coding mode set. The video encoding and decoding method as described in claim 2, further comprising: Sending or receiving an index for selecting a candidate encoding and decoding mode set; and Identifying a candidate encoding and decoding mode from the selected candidate encoding and decoding mode set based on the cost of the selected candidate encoding and decoding mode. A video encoding and decoding method as described in claim 1, wherein the waiting selection encoding and decoding mode includes multiple merging candidates of the current block. The video encoding and decoding method as described in claim 11, wherein the merge candidates of the current block include: (i) multiple merge candidates using combined inter-frame and intra-frame prediction; or (ii) multiple merge candidates using affine transformation motion compensation prediction. As described in claim 1, the video coding and decoding method, the waiting selected coding and decoding mode includes multiple candidate coding and decoding modes corresponding to multiple different combinations, and the different combinations are used to refine the distance and offset of motion information. As described in claim 1, the video coding and decoding method, the waiting selection coding and decoding mode includes multiple candidate coding and decoding modes corresponding to different candidate weights, and the different candidate weights are used to combine inter-frame predictions in different directions. As described in claim 1, the video coding and decoding method, the waiting selection coding and decoding mode includes multiple candidate coding and decoding modes corresponding to multiple different models, and the different models are used to derive multiple predictors of multiple chrominance samples of the current block based on multiple luminance samples of the current block. An electronic device, comprising: A video codec circuit, configured to perform multiple operations, including: Receiving data of a pixel block, the data of the pixel block to be encoded or decoded as a current block of a current picture of a video; Identifying multiple candidate codec modes applicable to the current block; Identifying a first candidate codec mode group, the first candidate codec mode group is a subset of the waiting codec mode, wherein the number of candidate codec modes in the first candidate codec mode group is less than the number of candidate codec modes in the waiting codec mode; Calculating a boundary matching cost for each candidate codec mode of the first candidate codec mode set, the boundary matching cost being calculated by comparing (i) a plurality of reconstructed samples adjacent to the current block and (ii) a plurality of predicted samples of the current block along the boundary of the current block, wherein the predicted samples are generated according to a candidate codec mode in the first candidate codec mode set; Indexing the identified plurality of candidate codec modes in the first candidate codec mode set according to the boundary matching costs; implicitly selecting a candidate codec candidate mode from the first candidate codec mode set based on the index; or explicitly selecting the candidate codec candidate mode from the first candidate codec mode set based on a plurality of assigned codewords, wherein the codewords are assigned to a plurality of candidate codec modes in the identified candidate codec mode set according to a priority of the candidate codec modes, and the shortest codeword corresponds to the candidate with the minimum cost; and encoding or decoding the current block using the selected candidate codec mode.