HK40022994B - Method and device for decoding video and computer apparatus - Google Patents

Description

Video decoding method and device, and computer equipment

Incorporated herein by reference

This application claims priority to U.S. Provisional Application No. 62/739,632, filed on October 1, 2018, entitled “Method for Constraining Sub-Block Merging Modes,” U.S. Provisional Application No. 62/742,322, filed on October 6, 2018, entitled “Method for Constraining Sub-Block-Based Merging List Creation,” and U.S. Application No. 16/559,257, filed on September 3, 2019, entitled “Method and Apparatus for Video Coding,” the entire contents of which are incorporated herein by reference.

Technical Field

The present application relates to video coding and decoding technology, and in particular to a method and apparatus for video decoding, and computer equipment.

Background Art

Video encoding and decoding are made possible by inter-picture prediction techniques with motion compensation. An uncompressed digital video may comprise a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures has a fixed or variable picture rate (also informally referred to as a frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has very large bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution, 60 Hz frame rate) with 8 bits per sample requires close to 1.5 Gbit/s of bandwidth. One hour of such video would require over 600 GB of storage space.

One goal of video encoding and decoding is to reduce redundant information in the input video signal through compression. Video compression can help reduce the bandwidth or storage space requirements mentioned above, in some cases by two or more orders of magnitude. Both lossless and lossy compression, as well as combinations of the two, can be used. Lossless compression refers to techniques that reconstruct an exact replica of the original signal from a compressed original. When lossy compression is used, the reconstructed signal may not be exactly the same as the original, but the distortion between the original and the reconstructed signal is small enough that the reconstructed signal is usable for the intended application. Lossy compression is widely used in video. The amount of distortion allowed depends on the application. For example, users of some consumer streaming applications may be able to tolerate higher distortion than users of television applications. The achievable compression ratio reflects the fact that higher allowed/tolerable distortion results in higher compression ratios.

Motion compensation can be a lossy compression technique and may involve using a block of sample data from a previously reconstructed picture or part of a reconstructed picture (reference picture) for prediction of a newly reconstructed picture or part of a picture, after being spatially shifted in the direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, where the third dimension indicates the reference picture being used (the latter may indirectly be the temporal dimension).

In some video compression techniques, the MV applied to a region of sample data can be predicted based on other MVs, such as those associated with another region of sample data that is spatially adjacent to the region being reconstructed and that precedes the MV in decoding order. This can significantly reduce the amount of data required to encode the MV, thereby eliminating redundant information and increasing compression. MV prediction can be effective, for example, when encoding an input video signal derived from a camera (referred to as natural video). There is a statistical probability that regions larger than the region to which a single MV applies will move in a similar direction. Therefore, in some cases, similar motion vectors derived from MVs in neighboring regions can be used for prediction. This results in the MV discovered for a given region being similar or identical to the MV predicted from surrounding MVs and, after entropy coding, can be represented using fewer bits than would be used if the MV were encoded directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, MV prediction can itself be lossy, for example due to rounding errors when calculating the predicted value based on several surrounding MVs.

H.265/HEVC (ITU-T Rec. H.265, “High Efficiency Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this document describes a technique hereinafter referred to as “spatial merging”.

Summary of the Invention

The embodiments of the present application provide a method and apparatus for video decoding, and a computer device, which aim to solve the problem of excessively high hardware implementation costs caused by using a sub-block-based merging mode to process a large number of small coding blocks when an area within a picture includes these small coding blocks.

An embodiment of the present application provides a method for video decoding. The method may include:

Receive the current block in the image;

determining whether a size of the current block is less than or equal to a size threshold;

When the size of the current block is less than or equal to the size threshold, disabling one or more types of sub-block based merge candidate decoding of the current block; when the size of the current block is greater than the size threshold, receiving a first flag indicating whether the sub-block based merge candidate list is used to decode the current block.

An embodiment of the present application provides another method for video decoding. The method may include:

receiving a current block in a to-be-decoded picture according to a sub-block based merge candidate list, wherein the sub-block based merge candidate list includes one or more types of sub-block based merge candidates;

For each of the one or more types of sub-block based merge candidates, determining whether a size of the current block is less than or equal to a size threshold corresponding to the type; and

When the size of the current block is less than or equal to the corresponding size threshold, disabling the sub-block based merge candidate list of the type for decoding the current block; when the size of the current block is greater than the corresponding size threshold, receiving a first flag indicating whether the sub-block based merge candidate list is used for decoding the current block.

An embodiment of the present application further provides a video decoding apparatus, comprising:

A receiving module, configured to receive a current block in the image;

a determining module, configured to determine whether the size of the current block is less than or equal to a size threshold; and

A processing module is configured to disable one or more types of sub-block based merge candidate lists for decoding the current block when the size of the current block is less than or equal to a size threshold; and receive a first flag when the size of the current block is greater than the size threshold, the first flag indicating whether the sub-block based merge candidate list is used for decoding the current block.

An embodiment of the present application further provides a video decoding apparatus, comprising:

a block receiving module, configured to receive a current block in a to-be-decoded picture according to a sub-block based merge candidate list, wherein the sub-block based merge candidate list includes one or more types of sub-block based merge candidates;

a size determination module configured to determine, for each of the one or more types of sub-block-based merge candidates, whether a size of the current block is less than or equal to a size threshold corresponding to the type; and

a block processing module, configured to, when the size of the current block is less than or equal to the corresponding size threshold, disable the sub-block-based merge candidate list of the type for decoding the current block; and, when the size of the current block is greater than the corresponding size threshold, receive a first flag indicating whether the sub-block-based merge candidate list is used for decoding the current block.

An embodiment of the present application also provides a computer device, which includes one or more processors and one or more memories, wherein the one or more memories store at least one instruction, and the at least one instruction is loaded and executed by the one or more processors to implement the video decoding method as described above.

In an embodiment of the present application, by disabling a certain type of sub-block based merge candidate decoding of the current block, the high computational cost that may be caused by pruning the sub-block merge list can be avoided, thereby solving the problem of excessive hardware implementation cost caused by using a sub-block based merge mode to process these small coding blocks when the area within the picture includes a large number of small coding blocks.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, in which:

FIG1 is a schematic diagram of a current block and its surrounding spatial merging candidates according to an embodiment.

FIG2 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

FIG3 is an illustration of a simplified block diagram of a communication system according to another embodiment.

FIG4 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

FIG5 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

FIG6 is a block diagram of an encoder according to another embodiment.

FIG7 is a block diagram of a decoder according to another embodiment.

FIG8 is a diagram illustrating an encoding process of a sub-block based temporal motion vector prediction (SbTMVP) mode according to an embodiment.

9A and 9B are diagrams illustrating a planar motion vector (MV) prediction process according to an embodiment.

FIG10 is a diagram illustrating a process of a spatio-temporal motion vector prediction (STMVP) mode according to an embodiment.

11-13 are syntax tables according to some embodiments.

FIG14 is a diagram illustrating a process of disabling one or more types of sub-block based merge candidates according to some embodiments.

FIG15 is a diagram illustrating a process of disabling certain types of sub-block based merge candidates according to some embodiments.

FIG16 is a schematic diagram of a computer system according to an embodiment.

DETAILED DESCRIPTION

1 , the current block (101) includes samples found by the encoder during motion search that can be predicted from a previous block that has been spatially shifted by the same amount. This MV is not encoded directly, but is derived from metadata associated with one or more reference pictures, such as the most recent (in decoding order) reference picture, using the MV associated with any of the five surrounding samples, denoted by A0, A1 and B0, B1, B2 (102 to 106 respectively). In H.265, MV prediction can use prediction values from the same reference picture being used by neighboring blocks.

I. Video Coding Encoders and Decoders

FIG2 is a simplified block diagram of a communication system (200) according to an embodiment disclosed in the present application. The communication system (200) includes a plurality of terminal devices, which can communicate with each other via, for example, a network (250). For example, the communication system (200) includes a first terminal device (210) and a second terminal device (220) interconnected via the network (250). In the embodiment of FIG2 , the first terminal device (210) and the second terminal device (220) perform unidirectional data transmission. For example, the first terminal device (210) can encode video data (e.g., a video picture stream collected by the terminal device (210)) for transmission to the second terminal device (220) via the network (250). The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device (220) can receive the encoded video data from the network (250), decode the encoded video data to recover the video data, and display the video picture based on the recovered video data. Unidirectional data transmission is more common in applications such as media services.

In another embodiment, a communication system (200) includes a third terminal device (230) and a fourth terminal device (240) that perform bidirectional transmission of encoded video data, which can occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (230) and the fourth terminal device (240) can encode video data (e.g., a video picture stream collected by the terminal device) for transmission to the other terminal device of the third terminal device (230) and the fourth terminal device (240) via a network (250). Each of the third terminal device (230) and the fourth terminal device (240) can also receive the encoded video data transmitted by the other terminal device of the third terminal device (230) and the fourth terminal device (240), and can decode the encoded video data to restore the video data, and can display the video picture on an accessible display device based on the restored video data.

In the embodiment of FIG2 , the first terminal device (210), the second terminal device (220), the third terminal device (230) and the fourth terminal device (240) may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and/or dedicated video conferencing equipment. The network (250) represents any number of networks that transmit encoded video data between the first terminal device (210), the second terminal device (220), the third terminal device (230) and the fourth terminal device (240), including, for example, wired (wired) and/or wireless communication networks. The communication network (250) may exchange data in circuit switching and/or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and/or the Internet. For the purposes of the present application, unless otherwise explained below, the architecture and topology of the network (250) may be irrelevant to the operations disclosed in the present application.

As an example, FIG3 shows the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

The streaming system may include an acquisition subsystem (313), which may include a video source (301) such as a digital camera, which creates an uncompressed video picture stream (302). In an embodiment, the video picture stream (302) includes samples captured by the digital camera. The video picture stream (302) is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video data (304) (or encoded video bitstream), and the video picture stream (302) can be processed by an electronic device (320), which includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream (302), the encoded video data (304) (or the encoded video code stream (304)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (304) (or the encoded video code stream (304)), which can be stored on the streaming server (305) for future use. One or more streaming client subsystems, such as the client subsystem (306) and the client subsystem (308) in Figure 3, can access the streaming server (305) to retrieve a copy (307) and a copy (309) of the encoded video data (304). The client subsystem (306) can include, for example, a video decoder (310) in an electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and produces an output video picture stream (311) that can be presented on a display (312) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the coded video data (304), video data (307), and video data (309) (e.g., a video bitstream) may be encoded according to certain video coding/compression standards. Examples of such standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application may be used in the context of the VVC standard.

It should be noted that the electronic device (320) and the electronic device (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

FIG4 is a block diagram of a video decoder (410) according to an embodiment disclosed herein. The video decoder (410) may be provided in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) may be used in place of the video decoder (310) in the embodiment of FIG3 .

A receiver (431) may receive one or more encoded video sequences to be decoded by a video decoder (410); in the same or another embodiment, one encoded video sequence is received at a time, wherein each encoded video sequence is decoded independently of the other encoded video sequences. The encoded video sequence may be received from a channel (401), which may be a hardware/software link to a storage device storing the encoded video data. The receiver (431) may receive the encoded video data as well as other data, such as encoded audio data and/or auxiliary data streams, which may be forwarded to their respective consuming entities (not shown). The receiver (431) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder/parser (420) (hereinafter referred to as "parser (420)"). In some applications, the buffer memory (415) is part of the video decoder (410). In other cases, the buffer memory (415) may be provided external to the video decoder (410) (not shown). In other cases, a buffer memory (not shown) is provided external to the video decoder (410) to, for example, mitigate network jitter, and another buffer memory (415) may be provided internally to the video decoder (410) to, for example, handle broadcast timing. Furthermore, when the receiver (431) receives data from a store/forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (415) may not be required, or the buffer memory may be smaller. Of course, for use on a traffic packet network such as the Internet, the buffer memory (415) may also be required. The buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or similar component (not shown) external to the video decoder (410).

The video decoder (410) may include a parser (420) to reconstruct symbols (421) from a coded video sequence. The categories of these symbols include information for managing the operation of the video decoder (410) and potential information for controlling a display device, such as a display device (412) (e.g., a display screen), which is not part of the electronic device (430) but can be coupled to the electronic device (430), as shown in Figure 4. The control information for the display device can be a parameter set segment (not shown) of Supplemental Enhancement Information (SEI) or Video Usability Information (VUI). The parser (420) can parse/entropy decode the received coded video sequence. The coding of the coded video sequence can be performed according to a video coding technique or standard and can follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (420) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

The parser (420) may perform entropy decoding/parsing operations on the video sequence received from the buffer memory (415), thereby creating symbols (421).

Depending on the type of coded video picture or portion of a coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (421) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (420). For the sake of brevity, the flow of such subgroup control information between the parser (420) and the multiple units below is not described.

In addition to the functional blocks already mentioned, the video decoder (410) can be conceptually broken down into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and may be integrated with each other. However, for the purposes of describing the disclosed subject matter, the conceptual breakdown into the following functional units is appropriate.

The first unit is a scaler/inverse transform unit (451). The scaler/inverse transform unit (451) receives quantized transform coefficients as symbols (421) from the parser (420) along with control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The scaler/inverse transform unit (451) may output a block comprising sample values, which may be input to an aggregator (455).

In some cases, the output samples of the scaler/inverse transform unit (451) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) uses reconstructed information extracted from the current picture buffer (458) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (458) buffers partially reconstructed current pictures and/or fully reconstructed current pictures. In some cases, the aggregator (455) adds the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler/inverse transform unit (451) on a per-sample basis.

In other cases, the output samples of the scaler/inverse transform unit (451) may belong to inter-frame coded and potentially motion compensated blocks. In this case, the motion compensated prediction unit (453) may access the reference picture memory (457) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (421), these samples may be added to the output of the scaler/inverse transform unit (451) (in this case referred to as residual samples or residual signal) by the aggregator (455) to generate output sample information. The retrieval of the prediction samples by the motion compensated prediction unit (453) from the address in the reference picture memory (457) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (453) in the form of the symbols (421), which may include, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values extracted from the reference picture memory (457) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

The output samples of the aggregator (455) may be used by various loop filtering techniques in a loop filter unit (456). The video compression techniques may include in-loop filtering techniques that are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (456) as symbols (421) from the parser (420). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of the coded video sequence, as well as to previously reconstructed and loop-filtered sample values.

The output of the loop filter unit (456) may be a sample stream that may be output to a display device (412) and stored in a reference picture memory (457) for subsequent inter-picture prediction.

Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture buffer (458) can become part of the reference picture memory (457) and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

The video decoder (410) may perform decoding operations according to a predetermined video compression technique, such as that in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata about the HRD buffer management signaled in the encoded video sequence.

In an embodiment, a receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and/or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

FIG5 is a block diagram of a video encoder (503) according to an embodiment disclosed herein. The video encoder (503) is provided in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) can be used to replace the video encoder (303) in the embodiment of FIG3 .

The video encoder (503) may receive video samples from a video source (501) (not part of the electronic device (520) in the embodiment of FIG5 ), which may capture video images to be encoded by the video encoder (503). In another embodiment, the video source (501) is part of the electronic device (520).

The video source (501) may provide a source video sequence in the form of a stream of digital video samples to be encoded by the video encoder (503), wherein the stream of digital video samples may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (501) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that are given motion when viewed sequentially. The pictures themselves may be constructed as a spatial array of pixels, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be readily understood by those skilled in the art. The following description focuses on samples.

According to an embodiment, the video encoder (503) may encode and compress pictures of a source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (550). In some embodiments, the controller (550) controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, the coupling is not shown in the figure. The parameters set by the controller (550) may include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (550) may be used to have other suitable functions that are related to the video encoder (503) optimized for a certain system design.

In some embodiments, the video encoder (503) operates in a coding loop. As a simplified description, in embodiments, the coding loop may include a source encoder (530) (e.g., responsible for creating symbols, such as a symbol stream, based on the input picture to be encoded and the reference picture) and a (local) decoder (533) embedded in the video encoder (503). The decoder (533) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression technology considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (534). Since the decoding of the symbol stream produces bit-accurate results regardless of the decoder location (local or remote), the contents of the reference picture memory (534) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same sample values that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

The operation of the "local" decoder (533) may be identical to the "remote" decoder of, for example, the video decoder (410) described in detail above in conjunction with FIG4. However, with additional brief reference to FIG4, when symbols are available and the entropy encoder (545) and parser (420) are capable of losslessly encoding/decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (410), including the buffer memory (415) and parser (420), may not be fully implemented in the local decoder (533).

At this point, it can be observed that any decoder technology other than parsing/entropy decoding present in the decoder must also be present in a substantially identical functional form in the corresponding encoder. For this reason, this application focuses on the decoder operation. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only required in certain areas and is provided below.

During operation, in some embodiments, the source encoder (530) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes an input picture with reference to one or more previously encoded pictures in a video sequence designated as "reference pictures." In this manner, the encoding engine (532) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

The local video decoder (533) can decode the encoded video data that can be designated as a reference picture based on the symbols created by the source encoder (530). The operation of the encoding engine (532) can be a lossy process. When the encoded video data can be decoded at a video decoder (not shown in Figure 5), the reconstructed video sequence can generally be a copy of the source video sequence with some errors. The local video decoder (533) replicates the decoding process that can be performed by the video decoder on the reference picture and can cause the reconstructed reference picture to be stored in the reference picture cache (534). In this way, the video encoder (503) can locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture that will be obtained by the remote video decoder.

The predictor (535) may perform a prediction search for the encoding engine (532). That is, for a new picture to be encoded, the predictor (535) may search the reference picture memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (535) may operate on a pixel-by-pixel-block basis based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (535), it may be determined that the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (534).

The controller (550) may manage encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

The outputs of all the above functional units may be entropy coded in an entropy encoder (545). The entropy encoder (545) losslessly compresses the symbols generated by the various functional units using techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

The transmitter (540) can buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission over a communication channel (560), which can be a hardware/software link to a storage device where the encoded video data will be stored. The transmitter (540) can combine the encoded video data from the video encoder (503) with other data to be transmitted, such as encoded audio data and/or an auxiliary data stream (source not shown).

The controller (550) can manage the operation of the video encoder (503). During encoding, the controller (550) can assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned to any of the following picture types:

An intra picture (I picture) can be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values for each block.

Bidirectionally predictive pictures (B pictures) can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict sample values for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

A source picture is typically spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples) and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded (spatial or intra-predicted) with reference to already coded blocks of the same picture. Pixel blocks of a P picture may be predictively coded using spatial prediction with reference to one previously coded reference picture or using temporal prediction. Blocks of a B picture may be predictively coded using spatial prediction with reference to one or two previously coded reference pictures or using temporal prediction.

The video encoder (503) may perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (503) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video coding technique or standard used.

In an embodiment, the transmitter (540) may transmit additional data along with the encoded video. The source encoder (530) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, redundant pictures and slices, and other forms of redundant data, SEI messages, VUI parameter set fragments, and the like.

The captured video may be presented as a temporal sequence of multiple source pictures (video pictures). Intra-picture prediction (often shortened to intra prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In an embodiment, a particular picture being encoded/decoded is divided into blocks, and the particular picture being encoded/decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a previously encoded and buffered reference picture in the video, the block in the current picture can be encoded using a vector called a motion vector. The motion vector points to the reference block in a reference picture, and when multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

In some embodiments, bidirectional prediction techniques can be used for inter-picture prediction. According to bidirectional prediction techniques, two reference pictures are used, for example, a first reference picture and a second reference picture, both preceding the current picture in the video in decoding order (but potentially in the past and future, respectively, in display order). A block in the current picture can be encoded using a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block can be predicted using a combination of the first and second reference blocks.

In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

According to some embodiments disclosed in the present application, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally speaking, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type used for the CU, such as an inter prediction type or an intra prediction type. In addition, depending on temporal and/or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in encoding (encoding/decoding) is performed in units of prediction blocks. Taking the luma prediction block as an example, the prediction block includes a matrix of pixel values (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

FIG6 is a diagram of a video encoder (603) according to another embodiment disclosed herein. The video encoder (603) is configured to receive a processed block (e.g., a prediction block) of sample values within a current video picture in a sequence of video pictures and encode the processed block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (603) replaces the video encoder (303) in the embodiment of FIG3 .

In an HEVC embodiment, a video encoder (603) receives a matrix of sample values for a processing block, such as a prediction block of 8×8 samples. The video encoder (603) uses, for example, rate-distortion (RD) optimization to determine whether to use intra mode, inter mode, or bi-prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (603) may use intra prediction techniques to encode the processing block into an encoded picture; and when encoding the processing block in inter mode or bi-prediction mode, the video encoder (603) may use inter prediction or bi-prediction techniques, respectively, to encode the processing block into an encoded picture. In some video coding techniques, merge mode may be an inter-picture prediction submode, in which motion vectors are derived from one or more motion vector predictors without resorting to coded motion vector components external to the predictor. In some other video coding techniques, there may be motion vector components applicable to the subject block. In an embodiment, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

In the embodiment of Figure 6, the video encoder (603) includes an inter-frame encoder (630), an intra-frame encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), a general controller (621) and an entropy encoder (625) coupled together as shown in Figure 6.

The inter-frame encoder (630) is configured to receive samples of a current block (e.g., a processing block), compare the current block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference picture is a decoded reference picture decoded based on the encoded video information.

The intra-frame encoder (622) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously encoded blocks in the same picture in some cases, generate quantized coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information based on one or more intra-frame coding techniques). In an embodiment, the intra-frame encoder (622) further calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same picture.

The general controller (621) is used to determine general control data and control other components of the video encoder (603) based on the general control data. In an embodiment, the general controller (621) determines the mode of the block and provides a control signal to the switch (626) based on the mode. For example, when the mode is intra mode, the general controller (621) controls the switch (626) to select the intra mode result for use by the residual calculator (623) and controls the entropy encoder (625) to select intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (621) controls the switch (626) to select the inter prediction result for use by the residual calculator (623) and controls the entropy encoder (625) to select inter prediction information and add the inter prediction information to the bitstream.

The residual calculator (623) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (622) or the inter-frame encoder (630). The residual encoder (624) is used to operate based on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (624) is used to convert the residual data from the time domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (622) and the inter-frame encoder (630). For example, the inter-frame encoder (630) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra-frame encoder (622) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

The entropy encoder (625) is used to format the codestream to produce encoded blocks. The entropy encoder (625) generates various information according to a suitable standard, such as the HEVC standard. In an embodiment, the entropy encoder (625) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information from the codestream. It should be noted that according to the disclosed subject matter, when encoding a block in inter-frame mode or the merge sub-mode of bidirectional prediction mode, there is no residual information.

FIG7 is a diagram of a video decoder (710) according to another embodiment disclosed herein. The video decoder (710) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In one embodiment, the video decoder (710) replaces the video decoder (310) in the embodiment of FIG3 .

In the FIG. 7 embodiment, the video decoder ( 710 ) includes an entropy decoder ( 771 ), an inter-frame decoder ( 780 ), a residual decoder ( 773 ), a reconstruction module ( 774 ), and an intra-frame decoder ( 772 ) coupled together as shown in FIG. 7 .

The entropy decoder (771) is operable to reconstruct certain symbols representing syntax elements constituting the coded picture from the coded picture. Such symbols may include, for example, the mode used to encode the block (e.g., intra mode, inter mode, bidirectional prediction mode, a combined submode of the latter two, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata for prediction by the intra decoder (772) or the inter decoder (780), respectively, residual information in the form of, for example, quantized transform coefficients, and the like. In an embodiment, when the prediction mode is inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (780); and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (772). The residual information may be inverse quantized and provided to the residual decoder (773).

The inter-frame decoder (780) is configured to receive inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

The intra-frame decoder (772) is configured to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

The residual decoder (773) is used to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require certain control information (to obtain the quantizer parameter QP), and this information can be provided by the entropy decoder (771) (the data path is not shown because this is only low-level control information).

The reconstruction module (774) is used to combine the residual output by the residual decoder (773) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block. The reconstructed block can be part of a reconstructed picture, which in turn can be part of a reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve visual quality.

It should be noted that the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710) may be implemented using any suitable technology. In one embodiment, the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710) may be implemented using one or more processors executing software instructions.

II. Sub-block-based merging mode and sub-block-based merging candidate list

Examples of sub-block (or sub-CU) based merge modes may include affine merge mode, sub-block based temporal motion vector prediction (SbTMVP) mode (also known as advanced temporal motion vector prediction (ATMVP) mode), inter-frame plane motion vector (MV) prediction mode, spatio-temporal motion vector prediction (STMVP) mode, etc. Different types of merge candidates corresponding to different types of sub-block based merge modes may be included in the sub-block based merge candidate list for predicting the motion vector of the coding block.

1. Affine Merge Mode

In affine motion compensation, a 6-parameter or simplified 4-parameter affine model can be used to efficiently predict the motion information of all samples within the current block. For example, in an affine-encoded or described block, different parts of the samples can have different motion vectors. In an affine-encoded or described block, the basic unit with a motion vector is called a sub-block. In different embodiments, the size of a sub-block can be as small as just one sample or as large as the size of the current block.

When the affine merge mode is used to encode or decode the current block, a motion vector (relative to the target reference picture) can be derived for each sample in the current block using the affine model represented by the affine merge candidate. In order to reduce implementation complexity, in some embodiments, affine motion compensation is performed on a sub-block basis rather than on a sample basis. For example, each sub-block will have a motion vector derived from the affine model. The motion vector is the same for the samples in each sub-block. Assuming that a specific position of each sub-block (such as the upper left corner or center point of the sub-block) is a representative position of the corresponding sub-block, it can be used to derive the corresponding motion vector based on the affine model. In one example, the sub-block has a size of 4×4 samples.

In one embodiment, when using affine merge mode, an affine model represented by the motion vectors at a set of control points of the current block is used as a merge candidate, which is included in the sub-block based merge candidate list. This type of affine merge candidate is called a constructed affine candidate, which is constructed based on the motion vectors at the control points of the current block.

In one embodiment, when using affine merge mode, the affine models of the spatial or temporal neighboring blocks of the current block are used as merge candidates, which are included in the sub-block based merge candidate list. This type of affine merge candidate is called an inherited affine candidate, which is inherited from the affine-encoded neighboring blocks of the current block.

Using affine merge mode can involve high computational cost and high memory access. For example, a bidirectional affine merge mode results in multiple bidirectionally predicted 4×4 blocks and can significantly increase memory bandwidth requirements. Therefore, when implemented in hardware, affine merge mode has high implementation complexity.

2. Sub-block based TMVP (SbTMVP) mode

In SbTMVP mode, the temporal motion vector prediction for the current block (or current coding unit) used in merge mode defined in the HEVC standard is modified. Multiple sets of motion information (including motion vectors and reference indices) are extracted from blocks smaller than the current block. Figure 8 shows the encoding process (800) of SbTMVP mode according to one embodiment.

In the SbTMVP mode encoding process (800), the prediction of the motion vector of the sub-CU812 within the CU811 in the current picture (810) can be obtained in two steps. The first step is to identify the corresponding block (821) in the reference picture (820) using the so-called time vector (801). The reference picture (820) is called the motion source picture (820). The second step is to split the current CU (811) into sub-CUs (812) and obtain the prediction of the motion vector and the reference index of each sub-CU (812) from the block (822) corresponding to each sub-CU (812). In one example, each sub-CU (812) is a square block with a size of N×N samples. In one example, N is set to 8 by default.

In a first step, a reference picture (820) and a corresponding block (821) may be determined by using motion information of one of the spatially neighboring blocks of the current CU (811). In some implementations, to avoid repeated scanning of neighboring blocks, the first merge candidate in the merge candidate list of the current CU (811) may be used as the temporal vector (801). For example, the first available motion vector on the merge candidate list and its associated reference index are set to the temporal vector (801) and the index (820) of the motion source picture. This allows for more accurate identification of the corresponding block (821) compared to temporal motion vector prediction in HEVC, where the corresponding block (sometimes referred to as a collocated block) is always in the lower right or center position relative to the current CU.

In a second step, for each sub-CU (812), motion information for the sub-CU (812) is derived using the motion information (823) of the block (822) (from the minimum motion grid covering the center sample of the block (822). After identifying the motion information (823) of the corresponding N×N block (822), the motion information (823) is converted into a motion vector and reference index for the current sub-CU (812) in the same manner as temporal motion vector prediction in HEVC. For example, motion scaling and other processes may be applied. For example, the decoder may check whether a low latency condition is met (i.e., the picture order number (POC) of all reference pictures of the current picture is less than the POC of the current picture). In some examples, the motion vector MVx (corresponding to the motion vector of reference picture list X) may be used to predict the motion vector MVy (where X is equal to 0 or 1 and Y is equal to 1-X) for each sub-CU (812).

3. Planar MV prediction mode

Planar MV prediction mode (also called inter-frame planar MV prediction mode, or inter-frame planar mode) can provide special merge candidates with sub-block MVs. Planar MV prediction mode can be used to generate smooth fine-grained motion fields. Figures 9A-9B show the planar MV prediction process according to one embodiment.

In Figures 9A-9B, the current block (901) is divided into sub-blocks, for example, each sub-block having a size of 4×4 samples. The current block (901) may have a width of W samples and a height of H samples. The sub-block (910) at position (x, y) is referred to as the current sub-block, which is used as an example to illustrate how the motion vector P(x, y) of the sub-block (910) is derived. The coordinates (x, y) are relative to the upper left sub-block (902) within the current block (901), and the coordinates of the upper left sub-block (902) are (0, 0).

For example, the motion vector P(x, y) may be derived by averaging the horizontal linear interpolation _Ph (x, y) and the vertical linear interpolation _Pv (x, y) on a 4×4 block basis using the following formula:

P(x, y)=(H×P _h (x, y)+W×P _v (x, y)+H×W)/(2×H×W)

In one example, the horizontal linear interpolation _Ph (x, y) and the vertical linear interpolation _Pv (x, y) can be derived according to the following formula:

P _h (x, y)=(W-1-x)×L(-1, y)+(x+1)×R(W, y)

P _v (x, y)=(H-1-y)×A(x,-1)+(y+1)×B(x, H)

Wherein, L(-1, y) and R(W, y) are the motion vectors of the 4×4 block (911) and the 4×4 block (912) on the left and right sides of the current block (901), and A(x, -1) and B(x, H) are the motion vectors of the 4×4 block (913) and the 4×4 block (914) above and below the current block (901).

The motion vector of the right column adjacent 4×4 block can be calculated based on the derived motion information (denoted by BR) of the lower right temporal adjacent 4×4 block (905) and the motion information (denoted by AR) of the upper right adjacent 4×4 block (904). The motion vector of the bottom row adjacent 4×4 block can be calculated based on the derived motion information of the lower right adjacent 4×4 block (905) and the motion information (denoted by BL) of the lower left adjacent 4×4 block (903).

For example, the motion vectors R(W, y) and B(x, H) can be determined according to the following equations:

R(W,y)=((H-y-1)AR+(y+1)BR)/H

B(x,H)=((W-x-1)BL+(x+1)BR)/W.

4.STMVP model

In STMVP mode, the motion vectors of the sub-CUs of the current CU (or current block) can be recursively derived in raster scan order. Figure 10 shows the STMVP process according to one embodiment. In Figure 10, the 8×8 CU (1010) is divided into four 4×4 sub-CUs (or sub-blocks) A, B, C, and D. The four adjacent 4×4 blocks of the current CU (1010) in the current picture are labeled a, b, c, and d. In other examples, the sub-CUs can have different sizes.

For example, the motion derivation of sub-CU A can start by identifying two spatial neighboring blocks of sub-CU A. The first neighboring block considered is the 4×4 block above sub-CU A (block c). If block c is not available or is intra-coded, check the other 4×4 blocks above sub-CU A (check from left to right starting from block c). The second neighboring block is the 4×4 block to the left of sub-CU A (block b). If block b is not available or is intra-coded, check the other blocks to the left of sub-CU A (check from top to bottom starting from block b). The motion information obtained from the two neighboring blocks of each reference picture list is scaled to the first reference picture of the given list. Next, for example, the temporal motion vector prediction value of sub-block A is derived according to the same process as the TMVP derivation specified in HEVC. For example, the motion information of the collocated block in the reference picture at position D is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information of two spatial neighboring blocks and one temporal neighboring block (if available), all available motion vectors (up to 3) can be averaged for each reference picture list. The average motion vector is designated as the motion vector prediction for the current sub-CU (1010).

5. Sub-block based inter-frame merging candidate list

Affine merge mode, inter-frame plane mode, STMVP mode, and SbTMVP mode are all sub-block-based merge modes. When using sub-block-based merge mode to process the current block, the current block is divided into sub-blocks. The MV values in these sub-blocks are derived and used as the MV prediction for encoding or decoding the current block. It should be noted that affine merge mode, inter-frame plane mode, STMVP mode, and SbTMVP mode are used as examples to illustrate sub-block-based merge modes. In other examples, other types of sub-block-based merge modes can be used.

In one embodiment, different types of sub-block based merge candidates corresponding to different sub-block based merge modes may be determined and included in a sub-block based merge candidate list for processing the current block. The sub-block based merge candidate list may be separate from the block based inter-frame merge candidate list, as used in the test model of HEVC or Universal Video Coding (VVC), VTM-2.0.

In one embodiment, a sub-block-based merge candidate list may be constructed according to the following steps:

(i) Insert SbTMVP candidates, which include sub-block motion vectors derived in SbTMVP mode.

(ii) Insert STMVP candidates, which include sub-block motion vectors derived in STMVP mode.

(iii) Insert a set of inherited affine candidates obtained in affine merge mode.

(iv) Insert a set of constructed affine candidates obtained in affine merge mode.

(v) Insert a plane MV prediction candidate, which includes the sub-block motion vector derived in plane MV prediction mode.

(vi) Fill with zero motion vectors.

In the above steps, different types of sub-block based merge candidates are added to the sub-block based merge candidate list until the maximum allowed number of sub-block based merge candidate lists is reached. When the available sub-block based merge candidates cannot fill all positions of the sub-block based merge candidate list, generated candidates (e.g., zero motion vectors) may be filled.

When a separate sub-block based inter merge candidate list is used, a usage flag may be present after the merge flag in the bitstream to indicate the use of the sub-block based merge candidate list.

III. Enabling and disabling sub-block based inter-frame merge candidate lists

In some embodiments, a sub-block based inter-frame merge candidate list (also referred to as a sub-block merge list or a sub-block merge candidate list) may be separated from a block-based inter-frame merge candidate list (or referred to as a block merge list). When a sub-block merge list is used, high computational costs may be incurred. For example, when a sub-block based merge candidate is added to a sub-block merge list, a pruning process may be performed. The pruning process may involve comparing the candidate to be added with the candidates on the sub-block merge list. As described above, the candidates on the sub-block merge list may include motion vectors from multiple sub-blocks, which are far more than the motion vectors of the candidates on the block merge list. Therefore, pruning the sub-block merge list may result in higher computational costs compared to the block merge list. Therefore, when an area within a picture includes a large number of small coding blocks, it may be too costly to process these small coding blocks using a sub-block based merge mode.

To address the above issues, in some embodiments, when the current block has a smaller size than a size threshold, the sub-block merge list may be disabled. As a result, the sub-block based merge mode will not be used to encode or decode the smaller current block.

In some embodiments, a threshold value of the current block size may be used when constructing sub-block based merge candidates in a separate list. When the current block size is below the threshold, the sub-block based merge list is disabled for the current block. In some embodiments, a threshold value of the number of pixels (or samples) of the current block may be used when constructing sub-block based merge candidates in a separate list. When the number of pixels of the block is below the threshold, the sub-block based merge list is disabled for the current block.

In some embodiments, when the sub-block merge candidate list is used as a separate merge list, only sub-block-based candidates are included in the sub-block merge candidate list. The sub-block-based candidate can be any type of sub-block-based inter-frame merge candidate and is not limited to the aforementioned sub-block-based merge candidate types (affine candidate, SbTMVP candidate, STMVP candidate, planar MV candidate, etc.). A unified set of conditions can be used to enable or disable the sub-block-based MV prediction mode associated with the sub-block merge candidate list.

Example A

In one embodiment, a predefined threshold can be used to constrain the size of the current block for which a sub-block-based merge list can be used. When the height or width of the current block is less than the threshold, the sub-block-based merge list will not be used. In other words, all sub-block merge modes are disabled for the current block. The flag for using a sub-block-based merge list can be inferred to be false. In one example, the predefined threshold can be 16 luma samples. In another example, the predefined threshold can be 8 luma samples. The threshold value can be any number and is not limited to these examples.

In another embodiment, the threshold value may be signaled in the bitstream instead of a predefined threshold value. For example, the threshold value may be signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a tile group header, etc.

FIG11 shows a syntax table (1100) according to one embodiment of the present application. The syntax table (1100) includes 16 lines of syntax that describe how syntax elements are transmitted in the bitstream and how to parse the syntax elements when the bitstream is processed in the decoder. These syntax elements provide the parameters required to decode the current block (or coding unit) at the coordinates [x0, y0].

Specifically, at line 3, it is determined whether skip mode is to be used by checking cu_skip_flag[x0][y0]. If skip mode is not to be used, then starting at line 9, merge mode is considered. When skip mode is used to process the current block, the residual of the current block is encoded and transmitted in the bitstream. When merge mode is used to process the current block, no residual signal is generated or encoded.

When skip mode is used, on line 4, whether to enable or disable the sub-block merge list is determined by checking the condition "(sps_affine_enabled_flag||sps_atmvp_enabled_flag||sps_mvplanar_enabled_flag)&&!(cbWidth<8||cbHeight<8)". In particular, the condition "!(cbWidth<8||cbHeight<8)" indicates whether the width (represented by cbWidth) or height (cbHeight) of the current block is less than a threshold (8 samples). When the width (represented by cbWidth) or height (cbHeight) of the current block is less than the threshold, the condition "!(cbWidth<8||cbHeight<8)" will be false. Accordingly, the condition "(sps_affine_enabled_flag||sps_atmvp_enabled_flag||sps_mvplanar_enabled_flag)&&!(cbWidth<8||cbHeight<8)" will be false, and the sub-block merge list will be disabled.

If the sub-block merge list is enabled (e.g., either the width or the height is greater than or equal to the threshold (8 samples)), the syntax element "merge_subblock_flag[x0][y0]" (which is a usage flag) is transmitted in the bitstream. Or in other words, the next syntax element in the bitstream is interpreted as "merge_subblock_flag[x0][y0]". For example, the syntax element "merge_subblock_flag[x0][y0]" can indicate whether the sub-block merge list is to be used.

Conversely, if the sub-block merge list is disabled, the syntax element "merge_subblock_flag[x0][y0]" will not be transmitted in the bitstream. Accordingly, the decoder infers "merge_subblock_flag[x0][y0]" to be false and will not use the sub-block merge list to decode the current block.

Similarly, when merge mode is used, on line 11, a size threshold (8 samples) is used to determine whether the next syntax element is "merge_subblock_flag[x0][y0]". When the weight or height of the current block is greater than or equal to the size threshold, the next syntax element is not "merge_subblock_flag[x0][y0]". The decoder interprets "merge_subblock_flag[x0][y0]] as false and will not use the sub-block merge list to decode the current block.

Example B

In another embodiment, a predefined threshold can be used to constrain the size of the current block for which a sub-block-based merge list can be used. When both the height and width of the current block are less than the threshold, all sub-block merge modes are disabled for the current block. The flag for using a sub-block-based merge list can be inferred to be false. In one example, the predefined threshold can be 16 luma samples. In another example, the predefined threshold can be 8 luma samples. The threshold value can be any number of samples and is not limited to these examples.

In another embodiment, the threshold value may be signaled in the bitstream instead of a predefined threshold value. For example, the threshold value may be signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a tile group header, etc.

Figure 12 shows a syntax table (1200) according to an embodiment of the present application. Syntax table (1200) is similar to syntax table (1100). A size threshold (16 samples) is used in lines 4 and 11 to determine whether to disable the sub-block merge list decoding of the current block at position [x0, y0]. Compared to Figure 11, the condition for comparing the width and height of the current block is expressed as "!(cbWidth<16&&cbHeight<16)". When the width and height of the current block are both less than 16 samples, the condition will be false and the sub-block merge list will be disabled. Accordingly, the next syntax element is not "merge_subblock_flag[x0][y0]". The syntax element "merge_subblock_flag[x0][y0]" will be inferred to be false.

Example C

In another embodiment, a predefined threshold representing the number of samples within a coding block can be used to constrain the size of the current block for which sub-block-based merge lists can be used. When the number of luma samples in the current block is less than the threshold, all sub-block merge modes are disabled for the current block. The flag for using sub-block-based merge lists can be inferred to be false. In one example, the threshold can be 64 luma samples. In another example, the threshold can be 128 luma samples. The value of the threshold can be any number and is not limited to these examples.

In another embodiment, the threshold value may be signaled in the bitstream instead of a predefined threshold value. For example, the threshold value may be signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a tile group header, etc.

FIG13 shows a syntax table (1300) according to an embodiment of the present application. The syntax table (1300) is similar to the syntax table (1100). A size threshold (64 samples within the coding block) is used in lines 4 and 11 to determine whether to disable sub-block based merge list decoding of the current block at position [x0, y0]. In contrast to FIG11 , the condition for comparing the size of the current block is expressed as “!(cbWidth*cbHeight<64)”. The size of the current block represented by the number of samples within the current block is represented by cbWidth*cbHeight. When the number of samples within the current block is less than 64 samples, the condition will be false and the sub-block merge list will be disabled. Accordingly, the next syntax element is not “merge_subblock_flag[x0][y0]”. The syntax element “merge_subblock_flag[x0][y0]” will be inferred to be false.

IV. Disabling Classes of Sub-Block-Based Merge Candidates

To address the high computational cost associated with using a sub-block-based merge list, in some embodiments, one or more specific types of sub-block merge modes can be individually disabled. When the size of the current block is smaller than a mode-specific size threshold (or type-dependent size threshold), the corresponding sub-block merge mode can be disabled. Consequently, candidates for the disabled sub-block merge modes are not added to the sub-block merge list used to encode or decode the current block.

In some embodiments, when constructing sub-block based merge candidates in a separate merge list (separate from the block based merge list), a separate condition of the current block size may be used to disable each category of sub-block based merge candidates so that candidates of the disabled category may not be added to the separate merge list.

In some embodiments, some categories of sub-block based merge candidates are conditionally included in the sub-block merge list.The sub-block merge list may not always include all categories of sub-block based merge candidates.

Similarly, in some embodiments, when the sub-block merge candidate list is used as a separate merge list, only sub-block based candidates are included. The sub-block based candidate can be any type of sub-block based inter-frame merge candidate, and is not limited to the aforementioned sub-block based merge candidate types (affine candidate, SbTMVP candidate, STMVP candidate, planar MV candidate, etc.).

Example D

In one embodiment, when the condition of the size of the current block is met, the affine merge candidate may not be added to the sub-block merge candidate list. The condition may be one of the following:

(a) The height or width of the current block is less than a threshold, such as the number of samples;

(b) the height or width of the current block is less than or equal to the threshold;

(c) The height and width of the current block are both smaller than the threshold;

(d) The height and width of the current block are both less than or equal to the threshold;

(e) The value of height*width is less than the threshold;

(f) The value of height*width is less than or equal to the threshold.

In one embodiment, when either of the width and height of the current block is less than 8 luma samples, the affine merge candidate may be disabled and not added to the sub-block merge candidate list.

Example E

In one embodiment, when another condition of the size of the current block is met, the SbTMVP candidate may not be added to the sub-block merge candidate list. The condition may be one of the following:

(a) The height or width of the current block is less than the threshold;

(b) the height or width of the current block is less than or equal to the threshold;

(c) The height and width of the current block are both smaller than the threshold;

(d) The height and width of the current block are both less than or equal to the threshold;

(e) The value of height*width is less than the threshold;

(f) The value of height*width is less than or equal to the threshold.

In one embodiment, when both the width and height of the current block are less than 8 luma samples, the SbTMVP candidate may be disabled and not added to the sub-block merge candidate list.

Example F

When another condition of the size of the current block is met, the plane MV prediction candidate may not be added to the sub-block merging candidate list. The condition may be one of the following:

(a) The height or width of the current block is less than the threshold;

(b) the height or width of the current block is less than or equal to the threshold;

(c) The height and width of the current block are both smaller than the threshold;

(d) The height and width of the current block are both less than or equal to the threshold;

(e) The value of height*width is less than the threshold;

(f) The value of height*width is less than or equal to the threshold.

In one embodiment, when both the width and height of the current block are less than 16 luma samples, the SbTMVP candidate may be disabled and not added to the candidate list.

Example G

It is also possible to conditionally disable categories of sub-block based merge candidates other than the categories of sub-block based merge candidates described in Embodiments D-F using separate conditions for each category.

Example H

In one embodiment, when constructing a sub-block merge list, when the sub-block merge candidate list is empty, or the number of available sub-block-based merge candidates is less than the maximum allowed number of candidates, some generated merge candidates may be added to the sub-block merge list until the number of candidates equals the maximum allowed number.

In one embodiment, zero motion candidates are used for the remaining empty positions on the sub-block merging candidate list until the sub-block merging candidate list is filled. These zero candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference pictures used by these candidates is 1 and 2, for unidirectional prediction and bidirectional prediction, respectively.

For example, the first zero candidate may use the first reference picture indexed 0 on the first reference picture list L0 and the first reference picture indexed 0 on the second reference picture list L1 for bidirectional prediction in the subblock of the current block (each including two motion vectors). The second zero candidate may use the second reference picture indexed 1 on the list L0 and the second reference picture indexed 1 on the list L1 for bidirectional prediction in the subblock of the current block (each including two motion vectors).

In another embodiment, if at least two sub-block-based candidates are added to the sub-block merge list, and at least the first candidate among the candidates has an MV on list L0 and the second candidate among the candidates has an MV on list L1, a combined bi-predicted sub-block-based candidate is generated. In one example, the combined bi-predicted sub-block merge candidate is only used for B-slices. The combined bi-predicted sub-block candidate is generated by combining the first reference picture list (e.g., table L0 or table L1) motion parameters of the first candidate with the second reference picture list (e.g., table L1 or table L0) motion parameters of the second candidate for each sub-block. If the two sets of motion parameters provide different motion hypotheses, they will form a new bi-predicted sub-block-based candidate. In one example, if the sub-block merge list is still not full after adding all available combined bi-predicted sub-block-based candidates, zero motion candidates are added to the sub-block merge list until the sub-block merge list is full.

Example 1

In one embodiment, type-dependent thresholds used in the aforementioned conditions may be predefined, where the aforementioned conditions are used to respectively determine whether to disable a specific type of sub-block merging candidate.

In another embodiment, type-dependent thresholds used in the aforementioned conditions for respectively determining whether to disable a particular type of sub-block merging candidate may be signaled in the bitstream. For example, the type-dependent thresholds may be signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a tile group header, etc.

Example J

The mechanism of disabling each category of sub-block-based merge candidates using a separate condition based on the size of the current block can be applied to sub-block-based merge modes with residual data, or to skip mode, which uses sub-block-based merge modes without encoding residual data. For example, when processing the current block using a sub-block merge list, the residual of the current block can be encoded when skip mode is not used. Conversely, in other examples, skip mode can be used. Accordingly, the current block is processed using a sub-block merge list without encoding the residual.

Furthermore, the constraints applied to sub-block merge candidates of the same type can be different for merge mode with residual data and skip mode for processing coding blocks. For example, a first size threshold can be used to disable affine mode for merge mode with residual data. A second size threshold can be used to disable affine mode for skip mode. The first size threshold and the second size threshold can be different.

In various embodiments, when the current block is encoded using skip mode (sub-block based merge mode without residual data) or sub-block based merge mode with residual data, sub-block based merge candidates may be disabled so that a sub-block based merge list may be constructed in one of the following ways.

In one embodiment, all sub-block based merge candidates may be disabled. In other words, the entire sub-block based merge list is disabled, and no candidate may be added to the sub-block based merge list.

In another embodiment, affine merge candidates are simply disabled and not added to the list.

In another embodiment, sub-block based merge candidates except affine merge candidates are simply disabled and not added to the list.

In another embodiment, different conditions are used to disable affine merge candidates and other sub-block based candidates.

V. Procedure for disabling sub-block based merging mode

FIG14 illustrates a process (1400) for disabling one or more types of sub-block based merge candidates according to some embodiments of the present application. The process (1400) may be performed to decode the current block. In various embodiments, the process (1400) is performed by a processing circuit, such as a processing circuit in a terminal device (210), (220), (230), and (240), a processing circuit that performs the functions of a video decoder (310), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (710), or the like. In some embodiments, the process (1400) is implemented as software instructions, so that when the processing circuit executes the software instructions, the processing circuit performs the process (1400). The process begins at step (S1401) and performs step (S1410).

At step (S1410), a current block may be received. For example, the current block may have a specific size represented by a width and a height of the current block.

At step (S1420), it is determined whether the size of the current block is less than or equal to a size threshold. In one example, the size of the current block may be determined to be less than or equal to the size threshold when one of the height and width of the current block is less than or equal to the threshold. In one example, the size of the current block may be determined to be less than or equal to the size threshold when both the height and width of the current block are less than or equal to the threshold. In one example, the size of the current block may be determined to be less than or equal to the size threshold when the number of samples in the current block is less than or equal to the threshold. In different examples, different size thresholds may be used.

In step ( S1430 ), when the size of the current block is less than or equal to the size threshold, one or more types of sub-block based merging candidates may be disabled for decoding the current block.

In one example, the size threshold is a type-dependent size threshold specified for one or more sub-block-based merge modes. Therefore, the corresponding sub-block-based merge modes are disabled. In other words, one or more types of sub-block-based merge candidates corresponding to the corresponding one or more disabled sub-block-based merge modes are not used to construct a sub-block merge list for decoding the current block.

In some examples, each of one or more types of sub-block based merge candidates may have a corresponding type-dependent size threshold. These corresponding type-dependent size thresholds may be different or the same for a subset or all of the different types of sub-block based merge candidates. Steps (S1420) and (S1430) may be performed for each type-dependent size threshold, such that each type of sub-block based merge candidate may be disabled separately according to its corresponding type-dependent size threshold.

In one embodiment, a size threshold is specified for disabling sub-block-based merge candidate lists. Accordingly, sub-block-based merge lists are disabled. In other words, no sub-block merge list is used for decoding the current block. The process (1400) may proceed to step (S1499) and end at step (S1499).

FIG15 illustrates a process (1600) for disabling a certain type of sub-block-based merge candidate according to some embodiments of the present application. The process (1600) may be performed to decode the current block. In various embodiments, the process (1600) is performed by a processing circuit, such as a processing circuit in a terminal device (210), (220), (230), and (240), a processing circuit that performs the functions of a video decoder (310), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (710), or the like. In some embodiments, the process (1600) is implemented as software instructions, so that when the processing circuit executes the software instructions, the processing circuit performs the process (1600). The process begins at step (S1601) and performs step (S1610).

At step (S1610), a current block in a to-be-decoded picture is received according to a sub-block-based merge candidate list, the sub-block-based merge candidate list including one or more types of sub-block-based merge candidates. For example, the current block may have a specific size represented by a width and a height of the current block.

At step (S1620), for each of one or more types of sub-block based merge candidates, it is determined whether the size of the current block is less than or equal to a size threshold corresponding to that type. In one example, when one of the height and width of the current block is less than or equal to the threshold, the size of the current block may be determined to be less than or equal to the size threshold. In one example, when both the height and width of the current block are less than or equal to the threshold, the size of the current block is determined to be less than or equal to the size threshold. In one example, when the number of samples in the current block is less than or equal to the threshold, the size of the current block is determined to be less than or equal to the size threshold. In an embodiment, the size thresholds for one or more types of sub-block based merge candidates may be the same.

In step (S1630), when the size of the current block is less than or equal to the corresponding size threshold, this type of sub-block based merging candidate decoding of the current block is disabled.

The process (1600) may proceed to step (S1699) and end at step (S1699).

An embodiment of the present application provides a video decoding apparatus, including:

A receiving module, configured to receive a current block in the image;

a determining module, configured to determine whether the size of the current block is less than or equal to a size threshold; and

A processing module is configured to disable one or more types of sub-block based merging candidates for decoding the current block when the size of the current block is less than or equal to a size threshold.

The embodiment of the present application further provides a video decoding apparatus, comprising:

a block receiving module, configured to receive a current block in a to-be-decoded picture according to a sub-block based merge candidate list, wherein the sub-block based merge candidate list includes one or more types of sub-block based merge candidates;

a size determination module configured to determine, for each of the one or more types of sub-block based merge candidates, whether a size of the current block is less than or equal to a corresponding size threshold; and

The block processing module is configured to disable the corresponding type of sub-block based merging candidate decoding of the current block when the size of the current block is less than or equal to the corresponding size threshold.

The specific functions and implementations of the modules described in the embodiments of the present application can refer to the specific processes of the video decoding method in the above embodiments.

An embodiment of the present application also provides a computer device, which includes one or more processors and one or more memories, wherein at least one instruction is stored in the one or more memories, and the at least one instruction is loaded and executed by the one or more processors to implement the video decoding method described in the above embodiment.

An embodiment of the present application further provides a non-volatile computer-readable storage medium storing instructions, which, when executed by a computer for video decoding, enables the computer to perform the video decoding method described in the above embodiment.

VI. Computer Systems

The above techniques can be implemented as computer software via computer-readable instructions and physically stored in one or more computer-readable media.For example, Figure 16 illustrates a computer system (1500) suitable for implementing certain embodiments of the disclosed subject matter.

The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or other mechanisms to create code comprising instructions, which may be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), or the like, or executed through decoding, microcode, or the like.

The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, and the like.

The components for the computer system (1500) shown in FIG16 are exemplary in nature and are not intended to limit the scope of use or functionality of the computer software implementing the embodiments of the present application. Nor should the configuration of the components be interpreted as having any dependency or requirement on any component or combination of components shown in the exemplary embodiment of the computer system (1500).

The computer system (1500) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-computer interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

The human-computer interface input device may include one or more of the following (only one of which is depicted): keyboard (1501), mouse (1502), touchpad (1503), touch screen (1510), data gloves (not shown), joystick (1505), microphone (1506), scanner (1507), camera (1508).

The computer system (1500) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell/taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (1510), a data glove (not shown), or a joystick (1505), but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (1509), headphones (not shown)), visual output devices (e.g., screens (1510) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities—some of which may output two-dimensional visual output or output in more than three dimensions through means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and cigarette boxes (not shown)), and printers (not shown).

The computer system (1500) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only/rewritable compact discs (CD/DVD ROM/RW) (1520) with CD/DVD or similar media (1521), thumb drives (1522), removable hard drives or solid-state drives (1523), traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM/ASIC/PLD-based devices such as security software dongles (not shown), and the like.

Those skilled in the art will also understand that the term "computer-readable media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

The computer system (1500) may also include an interface to one or more communication networks. For example, the network may be wireless, wired, or optical. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle network, an industrial network, a real-time network, a delay-tolerant network, and the like. Networks also include local area networks such as Ethernet, wireless local area networks, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), in-vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter for connecting to some universal data port or peripheral bus (1549) (for example, a USB port of the computer system (1500)); other systems are typically integrated into the core of the computer system (1500) by connecting to a system bus as described below (for example, an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer system (1500) can communicate with other entities. The communication can be one-way, for receiving only (e.g., wireless television), one-way, for sending only (e.g., a CAN bus to certain CAN bus devices), or two-way, such as to other computer systems via a local or wide area digital network. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the core (1540) of the computer system (1500).

The core (1540) may include one or more central processing units (CPUs) (1541), graphics processing units (GPUs) (1542), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1543), hardware accelerators for specific tasks (1544), and the like. These devices, as well as read-only memory (ROM) (1545), random access memory (1546), internal mass storage (e.g., internal non-user accessible hard disk drives, solid-state drives, etc.) (1547), and the like, may be connected via a system bus (1548). In some computer systems, the system bus (1548) may be accessed in the form of one or more physical plugs so that it can be expanded with additional central processing units, graphics processing units, and the like. Peripheral devices may be attached directly to the core's system bus (1548) or connected via a peripheral bus (1549). Peripheral bus architectures include PCI (Peripheral Controller Interface), USB (Universal Serial Bus), and the like.

The CPU (1541), GPU (1542), FPGA (1543), and accelerator (1544) can execute certain instructions, which, when combined, can constitute the aforementioned computer code. The computer code can be stored in ROM (1545) or RAM (1546). Transient data can also be stored in RAM (1546), while permanent data can be stored, for example, in internal mass storage (1547). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1541), GPUs (1542), mass storage (1547), ROM (1545), RAM (1546), etc.

The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of this application, or may be medium and code well known and available to those skilled in the art of computer software.

As an example and not a limitation, a computer system having architecture (1500), in particular core (1540), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the core (1540) having non-volatile properties, such as core internal mass storage (1547) or ROM (1545). Software implementing various embodiments of the present application can be stored in such a device and executed by the core (1540). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the core (1540), in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM (1546) and modifying such a data structure according to a software-defined process. Additionally or alternatively, the computer system may provide functionality hardwired in logic or otherwise contained in circuitry (e.g., accelerator (1544)) that may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing the executing software, circuitry containing the executing logic, or both. The present application includes any suitable combination of hardware and software.

Appendix A: Acronyms

JEM: Joint Development Model

VVC: Versatile Video Coding

BMS: Benchmark Collection

MV: Motion Vector

HEVC: High Efficiency Video Coding

SEI: Supplemental Enhancement Information

VUI: Video Availability Information

GOPs: Group of Pictures

TUs: Transformation Units

PUs: prediction units

CTUs: Coding Tree Units

CTBs: Coding Tree Blocks

PBs: prediction blocks

HRD: Hypothesized Reference Decoder

SNR: Signal-to-Noise Ratio

CPUs: Central Processing Units

GPUs: Graphics Processing Units

CRT: cathode ray tube

LCD: Liquid Crystal Display

OLED: Organic Light-Emitting Diode

CD: compact disc

DVD: Digital Video Disc

ROM: Read-Only Memory

RAM: Random Access Memory

ASIC: Application-Specific Integrated Circuit

PLD: Programmable Logic Device

LAN: Local Area Network

GSM: Global System for Mobile Communications

LTE: Long Term Evolution

CANBus: Controller Area Network Bus

USB: Universal Serial Bus

PCI: Peripheral Component Interconnect

FPGA: Field Programmable Gate Array

SSD: Solid State Drive

IC: integrated circuit

CU: Coding Unit

HMVP: History-based Motion Vector Prediction

MVP: Motion Vector Prediction

TMVP: Temporal Motion Vector Prediction

TPM: Triangular Prediction Model

VTM: Universal Video Coding Test Model

ATMVP: Advanced Temporal Motion Vector Prediction

SbTMVP: Sub-block Temporal Motion Vector Prediction

POC: Image sequence number

STMVP: Spatio-Temporal Motion Vector Prediction

SPS: Sequence Parameter Set

PPS: Picture Parameter Set

Although this application has described a number of exemplary embodiments, various modifications, permutations, and equivalent substitutions of the embodiments are within the scope of this application. Therefore, it should be understood that those skilled in the art will be able to design a variety of systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are therefore within the spirit and scope of this application.

Claims (14)

1. A method for video decoding, characterized in that it includes: Receive the current block in the image; Determine whether the size of the current block is less than or equal to a size threshold; and When the size of the current block is less than or equal to the size threshold, one or more types of sub-block-based merge candidate decoding of the current block are disabled; when the size of the current block is greater than the size threshold, a first flag is received, the first flag indicating whether the sub-block-based merge candidate list is used to decode the current block. 2. The method according to claim 1, wherein disabling one or more types of sub-block-based merge candidate decoding of the current block comprises: Disable decoding of the current block based on the sub-block merge candidate list; or infer that the first flag is false. 3. The method according to claim 1, wherein determining whether the size of the current block is less than or equal to the size threshold includes: Determine whether the height or width of the current block is less than or equal to the size threshold; Determine whether the height and width of the current block are both less than or equal to the size threshold; or Determine whether the number of samples in the current block is less than or equal to the size threshold. 4. The method according to claim 1, characterized in that it further comprises: Receive a syntax element that indicates the size threshold. 5. The method according to claim 1, characterized in that it further comprises: Receive a second flag, which indicates whether skip mode encoding of the current block is enabled; and When the skip mode is disabled, a third flag is received, indicating whether merge mode encoding of the current block is enabled. 6. The method according to claim 1, wherein the one or more types of sub-block-based merge candidates include one of the following: Affine merging candidate Sub-block-based temporal motion vector prediction (SbTMVP) candidates Candidates for Space-Time Motion Vector Prediction (STMVP), and Candidates for planar motion vector prediction. 7. A method for video decoding, characterized in that it includes: The current block in the image to be decoded is received according to a sub-block-based merge candidate list, wherein the sub-block-based merge candidate list includes one or more types of sub-block-based merge candidates; For each of the one or more types of sub-block-based merge candidates, determine whether the size of the current block is less than or equal to a size threshold corresponding to that type; and When the size of the current block is less than or equal to the corresponding size threshold, the decoding of the current block based on sub-block merge candidates of that type is disabled; when the size of the current block is greater than the corresponding size threshold, a first flag is received, the first flag indicating whether the sub-block based merge candidate list is used to decode the current block. 8. The method according to claim 7, wherein the size threshold of the various types of sub-block-based merging candidates is the same. 9. The method according to claim 7, further comprising: Construct the sub-block-based merge candidate list. Specifically, when the number of available sub-block merging candidates in the sub-block-based merging candidate list is less than the maximum allowed number of sub-block merging candidates, the generated sub-block merging candidate is added to the sub-block-based merging candidate list until the number of sub-block merging candidates in the sub-block-based merging candidate list equals the maximum allowed number of sub-block merging candidates. 10. The method according to claim 9, wherein adding the generated sub-block merge candidates to the sub-block-based merge candidate list comprises: Add the zero-motion candidate to the sub-block-based merge candidate list. 11. The method according to claim 9, wherein adding the generated sub-block merge candidates to the sub-block-based merge candidate list comprises: Add the combined bidirectional predictive sub-block merge candidate to the sub-block-based merge candidate list. 12. A video decoding apparatus, characterized in that it comprises: The receiving module is used to receive the current block in the image; A determining module is used to determine whether the size of the current block is less than or equal to a size threshold; and The processing module is configured to disable one or more types of sub-block-based merge candidate decoding of the current block when the size of the current block is less than or equal to a size threshold; and to receive a first flag when the size of the current block is greater than the size threshold, the first flag indicating whether the sub-block-based merge candidate list is used to decode the current block. 13. A video decoding apparatus, characterized in that it comprises: A block receiving module is configured to receive the current block in the image to be decoded according to a sub-block-based merging candidate list, wherein the sub-block-based merging candidate list includes one or more types of sub-block-based merging candidates; A size determination module is configured to, for each of the one or more types of sub-block-based merging candidates, determine whether the size of the current block is less than or equal to a size threshold corresponding to that type; and The block processing module is configured to disable the decoding of the current block based on sub-block merge candidates when the size of the current block is less than or equal to the corresponding size threshold; and to receive a first flag when the size of the current block is greater than the corresponding size threshold, the first flag indicating whether the sub-block-based merge candidate list is used to decode the current block. 14. A computer device, characterized in that the device comprises one or more processors and one or more memories, the one or more memories storing at least one instruction, the at least one instruction being loaded and executed by the one or more processors to implement the video decoding method as described in any one of claims 1-11.