HK40087723B - Method and apparatus for video decoding, electronic device and storage medium - Google Patents

Description

Video decoding methods and apparatus, electronic devices and storage media

By incorporating via reference

This application claims priority to U.S. Patent Application No. 17/826,806, filed May 27, 2022, entitled “CONTENT-ADAPTIVE ONLINE TRAINING METHOD AND APPARATUS FOR DEBLOCKING IN BLOCK-WISEIMAGE COMPRESSION”, and U.S. Provisional Application No. 63/211,408, filed June 16, 2021, entitled “Content Adaptive Online Training Deblocking for Block-Wise Image Compression”. The disclosure of the earlier applications is incorporated herein by reference in its entirety.

Technical Field

This application relates to the field of video encoding and decoding technology, and in particular to a video decoding method and apparatus, electronic device and computer-readable storage medium.

Background Technology

The purpose of the background description provided herein is to provide a general overview of the context of this disclosure. To the extent that the work described in this background section is intended, neither the work of the currently named inventors nor any aspect of the description which at the time of filing may not otherwise be described as prior art is expressly or implicitly acknowledged as prior art to this disclosure.

Image and/or video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital images and/or video can comprise a series of pictures, each with a spatial dimension of, for example, 1920×1080 luminance samples and associated chrominance samples. This series of pictures can have, for example, a fixed or variable picture rate (also informally referred to as frame rate) of 60 pictures per second or 60 Hz. Uncompressed images and/or video have specific bit rate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a frame rate of 60 Hz) requires close to 1.5 Gbit/s of bandwidth. One hour of such video would require more than 600 gigabytes of storage space.

One objective of image and/or video encoding and decoding can be to reduce redundancy in the input image and/or video signal through compression. Compression can help reduce the aforementioned bandwidth and/or storage requirements, in some cases by two orders of magnitude or more. Although the description herein uses video encoding/decoding as an illustrative example, the same techniques can be applied to image encoding/decoding in a similar manner without departing from the spirit of this disclosure. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique that reconstructs an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. In the case of video, lossy compression is widely used. The amount of distortion tolerated depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can be reflected in the fact that higher allowable/tolerable distortion can result in a higher compression ratio.

Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

Video codec techniques can include techniques known as intra-frame coding. In intra-frame coding, sample values are represented without reference to samples or other data from previously reconstructed reference images. In some video codecs, an image is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the image can be an intra-frame image. Intra-frame images and their derivatives (e.g., independent decoder refresh images) can be used to reset the decoder state and therefore can be used as the first image in the encoded video bitstream and video session, or as a still image. Samples of an intra-frame block can be transformed, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique that minimizes the sample values in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block at a given quantization step size.

Traditional intra-frame coding, such as intra-frame coding techniques known from, for example, MPEG-2 generation coding techniques, does not use intra-frame prediction. However, some newer video compression techniques include those that attempt to predict based on surrounding sample data and/or metadata obtained, for example, during the encoding and/or decoding of spatially adjacent and first-decoding data blocks. Such techniques are referred to below as "intra-frame prediction" techniques. Note that in at least some cases, intra-frame prediction uses reference data only from the current picture under reconstruction, and not reference data from a reference picture.

There can be many different forms of intra-prediction. When more than one such technique can be used in a given video coding technique, the technique used can be encoded in an intra-prediction mode. In some cases, a mode can have sub-modes and/or parameters, and these sub-modes and/or parameters can be encoded individually or included in the mode codeword. Which codeword is used for a given combination of modes, sub-modes, and/or parameters can affect the coding efficiency gain through intra-prediction, and therefore the entropy coding technique used to convert the codeword into a bitstream can also affect the coding efficiency gain through intra-prediction.

Some intra-frame prediction modes were introduced in H.264, refined in H.265, and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Sets (BMS). Predictor blocks can be formed using neighboring sample values belonging to an already available sample. The sample values of neighboring samples are copied into the predictor block according to the direction. The reference for the direction used can be encoded in the bitstream, or the reference for the direction used can be predicted manually.

In the prior art, improvements in a single module (e.g., encoder) in a hybrid video codec cannot be passed on to another module (e.g., decoder), and therefore may not result in an overall performance gain.

Summary of the Invention

This application also provides a video decoding method, comprising: reconstructing blocks of an image to be reconstructed from an encoded video bitstream; decoding first deblocking information in the encoded video bitstream, the first deblocking information including first deblocking parameters of a deep neural network (DNN) in a video decoder, the first deblocking parameters of the DNN being update parameters previously determined through a content-adaptive training process; determining a DNN in the video decoder for a first boundary region including a subset of samples in the reconstructed blocks based on the first deblocking parameters included in the first deblocking information; and deblocking the first boundary region including a subset of samples in the reconstructed blocks based on the determined DNN corresponding to the first deblocking parameters.

In this implementation, the reconstructed block includes a first adjacent reconstructed block, which has a first shared boundary and includes a first boundary region of samples on both sides of the first shared boundary. The first adjacent reconstructed block also includes a non-boundary region outside the first boundary region. The first boundary region in the first adjacent reconstructed block is replaced with the first boundary region after deblocking.

In the implementation, the first deblocking parameter is a bias term or weight coefficient in the DNN.

In this implementation, the DNN is configured with initial parameters. Determining the DNN in the video decoder includes updating one of the initial parameters based on the first deblocking parameters.

In one implementation, the first deblocking information indicates the difference between the first deblocking parameter and one of the initial parameters. The video decoding method further includes determining the first deblocking parameter based on the difference and the sum of the initial parameters.

In an implementation, the reconstructed block includes a second adjacent reconstructed block, the second adjacent reconstructed block having a second shared boundary and including a second boundary region of samples on both sides of the second shared boundary. The video decoding method further includes: decoding second deblocking information in the encoded video bitstream corresponding to the second boundary region, the second deblocking information indicating second deblocking parameters previously determined through a content-adaptive training process, the second boundary region being different from the first boundary region; updating a DNN based on the first and second deblocking parameters, the updated DNN corresponding to the second boundary region and configured with the first and second deblocking parameters; and deblocking the second boundary region based on the updated DNN corresponding to the second boundary region.

In one implementation, the reconstructed block includes a second adjacent reconstructed block, the second adjacent reconstructed block having a second shared boundary and including a second boundary region having samples on both sides of the second shared boundary. The video decoding method further includes deblocking the second boundary region based on a determined DNN corresponding to the first boundary region.

In this implementation, the number of layers in the DNN depends on the size of the first boundary region.

In one implementation, the first boundary region also includes samples on both sides of a third shared boundary, which lies between third adjacent reconstruction blocks included in the reconstruction block. The first adjacent reconstruction block is different from the third adjacent reconstruction block.

This application embodiment also provides a video decoding apparatus including a processing circuit system, comprising: a reconstruction module for reconstructing blocks of an image to be reconstructed from an encoded video bitstream; a first decoding module for decoding first deblocking information in the encoded video bitstream, the first deblocking information including first deblocking parameters of a deep neural network (DNN) in a video decoder, the first deblocking parameters of the DNN being update parameters previously determined through a content adaptive training process; a first determination module for determining a DNN in the video decoder for a first boundary region including a subset of samples in the reconstructed blocks based on the first deblocking parameters included in the first deblocking information; and a first deblocking module for deblocking the first boundary region including a subset of samples in the reconstructed blocks based on the determined DNN corresponding to the first deblocking parameters.

This application also provides an electronic device, including a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the electronic device to perform the video decoding method described above.

This application also provides a non-transitory computer-readable storage medium that stores a program, which can be executed by at least one processor to perform the video decoding method described above.

Therefore, this application provides a video encoding/decoding method for content-adaptive online training in neural image compression (NIC). In an NN-based video coding framework, different modules can be jointly optimized from input to output to improve the final goal through a learning or training process, thereby producing an end-to-end optimized NIC.

Attached Figure Description

Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

Figure 1A is a schematic illustration of an exemplary subset of intra-prediction modes according to an embodiment of this application.

Figure 1B is an illustration of an exemplary intra-frame prediction direction according to an embodiment of this application.

Figure 2 shows the current block and surrounding samples according to an embodiment of this application.

Figure 3 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present application.

Figure 4 is a schematic illustration of a simplified block diagram of a communication system according to another embodiment of this application.

Figure 5 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment of this application.

Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment of the present application.

Figure 7 shows a block diagram of an encoder according to another embodiment of this application.

Figure 8 shows a block diagram of a decoder according to another embodiment of this application.

Figure 9A shows an example of block image encoding according to an embodiment of this application.

Figure 9B illustrates an exemplary NIC framework according to an embodiment of this application.

Figure 10 illustrates an exemplary convolutional neural network (CNN) of a master encoder network according to an embodiment of this application.

Figure 11 illustrates an exemplary CNN of a master decoder network according to an embodiment of this application.

Figure 12 illustrates an exemplary CNN of a super encoder according to an embodiment of this application.

Figure 13 illustrates an exemplary CNN of a super decoder according to an embodiment of this application.

Figure 14 illustrates an exemplary CNN of a context model network according to an embodiment of this application.

Figure 15 illustrates an exemplary CNN of an entropy parameter network according to an embodiment of this application.

Figure 16A illustrates an exemplary video encoder according to an embodiment of this application.

Figure 16B illustrates an exemplary video decoder according to an embodiment of this application.

Figure 17 illustrates an exemplary video encoder according to another embodiment of this application.

Figure 18 illustrates an exemplary video decoder according to another embodiment of this application.

Figures 19A to 19C illustrate exemplary deblocking processes according to embodiments of this application.

Figure 20 illustrates an exemplary deblocking process according to another embodiment of this application.

Figure 21 illustrates an exemplary deblocking process based on multiple deblocking models according to an embodiment of this application.

Figures 22A and 22B illustrate exemplary deblocking processes of other embodiments of this application.

Figure 23 shows a flowchart outlining an encoding process according to an embodiment of this application.

Figure 24 shows a flowchart outlining a decoding process according to an embodiment of this application.

Figure 25 is a schematic diagram of a computer system according to an embodiment of the present application.

Detailed Implementation

Referring to Figure 1A, the lower right corner depicts a subset of nine predictor directions known from the 33 possible predictor directions of H.265 (corresponding to 33 angular modes of 35 intra-frame modes). The point (101) where the arrows intersect represents the sample being predicted. The arrows indicate the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples at a 45-degree angle to the horizontal line in the upper right. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples at a 22.5-degree angle to the horizontal line in the lower left of sample (101).

Referring again to Figure 1A, the upper left corner depicts a 4×4 square block (104) of samples (indicated by a bold dashed line). This square block (104) comprises 16 samples, each labeled with “S”, its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from top) and the first sample in the X dimension (from left). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. Also shown are reference samples following a similar numbering scheme. Reference samples are labeled with R, their Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, predicted samples are adjacent to the reconstructed block; therefore, negative values are not required.

Intra-frame image prediction can work by copying reference sample values from appropriate neighboring samples along the prediction direction notified by signaling. For example, suppose the encoded video bitstream includes signaling for that block indicating a prediction direction consistent with arrow (102)—that is, predicting samples based on one or more prediction samples at a 45-degree angle to the upper right of the horizontal line. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Then, sample S44 is predicted based on reference sample R08.

In some cases, for example, the values of multiple reference samples can be combined by interpolation to calculate the reference sample; especially when the orientation cannot be uniformly divided at 45 degrees.

As video coding technologies have evolved, the number of possible directions has also increased. In H.264 (2003), nine different directions could be represented. In H.265 (2013), this increased to 33, and JEM/VVC/BMS, when publicly released, could support up to 65 directions. Experiments have been conducted to identify the most probable directions, and certain techniques in entropy coding have been used to represent these possible directions with a small number of bits, at the cost of fewer possible directions. Furthermore, the direction itself can sometimes be predicted based on adjacent directions used in adjacent decoded blocks.

Figure 1B shows a schematic diagram (110) depicting 65 intra-frame predicted directions based on JEM to illustrate the number of predicted directions increasing over time.

The mapping of intra-predicted direction bits representing direction in a encoded video bitstream can vary depending on the video coding technique; and this mapping can range from, for example, a simple direct mapping of the predicted direction to intra-predicted modes, codewords, complex adaptive schemes involving the most probable modes, and similar techniques. However, in all cases, there may be certain directions that are statistically less likely to appear in the video content compared to some other directions. Since the goal of video compression is to reduce redundancy, in well-functioning video coding techniques, those less probable directions will be represented by a larger number of bits compared to the more probable directions.

Motion compensation can be a lossy compression technique and can involve techniques in which blocks of sample data from a previously reconstructed image or a portion thereof (the reference image), spatially shifted in a direction indicated by a motion vector (MV), are used to predict the reconstructed image or a portion thereof. In some cases, the reference image may be the same as the image under the current reconstruction. The MV may have two dimensions, X and Y, or three dimensions, the third dimension being an indication of the reference image in use (indirectly, the third dimension may be a temporal dimension).

In some video compression techniques, an MV applicable to a specific region of sample data can be predicted based on other MVs, for example, based on another MV that is spatially adjacent to the region in the reconstructed sample data and precedes that MV in the decoding order. This prediction can significantly reduce the amount of data required to encode the MV, thereby eliminating redundancy and increasing compression. MV prediction works effectively, for example, because when encoding the input video signal (called natural video) derived from the camera, there is a statistically likely area larger than the region to which a single MV is applicable has moved in similar directions, and therefore, in some cases, the larger area can be predicted using similar motion vectors derived from the MVs of adjacent regions. This makes the MV obtained for a given region similar to or the same as the MV predicted based on the surrounding MVs, and can again be represented by fewer bits after entropy encoding compared to the number of bits used in directly encoding the MV. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating the predictor based on several surrounding MVs.

Various MV prediction mechanisms are described in H.265/HEVC (ITU-T H.265 Recommendation, “High Efficiency Video Coding”, December 2016). Among the various MV prediction mechanisms provided by H.265, the technique referred to here as “spatial merging” is described.

Referring to Figure 2, the current block (201) includes samples obtained by the encoder during the motion search process, which can be predicted based on previous blocks of the same size that have been spatially shifted. Instead of directly encoding this MV, the MV can be derived from, for example, the most recent (in decoding order) reference image associated with one or more reference images, using the MV associated with any of five surrounding samples denoted by A0, A1 and B0, B1, B2 (corresponding to 202 to 206, respectively). In H.265, MV prediction can use a predictor from the same reference image being used by adjacent blocks.

Figure 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, terminal device (310) may encode video data (e.g., a video image stream captured by terminal device (310)) for transmission via the network (350) to another terminal device (320). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. Terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover the video images, and display the video images based on the recovered video data. Unidirectional data transmission can be common in media service applications, etc.

In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For the bidirectional transmission of data, in this example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video images captured by the terminal device) for transmission via a network (350) to the other terminal device (330) and (340). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other terminal device (330) and (340), and may decode the encoded video data to recover video images, and may display the video images on an accessible display device based on the recovered video data.

In the example of Figure 3, the terminal devices (310), (320), (330), and (340) may be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and/or dedicated video conferencing equipment. Network (350) refers to any number of networks that transmit encoded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired (connected) and/or wireless communication networks. Communication networks (350) may exchange data in circuit-switched channels and/or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may be irrelevant to the operation of this disclosure unless explained herein.

As an example of the application of the disclosed subject matter, Figure 4 illustrates the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter can also be applied to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

The streaming system may include a capture subsystem (413) that may include a video source (401), such as a digital camera, that creates, for example, an uncompressed video picture stream (402). In the example, the video picture stream (402) includes samples captured by the digital camera. The video picture stream (402) is depicted as a thick line to emphasize the high amount of data when compared to encoded video data (404) (or encoded video bitstream), which may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to implement or carry out aspects of the disclosed subject matter as described in more detail below. Encoded video data (404) (or encoded video bitstream) is depicted as a thin line to emphasize its lower data volume when compared to the video picture stream (402). The encoded video data (404) (or encoded video bitstream (404)) may be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, may access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an outgoing video picture stream (411) that can be displayed on a display (412) (e.g., a screen) or another presentation device (not depicted). In some streaming systems, encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to certain video coding/compression standards. Examples of these standards include ITU-T Recommendation H.265. In this example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed topics can be used in the context of VVC.

Note that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

Figure 5 shows a block diagram of a video decoder (510) according to an embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuitry system). The video decoder (510) may be used in place of the video decoder (410) in the example of Figure 4.

The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences can be received from a channel (501), which can be a hardware/software link to a storage device storing the encoded video data. The receiver (531) can receive encoded video data as well as other data, such as encoded audio data and/or auxiliary data streams, which can be forwarded to their respective user entities (not depicted). The receiver (531) can separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder/parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other applications, the buffer memory (515) can be external to the video decoder (510) (not depicted). In some other applications, a buffer memory (not depicted) may exist outside the video decoder (510) to prevent network jitter, for example, and another buffer memory (515) may exist inside the video decoder (510) to handle broadcast timing, for example. The buffer memory (515) may not be needed when the receiver (531) receives data from a store/forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, or the buffer memory (515) may be small. For use on best-effort packet networks such as the Internet, a buffer memory (515) may be required, which may be relatively large and advantageously have an adaptive size, and may be implemented at least partially outside the video decoder (510) in an operating system or similar component (not depicted).

The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (510), and possibly information for controlling a presentation device such as a presentation device (512) (e.g., a display screen), which is not part of the electronic device (530) but may be coupled to it, as shown in FIG. 5. The control information for the presentation device may be in the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not depicted). The parser (520) may perform parsing/entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may conform to video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) can extract a subgroup parameter set from the encoded video sequence for at least one subgroup of pixel subgroups in the video decoder, based on at least one parameter corresponding to the group. Subgroups may include: Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser (520) can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc., from the encoded video sequence.

The parser (520) can perform entropy decoding/parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

The reconstruction of the symbol (521) may involve multiple different units depending on the type of the encoded video picture or its parts (e.g., inter-frame picture and intra-frame picture, inter-frame block and intra-frame block) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by the parser (520). For clarity, such subgroup control information flow between the parser (520) and the following multiple units is not depicted.

In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

The first unit is the scaler/inverse transform unit (551). The scaler/inverse transform unit (551) receives quantized transform coefficients and control information (including which transform to use, block size, quantization factor, quantization scaling matrix, etc.) as one or more symbols (521) from the parser (520). The scaler/inverse transform unit (551) can output a block containing sample values, which can be input into the aggregator (555).

In some cases, the output samples of the scaler/inverse transform unit (551) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates blocks of the same size and shape as the blocks in the reconstruction using surrounding reconstructed information obtained from the current picture buffer (558). For example, the current picture buffer (558) buffers partially reconstructed current images and/or fully reconstructed current images. In some cases, the aggregator (555) adds the predictive information already generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler/inverse transform unit (551) based on each sample.

In other cases, the output samples of the scaler/inverse transform unit (551) may belong to inter-frame coded and possibly motion-compensated blocks. In such cases, the motion compensation prediction unit (553) can access the reference image memory (557) to obtain samples for prediction. After motion compensation of the obtained samples according to the symbols (521) belonging to the block, these samples can be added by the aggregator (555) to the output of the scaler/inverse transform unit (551) (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory (557) from which the motion compensation prediction unit (553) obtains the predicted samples can be controlled by motion vectors, which can be obtained by the motion compensation prediction unit (553) in the form of symbols (521), which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values obtained from the reference image memory (557) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression technique may include an in-loop filtering technique controlled by parameters included in the encoded video sequence (also known as the encoded video bitstream), which may be obtained by the loop filter unit (556) as symbols (521) from the parser (520). However, the in-loop filtering technique may also respond to metadata obtained during the decoding of previous portions of the encoded picture or encoded video sequence (in the decoding order), as well as to previously reconstructed and loop-filtered sample values.

The output of the loop filter unit (556) can be a sample stream, which can be output to the presentation device (512) and stored in the reference image memory (557) for use in future inter-frame image prediction.

Once fully reconstructed, certain encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and that encoded image (by, for example, the parser (520)) is identified as a reference image, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before reconstructing subsequent encoded images begins.

The video decoder (510) can perform decoding operations according to a predetermined video compression technique as specified in a standard such as ITU-T Recommendation H.265. The encoded video sequence can conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and in the sense that the configuration file recorded in the video compression technique or standard is available. Specifically, the configuration file can select certain tools from all available tools in the video compression technique or standard as tools available only under that configuration file. For compliance, the complexity of the encoded video sequence also needs to be within the limits defined by the level of the video compression technique or standard. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sample rate (measured, for example, in megapixels per second), maximum reference image size, etc. In some cases, the limitations set by the level can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is notified by signaling in the encoded video sequence.

In this implementation, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of one or more encoded video sequences. The additional data may be used by the video decoder (510) to properly decode the data and/or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

Figure 6 shows a block diagram of a video encoder (603) according to an embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit system). The video encoder (603) can be used in place of the video encoder (403) in the example of Figure 4.

The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of Figure 6), which can capture one or more video images to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

The video source (601) can provide a sequence of source video in the form of a digital video sample stream to be encoded by the video encoder (603). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit…), any color space (e.g., BT.601YCrCb, RGB…), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device storing previously prepared video. In a video conferencing system, the video source (601) can be a camera device capturing local image information as a video sequence. The video data can be provided as multiple individual pictures given motion when viewed sequentially. The pictures themselves can be organized as spatial pixel arrays, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

According to the implementation, the video encoder (603) can encode and compress images of the source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (650). In some implementations, the controller (650) controls and is functionally coupled to other functional units as described below. Coupling is not depicted for simplicity. Parameters set by the controller (650) may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions belonging to the video encoder (603) optimized for a specific system design.

In some implementations, the video encoder (603) is configured to operate within an encoding loop. As a highly simplified description, in this example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input picture to be encoded and one or more reference pictures) and a (local) decoder (633) embedded within the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local and remote encoders. In other words, the encoder's prediction portion "treats" the reference picture samples as if they were the exact same sample values that the decoder "sees" during prediction. This fundamental principle of image synchronization (and the offset that occurs when synchronization cannot be maintained, for example, due to channel errors) is also used in some related techniques.

The operation of the “local” decoder (633) can be the same as that of the “remote” decoder, such as the video decoder (510) which has been described in detail above in conjunction with FIG5. However, referring briefly to FIG5 again, since symbols are available and the encoding of symbols into a encoded video sequence by the entropy encoder (645) and the decoding of symbols by the parser (520) can be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), can be completely omitted in the local decoder (633).

In the implementation, any decoder technique other than parsing/entropy decoding present in the decoder exists in the corresponding encoder with the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are contrasted with the fully described decoder techniques. In some aspects, a more detailed description is provided below.

In some examples, during operation, the source encoder (630) may perform motion-compensated predictive coding, which predictively codes the input image with reference to one or more previously encoded images designated as "reference images" from the video sequence. In this way, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of one or more reference images (which may be selected as prediction references for the input image(s)).

The local video decoder (633) can decode the encoded video data of a picture that can be designated as a reference picture based on symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be lossy. When the encoded video data can be decoded at the video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding processing that can be performed on the reference picture by the video decoder, and can store the reconstructed reference picture in the reference picture memory (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that shares common content with the reconstructed reference picture that will be obtained by the remote video decoder (no transmission errors).

The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference image motion vectors, block shapes, etc., that can be used as suitable prediction references for the new image. The predictor (635) can operate on a sample-by-sample block-by-pixel basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

The outputs of all the functional units mentioned above can be entropy encoded in the entropy encoder (645). The entropy encoder (645) converts these symbols into a coded video sequence by performing lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable-length coding, arithmetic coding, etc.

The transmitter (640) can buffer one or more encoded video sequences created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware/software link to a storage device storing the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (source not shown).

The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoding image type to each encoded image, which may affect the encoding techniques that can be applied to the corresponding image. For example, one of the following image types can typically be assigned to an image:

Intra-frame pictures (I-pictures) are pictures that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh.

"IDR" images. Those skilled in the art will understand the variations of I images and their corresponding applications and characteristics.

Predictive images (P-images) can be images that are encoded and decoded using inter-frame or intra-frame prediction that uses at most one motion vector and a reference index to predict the sample values of each block.

A bidirectional predictive picture (B-picture) can be an image that can be encoded and decoded using inter-frame or intra-frame prediction that uses at most two motion vectors and a reference index to predict sample values for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

The source image can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and encoded block by block. These blocks can be predictively encoded with reference to other (encoded) blocks, which are determined by the coding assignments of the corresponding images applied to the blocks. For example, blocks of image I can be non-predictively encoded, or blocks of image I can be predictively encoded (spatial prediction or intra-frame prediction) with reference to coded blocks of the same image. Pixel blocks of image P can be predictively encoded with reference to a previously encoded reference image via spatial prediction or via temporal prediction. Blocks of image B can be predictively encoded with reference to one or two previously encoded reference images via spatial prediction or via temporal prediction.

The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

In this implementation, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, etc.

Video can be captured in a time-series manner as multiple source images (video frames). Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In the example, the specific image being encoded/decoded (referred to as the current image) is segmented into blocks. Where a block in the current image resembles a reference block in a previously encoded and buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference images.

In some implementations, bidirectional prediction techniques can be used for inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image that both precede the current image in the video in decoding order (but may be in the past and future in display order, respectively). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted using a combination of the first and second reference blocks.

In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

According to some embodiments of this disclosure, predictions such as inter-frame picture prediction and intra-frame picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video picture sequence are segmented into coding tree units (CTUs) for compression, with CTUs in the images having the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU comprises three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively divided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In the example, each CU is analyzed to determine the prediction type for that CU, such as inter-frame prediction or intra-frame prediction. Depending on temporal and/or spatial predictability, a CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In implementations, prediction operations in encoding/decoding are performed on a block-by-block basis. Using a luma prediction block as an example, a prediction block includes a matrix of pixel values (e.g., luma values) such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

Figure 7 illustrates a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive processing blocks (e.g., prediction blocks) of sample values within a current video picture in a video picture sequence, and to encode the processing blocks into encoded pictures that are part of an encoded video sequence. In this example, the video encoder (703) is used instead of the video encoder (403) in the example of Figure 4.

In the HEVC example, the video encoder (703) receives a matrix of sample values for a processing block, such as an 8×8 sample prediction block. The video encoder (703) uses, for example, rate-distortion optimization to determine whether to best encode the processing block using intra-frame mode, inter-frame mode, or bidirectional prediction mode. When encoding the processing block in intra-frame mode, the video encoder (703) can encode the processing block into an encoded picture using intra-frame prediction techniques; while when encoding the processing block in inter-frame mode or bidirectional prediction mode, the video encoder (703) can encode the processing block into an encoded picture using inter-frame prediction or bidirectional prediction techniques, respectively. In some video coding techniques, the merging mode can be an inter-frame picture prediction sub-mode, where motion vectors are derived from the predictor without the aid of encoded motion vector components outside of one or more motion vector predictors. In some other video coding techniques, motion vector components applicable to the object block may exist. In this example, the video encoder (703) includes other components, such as a mode decision module (not shown) that determines the mode of the processing block.

In the example of Figure 7, the video encoder (703) includes an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in Figure 7.

An inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images), generate inter-frame prediction information (e.g., a description based on redundancy information, motion vectors, and merging mode information of the inter-frame coding technique), and compute inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image based on the encoded video information.

The intra encoder (722) is configured to: receive samples of the current block (e.g., the processing block); in some cases compare the block with already encoded blocks in the same image; generate quantization coefficients after transformation; and in some cases also generate intra prediction information (e.g., intra prediction direction information based on one or more intra coding techniques). In the example, the intra encoder (722) also calculates an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same image.

The overall controller (721) is configured to determine overall control data and control other components of the video encoder (703) based on the overall control data. In an example, the overall controller (721) determines the mode of the block and provides control signals to the switch (726) based on the mode. For example, when the mode is intra-frame mode, the overall controller (721) controls the switch (726) to select intra-frame mode results for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; and when the mode is inter-frame mode, the overall controller (721) controls the switch (726) to select inter-frame prediction results for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

A residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from an intra encoder (722) or an inter encoder (730). A residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In an example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. In some examples, the decoding blocks are properly processed to generate decoded images, and these decoded images can be buffered in memory circuitry (not shown) and used as reference images.

An entropy encoder (725) is configured to format the bitstream to include coded blocks. The entropy encoder (725) is configured to include various information according to a suitable standard, such as the HEVC standard. In this example, the entropy encoder (725) is configured to include overall control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Note that, according to the disclosed subject matter, residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

Figure 8 illustrates a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In this example, the video decoder (810) is used instead of the video decoder (410) in the example of Figure 4.

In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together as shown in Figure 8.

The entropy decoder (871) can be configured to reconstruct certain symbols from the encoded picture, which represent the syntax elements constituting the encoded picture. Such symbols may include, for example, the mode encoding the block (e.g., intra-mode, inter-mode, bidirectional prediction mode, a combined sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-prediction information or inter-prediction information) that can identify certain samples or metadata used by the intra-decoder (872) or inter-decoder (880) for prediction, such as residual information in the form of quantized transform coefficients, etc. In the example, when the prediction mode is inter-mode or bidirectional prediction mode, the inter-prediction information is provided to the inter-decoder (880); and when the prediction type is intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may be inversely quantized and provided to the residual decoder (873).

The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

The intra-frame decoder (872) is configured to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

The residual decoder (873) is configured to perform inverse quantization to extract the dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to include quantizer parameters (QP)), and this information may be provided by the entropy decoder (871) (the data path is not depicted because this may only be a small amount of control information).

The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) with the prediction result (output by the inter-frame prediction module or intra-frame prediction module, depending on the situation) in the spatial domain to form a reconstructed block, which can be a part of a reconstructed image, which in turn can be a part of a reconstructed video. Note that other suitable operations, such as deblocking, can be performed to improve visual quality.

Note that any suitable technology can be used to implement the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810). In one implementation, one or more integrated circuits can be used to implement the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810). In another implementation, one or more processors that execute software instructions can be used to implement the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810).

This disclosure describes video coding techniques related to neural image compression and/or neural video compression, such as artificial intelligence (AI)-based neural image compression (NIC). Aspects of this disclosure include content-adaptive online training in NICs, such as a post-filtered content-adaptive online training NIC method with an end-to-end (E2E) optimized image coding framework based on neural networks. Neural networks (NNs) can include artificial neural networks (ANNs), such as deep neural networks (DNNs), convolutional neural networks (CNNs), etc.

In implementations, related hybrid video codecs are difficult to optimize as a whole. For example, improvements to individual modules (e.g., encoders) in a hybrid video codec may not result in coding gains for the overall performance. In a neural network-based video coding framework, different modules can be jointly optimized from input to output to improve the final objective (e.g., rate-distortion performance, such as the rate-distortion loss L described in the published content) by performing a learning or training process (e.g., a machine learning process), thereby producing an end-to-end optimized NIC.

An exemplary NIC framework or system can be described as follows. The NIC framework can use an input block x as input to a neural network encoder (e.g., an encoder based on a neural network such as a DNN) to compute a compressed representation (e.g., a compact representation) that can be compacted for purposes such as storage and transmission. A neural network decoder (e.g., a decoder based on a neural network such as a DNN) can use the compressed representation as input to reconstruct an output block (also called a reconstructed block). In various implementations, the input block x and the reconstructed block are in a spatial domain, and the compressed representation is in a domain different from the spatial domain. In some examples, the compressed representation is quantized and entropy-encoded.

In some examples, the NIC framework can use a variational autoencoder (VAE) architecture. In a VAE architecture, the neural network encoder can directly take the entire input block x as input. The entire input block x is then processed by a set of neural network layers, which act as black boxes to compute a compressed representation. This compressed representation is the output of the neural network encoder. The neural network decoder can take the entire compressed representation as input. The compressed representation is then processed by another set of neural network layers, which act as another black box to compute. The reconstructed block can be optimized for rate-distortion (R-D) loss to achieve a trade-off between the distortion loss of the reconstructed block and the bit consumption R of the compact representation with a trade-off hyperparameter λ.

Neural networks (e.g., ANNs) can learn to perform tasks from examples without being programmed for a specific task. An ANN can be configured with connected nodes or artificial neurons. Connections between nodes can transmit signals from a first node to a second node (e.g., a receiving node) and can modify the signals via weights, which can be indicated by the weight coefficients of the connections. A receiving node can process the signals from the nodes that transmitted the signals to it (i.e., the input signals to the receiving node) and then generate an output signal by applying a function to the input signals. This function can be a linear function. In the example, the output signal is a weighted sum of the input signals. In the example, the output signal is also modified by a bias, which can be indicated by a bias term, so the output signal is the sum of the weighted sum of the bias and the input signals. The function can include nonlinear operations, such as a weighted sum of the input signals or a sum of the bias and the weighted sum. The output signal can be sent to nodes connected to the receiving node (downstream nodes). An ANN can be represented or configured via parameters (e.g., the weights and/or biases of the connections). The weights and/or biases can be obtained by training the ANN using examples where the weights and/or biases can be iteratively adjusted. A trained ANN configured with defined weights and/or defined biases can be used to perform tasks.

Nodes in an ANN can be organized in any suitable architecture. In various implementations, nodes in an ANN are organized into layers, including an input layer that receives input signals into the ANN and an output layer that outputs signals from the ANN. In some implementations, the ANN also includes layers between the input and output layers, such as hidden layers. Different layers can perform different kinds of transformations on their respective inputs. Signals can be transmitted from the input layer to the output layer.

An ANN with multiple layers between its input and output layers can be called a DNN. In implementations, a DNN is a feedforward network where data flows from the input layer to the output layer without loopback. In an example, a DNN is a fully connected network where each node in one layer is connected to all nodes in the next layer. In implementations, a DNN is a recurrent neural network (RNN) where data can flow in any direction. In implementations, a DNN is a CNN.

A CNN may include an input layer, an output layer, and hidden layers between the input and output layers. Hidden layers may include convolutional layers that perform convolutions, such as two-dimensional (2D) convolutions (e.g., used in an encoder). In implementations, the 2D convolution performed in the convolutional layer lies between the convolutional kernel (also called a filter or channel, e.g., a 5×5 matrix) and the input signal to the convolutional layer (e.g., a 2D matrix, such as a 2D block, a 256×256 matrix). In various examples, the dimension of the convolutional kernel (e.g., 5×5) is smaller than the dimension of the input signal (e.g., 256×256). Therefore, the portion of the input signal (e.g., a 256×256 matrix) covered by the convolutional kernel (e.g., a 5×5 region) is smaller than the region of the input signal (e.g., a 256×256 region), and can therefore be referred to as the receptive field in the corresponding node of the next layer.

During convolution, the dot product between the convolution kernel and the corresponding receptive field in the input signal is calculated. Therefore, each element of the convolution kernel is a weight applied to the corresponding sample in the receptive field, and thus the convolution kernel comprises weights. For example, a convolution kernel represented by a 5×5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolutional layer, and the output signal is based on the sum of the dot product and the bias.

A convolutional kernel can move along the input signal (e.g., a 2D matrix) by a size called stride, so the convolution operation generates feature maps or activation maps (e.g., another 2D matrix), which in turn contribute to the input of the next layer in a CNN. For example, the input signal is a 2D block with 256×256 samples, and the stride is 2 samples (e.g., stride 2). For a stride of 2, the convolutional kernel moves 2 samples along the X direction (e.g., horizontal) and/or the Y direction (e.g., vertical).

Multiple convolutional kernels can be applied to the input signal within the same convolutional layer to generate multiple feature maps, each representing a specific feature of the input signal. Generally, a convolutional layer has N channels (i.e., N kernels), each kernel has M×M samples, and the stride S can be specified as convolution: MxM cN sS. For example, a convolutional layer with 192 channels, each kernel with 5×5 samples, and a stride 2 is specified as Conv: 5x5 c192 s2. Hidden layers can include deconvolutional layers that perform deconvolution (e.g., 2D deconvolution) (e.g., used in decoders). Deconvolution is the inverse of convolution. A deconvolutional layer with 192 channels, each deconvolution kernel with 5×5 samples, and a stride 2 is specified as deconvolution: 5x5 c192 s2.

In various implementations, CNNs offer the following advantages. The number of learnable parameters (i.e., parameters to be trained) in a CNN can be significantly smaller than the number of learnable parameters in a DNN (e.g., a feedforward DNN). In a CNN, a relatively large number of nodes can share the same filters (e.g., the same weights) and the same biases (if biases are used), thus reducing memory footprint, as a single vector of a single bias and weights can be used across all receptive domains sharing the same filters. For example, for an input signal with 100×100 samples, a convolutional layer with a kernel of 5×5 samples has 25 learnable parameters (e.g., weights). If biases are used, one channel uses 26 learnable parameters (e.g., 25 weights and one bias). If the convolutional layer has N channels, the total number of learnable parameters is 26×N. On the other hand, for a fully connected layer in a DNN, 100×100 (i.e., 10000) weights are used for each node in the next layer. If the next layer has L nodes, the total number of learnable parameters is 10000×L.

CNNs can also include one or more other layers, such as pooling layers, fully connected layers that connect each node in one layer to each node in another layer, normalization layers, etc. The layers in a CNN can be arranged in any suitable order and with any suitable architecture (e.g., feedforward architecture, recurrent architecture). In the example, convolutional layers are followed by other layers, such as pooling layers, fully connected layers, normalization layers, etc.

Pooling layers can be used to reduce the dimensionality of data by combining the outputs from multiple nodes in one layer into a single node in the next layer. The pooling operation of a pooling layer with a feature map as input is described below. This description can be appropriately applied to other input signals. The feature map can be divided into sub-regions (e.g., rectangular sub-regions), and the features in each sub-region can be independently downsampled (or pooled) to a single value, for example, by taking the average value in the average pooling or the maximum value in the max pooling.

Pooling layers can perform pooling operations, such as local pooling, global pooling, max pooling, and average pooling. Pooling is a form of non-linear downsampling. Local pooling combines a small number of nodes in the feature map (e.g., a local cluster of nodes, such as 2×2 nodes). Global pooling can combine, for example, all nodes in the feature map.

Pooling layers can reduce the size of the representation, thereby reducing the number of parameters, memory footprint, and computational cost in a CNN. In the example, pooling layers are inserted between consecutive convolutional layers in the CNN. In the example, the pooling layer is followed by an activation function, such as a Rectified Linear Unit (ReLU) layer. In the example, pooling layers are omitted between consecutive convolutional layers in the CNN.

The normalization layer can be ReLU, Leaked ReLU, Generalized Division Normalization (GDN), Inverse GDN (IGDN), etc. ReLU can apply a non-saturating activation function to remove negative values from the input signal (e.g., feature map) by setting negative values to zero. For negative values, Leaked ReLU can have a small slope (e.g., 0.01) instead of a flat slope (e.g., 0). Therefore, if the value x is greater than 0, the output from Leaked ReLU is x. Otherwise, the output from Leaked ReLU is the value x multiplied by a small slope (e.g., 0.01). In the example, the slope is determined before training and therefore not learned during training.

In NN-based image compression methods (such as DNN-based or CNN-based image compression methods), block-based or per-block encoding mechanisms are effective for compressing images using DNN-based video coding standards such as FVC, rather than directly encoding the entire image. The entire image can be divided into blocks of the same (or different) size, and these blocks can be compressed individually. In implementations, the image can be divided into blocks of equal or unequal size. The divided blocks, rather than the image itself, can be compressed. Figure 9A illustrates an example of per-block image encoding according to an implementation of this disclosure. The image (980) can be divided into blocks, such as blocks (981) to (996). For example, blocks (981) to (996) can be compressed according to the scan order. In the example shown in Figure 9A, blocks (981) to (989) have already been compressed, and blocks (990) to (996) will be compressed.

In this implementation, the image is treated as a block, where a block is the entire image, and the image is compressed without being segmented into blocks. The entire image can be the input to an E2E NIC framework.

Figure 9B illustrates an exemplary NIC framework (900) (e.g., a NIC system) according to an embodiment of this disclosure. The NIC framework (900) may be based on neural networks, such as DNNs and/or CNNs. The NIC framework (900) may be used to compress (e.g., encode) blocks and decompress (e.g., decode or reconstruct) compressed blocks (e.g., encoded blocks). The NIC framework (900) may include two sub-neural networks implemented using neural networks, a first sub-NN (951) and a second sub-NN.

(952).

The first sub-NN (951) can be analogous to an autoencoder and can be trained to generate compressed blocks of input block x and decompress the compressed blocks (i.e., encoded blocks) to obtain reconstructed blocks. The first sub-NN (951) can include multiple components (or modules), such as a master encoder neural network (or master encoder network) (911), a quantizer (912), an entropy encoder (913), an entropy decoder (914), and a master decoder neural network (or master encoder network) (915). Referring to Figure 9B, the master encoder network (911) can generate a latent or latent representation y from the input block x (e.g., the block to be compressed or encoded). In the example, a CNN is used to implement the master encoder network (911). The relationship between the latent representation y and the input block x can be described using Equation 2.

y = f ₁ (x; θ ₁ ) Equation 2

Here, parameter _θ1 represents the following parameters: such as the weights used in the convolution kernels in the master encoder network (911) and the bias (if a bias is used in the master encoder network (911)).

The latent representation y can be quantized using a quantizer (912) to generate a quantized latent. For example, the quantized latent can be compressed using lossless compression by an entropy encoder (913) to generate a compressed block (e.g., an encoded block) (931) as a compressed representation of the input block x. The entropy encoder (913) can use entropy coding techniques such as Huffman coding, arithmetic coding, etc. In the example, the entropy encoder (913) uses arithmetic coding and is an arithmetic encoder. In the example, the encoded block (931) is transmitted in an encoded bitstream.

The encoded block (931) can be decompressed (e.g., entropy decoding) by the entropy decoder (914) to generate an output. The entropy decoder (914) can use an entropy coding technique such as Huffman coding, arithmetic coding, etc., corresponding to the entropy coding technique used in the entropy encoder (913). In the example, the entropy decoder (914) uses arithmetic decoding and is an arithmetic decoder. In the example, lossless compression is used in the entropy encoder (913), lossless decompression is used in the entropy decoder (914), and noise such as that generated due to the transmission of the encoded block (931) can be ignored. The output from the entropy decoder (914) is a quantized latent...

The master decoder network (915) can decode the quantized latent to generate reconstructed blocks. In the example, a CNN is used to implement the master decoder network (915). The relationship between the reconstructed blocks (i.e., the output of the master decoder network (915)) and the quantized latent (i.e., the input of the master decoder network (915)) can be described by Equation 3.

Here, parameter _θ2 represents parameters such as the weights used in the convolutional kernels in the main decoder network (915) and the bias (if a bias is used in the main decoder network (915)). Thus, the first sub-NN (951) can compress (e.g., encode) the input block x to obtain the encoded block (931), and decompress (e.g., decode) the encoded block (931) to obtain the reconstructed block. Due to the quantization loss introduced by the quantizer (912), the reconstructed block may differ from the input block x.

The second sub-NN (952) can be trained on a quantized latent entropy model (e.g., a prior probability model) for entropy encoding. Thus, the entropy model can be a conditional entropy model dependent on the input block x, such as a Gaussian mixture model (GMM) or a Gaussian scaling model (GSM). The second sub-NN (952) can include a context model NN (916), an entropy parameter NN (917), a super encoder (921), a quantizer (922), an entropy encoder (923), an entropy decoder (924), and a super decoder (925). The entropy model used in the context model NN (916) can be an autoregressive model about the latent (e.g., a quantized latent). In the example, the super encoder (921), quantizer (922), entropy encoder (923), entropy decoder (924), and super decoder (925) form a super neural network (e.g., a super-prior NN). The super neural network can represent information useful for correcting context-based predictions. Data from the context model NN (916) and the super neural network can be combined using the entropy parameter NN (917). The entropy parameter NN(917) can generate parameters such as the mean and scale parameters for entropy models such as conditional Gaussian entropy models (e.g., GMM).

Referring to Figure 9B, on the encoder side, the quantized latent from the quantizer (912) is fed into the context model NN (916). On the decoder side, the quantized latent from the entropy decoder (914) is fed into the context model NN (916). The context model NN (916) can be implemented using a neural network such as a CNN. The context model NN (916) can generate an output o _cm,i based on a context, which is a quantized latent context available to the context model NN (916), which may include previously quantized latents on the encoder side or previously entropy-decoded quantized latents on the decoder side. The relationship between the output o _cm,i of the context model NN (916) and the input (e.g., ) can be described using Equation 4.

Here, parameter _θ3 represents parameters such as the weights used in the convolution kernels of the context model NN(916) and the bias (if a bias is used in the context model NN(916)).

The outputs o _{cm, i} from the context model NN (916) and the output o _hc from the super decoder (925) are fed into the entropy parameter NN (917) to generate the output o _ep . The entropy parameter NN (917) can be implemented using a neural network such as CNN. The relationship between the output o _ep of the entropy parameter NN (917) and the inputs (e.g., o _{cm, i} and o _hc ) can be described by Equation 5.

o _ep = f ₄ (o _{cm, i} , o _hc ; θ ₄ ) Equation 5

Here, parameter _θ4 represents parameters such as the weights used in the convolutional kernel of the entropy parameter NN (917) and the bias (if a bias is used in the entropy parameter NN (917)). The output _oep of the entropy parameter NN (917) can be used to determine (e.g., adjust) the entropy model, and thus the adjusted entropy model can depend on the input block x, for example, via the output _ohc from the superdecoder (925). In the example, the output _Oep includes parameters for adjusting the entropy model (e.g., GMM), such as the mean and scale parameters. Referring to FIG9B, the entropy encoder (913) and entropy decoder (914) can use the entropy model (e.g., the conditional entropy model) in entropy encoding and entropy decoding, respectively.

The second sub-NN (952) can be described below. The latent y can be fed into the super encoder (921) to generate the hyper-latest z. In the example, the super encoder (921) is implemented using a neural network such as a CNN. The relationship between the hyper-latest z and the latent y can be described using Equation 6.

z = _f5 (y; _θ5 ) Equation 6

Here, parameter _θ5 represents parameters such as the weights used in the convolution kernel of the super encoder (921) and the bias (if a bias is used in the super encoder (921)).

The hyperpotential z is quantized by a quantizer (922) to generate a quantized potential. For example, the quantized potential can be compressed by an entropy encoder (923) using lossless compression to generate side information such as coded bits (932) from a hyperneural network. The entropy encoder (923) can use entropy coding techniques such as Huffman coding, arithmetic coding, etc. In the example, the entropy encoder (923) uses arithmetic coding and is an arithmetic encoder. In the example, side information such as coded bits (932) can be transmitted in the coded bit stream, for example, along with coded blocks (931).

Side information, such as encoded bits (932), can be decompressed (e.g., entropy decoding) by an entropy decoder (924) to generate an output. The entropy decoder (924) can use entropy coding techniques such as Huffman coding, arithmetic coding, etc. In the example, the entropy decoder (924) uses arithmetic decoding and is an arithmetic decoder. In the example, lossless compression is used in the entropy encoder (923), lossless decompression is used in the entropy decoder (924), and noise such as that caused by the transmission of side information can be ignored. The output from the entropy decoder (924) can be a quantized latent superdecoder (925) that can decode the quantized latent to generate an output o _hc . The relationship between the output o _hc and the quantized latent can be described by Equation 7.

Here, parameter _θ6 represents parameters such as the weights used in the convolution kernels in the superdecoder (925) and the bias (if a bias is used in the superdecoder (925)).

As described above, compressed bits or encoded bits (932) can be added to the encoded bitstream as side information, which enables the entropy decoder (914) to use a conditional entropy model. Therefore, the entropy model can be block-dependent and spatially adaptive, and thus more accurate than the fixed entropy model.

The NIC framework (900) can be appropriately adapted to, for example, omit one or more components shown in FIG. 9B, modify one or more components shown in FIG. 9B, and/or include one or more components not shown in FIG. 9B. In the example, the NIC framework using the fixed entropy model includes a first sub-NN (951) but excludes a second sub-NN (952). In the example, the NIC framework includes all components of the NIC framework (900) except for the entropy encoder (923) and the entropy decoder (924).

In implementation, one or more components of the NIC framework (900) shown in Figure 9B are implemented using neural networks such as CNNs. Each NN-based component in the NIC framework (e.g., the NIC framework (900)) (e.g., the master encoder network (911), the master decoder network (915), the context model NN (916), the entropy parameter NN (917), the super encoder (921), or the super decoder (925)) may include any suitable architecture (e.g., with any suitable combination of layers), any suitable type of parameters (e.g., weights, biases, combinations of weights and biases, etc.), and any suitable number of parameters.

In the implementation, the main encoder network (911), the main decoder network (915), the context model NN (916), the entropy parameter NN (917), the super encoder (921), and the super decoder (925) are implemented using the corresponding CNN.

Figure 10 illustrates an exemplary CNN of a master encoder network (911) according to an embodiment of the present disclosure. For example, the master encoder network (911) includes four sets of layers, each set of layers including a 5×5c192 s2 convolutional layer followed by a GDN layer. One or more layers shown in Figure 10 may be modified and/or omitted. Additional layers may be added to the master encoder network (911).

Figure 11 illustrates an exemplary CNN of a master decoder network (915) according to an embodiment of the present disclosure. For example, the master decoder network (915) includes three sets of layers, each set including a 5x5 c192s2 deconvolutional layer followed by an IGDN layer. Furthermore, the three sets of layers are followed by a 5x5 c3s2 deconvolutional layer also followed by an IGDN layer. One or more layers shown in Figure 11 may be modified and/or omitted. Additional layers may be added to the master decoder network (915).

Figure 12 illustrates an exemplary CNN of a super encoder (921) according to an embodiment of the present disclosure. For example, the super encoder (921) includes a 3x3 c192 s1 convolutional layer followed by leaky ReLU, a 5x5 c192 s2 convolutional layer followed by leaky ReLU, and a 5x5 c192 s2 convolutional layer. One or more layers shown in Figure 12 may be modified and/or omitted. Additional layers may be added to the super encoder (921).

Figure 13 illustrates an exemplary CNN of a superdecoder (925) according to an embodiment of the present disclosure. For example, the superdecoder (925) includes a 5x5 c192 s2 deconvolutional layer followed by leaky ReLU, a 5x5 c288 s2 deconvolutional layer followed by leaky ReLU, and a 3x3 c384 s1 deconvolutional layer. One or more layers shown in Figure 13 may be modified and/or omitted. Additional layers may be added to the superdecoder (925).

Figure 14 illustrates an exemplary CNN of a context model NN (916) according to an embodiment of the present disclosure. For example, the context model NN (916) includes a masked convolution 5x5 c384 s1 for context prediction, and thus the context in Equation 4 includes a finite context (e.g., a 5×5 convolutional kernel). The convolutional layers in Figure 14 can be modified. Additional layers can be added to the context model NN (916).

Figure 15 illustrates an exemplary CNN with an entropy parameter NN (917) according to an embodiment of the present disclosure. For example, the entropy parameter NN (917) includes a convolutional layer 1x1 c640 s1 followed by a leaky ReLU, a convolutional layer 1x1 c512 s1 followed by a leaky ReLU, and a convolutional layer 1x1 c384 s1. One or more layers shown in Figure 15 may be modified and/or omitted. Additional layers may be added to the entropy parameter NN (917).

As described with reference to Figures 10 to 15, the NIC framework (900) can be implemented using a CNN. The NIC framework (900) can be appropriately adapted to enable the implementation of one or more components of the NIC framework (900) (e.g., (911), (915), (916), (917), (921), and/or (925)) using any suitable type of neural network (e.g., CNN-based or non-CNN-based neural networks). One or more other components of the NIC framework (900) can be implemented using neural networks.

The NIC framework (900), which includes neural networks (e.g., CNNs), can be trained to learn the parameters used in the neural networks. For example, when using a CNN, parameters represented by _θ1 to _θ6 can be learned during training, such as the weights and biases used in the convolutional kernels of the master encoder network (911) (if biases are used in the master encoder network (911)), the weights and biases used in the convolutional kernels of the master decoder network (915) (if biases are used in the master decoder network (915)), the weights and biases used in the convolutional kernels of the super encoder (921) (if biases are used in the super encoder (921)), the weights and biases used in the convolutional kernels of the super decoder (925) (if biases are used in the super decoder (925)), the weights and biases used in the convolutional kernels of the context model NN (916) (if biases are used in the context model NN (916)), and the weights and biases used in the convolutional kernels of the entropy parameter NN (917) (if biases are used in the entropy parameter NN (917)).

In the example, referring to Figure 10, the master encoder network (911) includes four convolutional layers, each with a 5×5 kernel and 192 channels. Therefore, the number of weights used in the convolutional kernels of the master encoder network (911) is 19200 (i.e., 4×5×5×192). The parameters used in the master encoder network (911) include 19200 weights and optional biases. Additional parameters may be included when biases and/or additional neural networks are used in the master encoder network (911).

Referring to Figure 9B, the NIC framework (900) includes at least one component or module built on a neural network. The at least one component may include one or more of a master encoder network (911), a master decoder network (915), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The at least one component can be trained individually. In the example, the training process is used to learn the parameters of each component separately. The at least one component can be jointly trained as a group. In the example, the training process is used to jointly learn the parameters of a subset of at least one component. In the example, the training process is used to learn the parameters of all at least one component, and is therefore referred to as E2E optimization.

During the training of one or more components in the NIC framework (900), the weights (or weight coefficients) of one or more components can be initialized. In the example, the weights are initialized based on the pre-trained corresponding neural network model (e.g., DNN model, CNN model). In the example, the weights are initialized by setting them to random numbers.

For example, after initializing the weights, a set of training blocks can be used to train one or more components. The set of training blocks can include any suitable blocks of any appropriate size. In some examples, the set of training blocks includes blocks from the original image, natural image, computer-generated image, etc., in the spatial domain. In some examples, the set of training blocks includes blocks from residual blocks or residual images that have residual data in the spatial domain. The residual data can be computed by a residual calculator (e.g., residual calculator (723)). In some examples, the original image and/or the residual image including the residual data can be directly used to train a neural network in the NIC framework (e.g., NIC framework (900)). Therefore, the original image, the residual image, the blocks from the original image, and/or the blocks from the residual image can be used to train a neural network in the NIC framework.

For simplicity, training blocks are used as an example to describe the following training process. This description can be appropriately adapted to training images. Training block t in the set of training blocks can be encoded using the encoding process in Figure 9B to generate a compressed representation (e.g., encoded information to a bitstream). The encoded information can be decoded using the decoding process described in Figure 9B to compute and reconstruct the reconstructed block.

For the NIC framework (900), two competing objectives, such as reconstruction quality and bit consumption, are balanced. A quality loss function (e.g., distortion or distortion loss) can be used to indicate reconstruction quality, such as the difference between the reconstructed block (e.g., the reconstructed block) and the original block (e.g., the training block t). A rate (or rate loss) R can be used to indicate the bit consumption of the compressed representation. In the example, the rate loss R also includes, for example, side information used when determining the context model.

For neural image compression, a differentiable approximation of quantization can be used in E2E optimization. In various examples, noise injection is used to simulate quantization during the training process of neural network-based image compression. Thus, quantization is simulated by noise injection rather than performed by a quantizer (e.g., quantizer (912)). Therefore, training using noise injection can approximate the quantization error in a varied manner. A bit-per-pixel (BPP) estimator can be used to simulate an entropy encoder. Thus, entropy encoding is simulated by the BPP estimator rather than performed by an entropy encoder (e.g., (913)) and an entropy decoder (e.g., (914)). Therefore, the rate loss R in the loss function L shown in Equation 1 during training can be estimated, for example, based on noise injection and the BPP estimator. Generally, a higher rate R allows for a lower distortion D, while a lower rate R results in a higher distortion D. Therefore, the tradeoff hyperparameter λ in Equation 1 can be used to optimize the joint R-D loss L, where L, as the sum of λD and R, can be optimized. The training process can be used to tune the parameters of one or more components (e.g., (911)(915)) of the NIC framework (900) such that the joint R-D loss L is minimized or optimized. In some examples, the tradeoff hyperparameter λ can be used to optimize the joint rate-distortion (R-D) loss according to the following formula:

Here, E measures the distortion of the decoded block residual compared to the original block residual before encoding, and it serves as the regularization loss for both the residual encoding/decoding DNN and the encoding/decoding DNN. β is a hyperparameter that balances the importance of the regularization loss.

Various models can be used to determine the distortion loss D and the rate loss R, and thus the joint R-D loss L in Equation 1. In the example, the distortion loss is expressed as the peak signal-to-noise ratio (PSNR), which is a measure based on mean squared error, multi-scale structural similarity (MS-SSIM) quality index, a weighted combination of PSNR and MS-SSIM, etc.

In this example, the goal of the training process is to train an encoding neural network (e.g., an encoding DNN) for a video encoder to be used on the encoder side and a decoding neural network (e.g., a decoding DNN) for a video decoder to be used on the decoder side. Referring to FIG9B, the encoding neural network may include a master encoder network (911), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The decoding neural network may include a master decoder network (915), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The video encoder and/or video decoder may include other components based on and/or not based on NN.

The NIC framework can be trained in an E2E manner (e.g., NIC framework (900)). In the example, the encoding neural network and the decoding neural network are jointly updated in an E2E manner based on backpropagation gradients during training.

After the parameters of the neural network in the NIC framework (900) are trained, one or more components of the NIC framework (900) can be used to encode and/or decode blocks. In an embodiment, on the encoder side, the video encoder is configured to encode the input block x into a coded block (931) to be transmitted in the bitstream. The video encoder may include multiple components in the NIC framework (900). In an embodiment, on the decoder side, a corresponding video decoder is configured to decode the coded block (931) in the bitstream into a reconstructed block. The video decoder may include multiple components in the NIC framework (900).

In the example, the video encoder includes all components in the NIC framework (900).

Figure 16A illustrates an exemplary video encoder (1600A) according to an embodiment of the present disclosure. For example, the video encoder (1600A) includes a master encoder network (911), a quantizer (912), an entropy encoder (913), and a second sub-NN (952) as described with reference to Figure 9B. Figure 16B illustrates an exemplary video decoder (1600B) according to an embodiment of the present disclosure. The video decoder (1600B) may correspond to the video encoder (1600A). The video decoder (1600B) may include a master decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924), and a super decoder (925). Referring to Figures 16A and 16B, on the encoder side, the video encoder (1600A) may generate coded blocks (931) and coded bits (932) to be transmitted in the bitstream. On the decoder side, the video decoder (1600B) can receive the coded block (931) and the coded bits (932) and decode the coded block (931) and the coded bits (932).

Figures 17 and 18 illustrate exemplary video encoders (1700) and corresponding video decoders (1800) according to embodiments of the present disclosure. Referring to Figure 17, the encoder (1700) includes a master encoder network (911), a quantizer (912), and an entropy encoder (913). An example of the master encoder network (911), quantizer (912), and entropy encoder (913) is described with reference to Figure 9B. Referring to Figure 18, the video decoder (1800) includes a master decoder network (915) and an entropy decoder (914). An example of the master decoder network (915) and entropy decoder (914) is described with reference to Figure 9B. Referring to Figures 17 and 18, the video encoder (1700) can generate coded blocks (931) to be included in the bitstream. The video decoder (1800) can receive the coded blocks (931) and decode the coded blocks (931).

Video coding techniques may include filtering operations that reduce artifacts (e.g., blockiness) by applying filtering to reconstructed samples. Deblocking filtering can be used in such filtering operations, where block boundaries (e.g., boundary regions) between adjacent blocks can be filtered to achieve a smoother transition of sample values from one block to another.

In some relevant examples (e.g., HEVC), deblocking or deblocking filtering can be applied to samples adjacent to block boundaries. Deblocking filtering can be performed on each CU in the same order as the decoding process. For example, deblocking can be performed by first horizontally filtering the vertical boundaries of the image, and then vertically filtering the horizontal boundaries. Filtering can be applied to both the luma and chroma components of the 8×8 block boundaries that are determined to be filtered. In this example, the 4×4 block boundaries are not processed to reduce complexity.

Boundary strength (BS) can be used to indicate the degree or intensity of deblocking filtering. In this implementation, a BS value of 2 indicates strong filtering, a BS value of 1 indicates weak filtering, and a BS value of 0 indicates no deblocking filtering.

Blocks of an image can be reconstructed from an encoded video bitstream using any suitable method, such as the embodiments described in this disclosure. For example, blocks can be reconstructed using a video decoder (e.g., (1600B), (1800)) that includes a neural network (e.g., CNN). In some examples, a non-NN-based video decoder is used to reconstruct blocks. One or more deblocking methods (also called deblocking, deblocking processing, deblocking filtering) can be used to reduce artifacts between blocks, such as artifacts in the boundary regions of adjacent reconstructed blocks. Deblocking can reduce block artifacts or block effects between adjacent reconstructed blocks, for example, caused by independent quantization in adjacent reconstructed blocks. Deblocking can be performed on the boundary regions of adjacent reconstructed blocks using a deblocking module. To reduce artifacts between blocks, an NN-based deblocking model can be used. NN-based deblocking models can be DNN-based deblocking models, CNN-based deblocking models, etc. NN-based deblocking models can be implemented using NNs including DNNs, CNNs, etc. (e.g., deblocking NNs).

Figures 19A to 19C illustrate an exemplary deblocking process (1900) according to an embodiment of the present disclosure. Referring to Figure 19A, an image (1901) may include multiple blocks (1911) to (1914). For simplicity, four blocks (1911) to (1914) of equal size are shown in Figure 19A. Typically, an image can be divided into any suitable number of blocks, and the block sizes may be different or the same, and the description may be adapted accordingly. In some examples, deblocking can be used to process regions containing artifacts, such as those caused by dividing the image into blocks.

In the example, blocks (1911) through (1914) are reconstructed blocks from the master decoder network (915). Image (1901) is the reconstructed image. In some examples, the reconstructed image (1901) includes residual data.

The first two adjacent reconstruction blocks of reconstruction blocks (1911) to (1914) may include blocks (1911) and (1913) separated by a first shared boundary (1941). Blocks (1911) and (1913) may include a boundary region A having samples on both sides of the first shared boundary (1941). Referring to Figures 19A to 19B, boundary region A may include sub-boundary regions A1 and A2 located in blocks (1911) and (1913), respectively. The first two adjacent reconstruction blocks (1911) and (1913) may include a boundary region A having samples on both sides of the first shared boundary (1941) and non-boundary regions (1921) and (1923) outside boundary region A.

Two adjacent reconstruction blocks in reconstruction blocks (1911) to (1914) may include blocks (1912) and (1914) separated by a second shared boundary (1942). Blocks (1912) and (1914) may include boundary regions B having samples on both sides of the second shared boundary (1942). Referring to Figures 19A to 19B, boundary regions B may include sub-boundary regions B1 and B2 located in blocks (1912) and (1914), respectively. Blocks (1912) and (1914) may include boundary regions B having samples on both sides of the shared boundary (1942) and non-boundary regions (1922) and (1924) outside boundary regions B.

Two adjacent reconstruction blocks in reconstruction blocks (1911) to (1914) may include blocks (1911) and (1912) separated by a shared boundary (1943). Blocks (1911) and (1912) may include a boundary region C having samples on both sides of the shared boundary (1943). Referring to Figures 19A to 19B, the boundary region C may include sub-boundary regions C1 and C2 located in blocks (1911) and (1912), respectively. Blocks (1911) and (1912) may include a boundary region C having samples on both sides of the shared boundary (1943) and non-boundary regions (1921) and (1922) outside the boundary region C.

Two adjacent reconstruction blocks in reconstruction blocks (1911) to (1914) may include blocks (1913) and (1914) separated by a shared boundary (1944). Blocks (1913) and (1914) may include a boundary region D having samples on both sides of the shared boundary (1944). Referring to Figures 19A to 19B, the boundary region D may include sub-boundary regions D1 and D2 located in blocks (1913) and (1914), respectively. Blocks (1913) and (1914) may include the boundary region D having samples on both sides of the shared boundary (1944) and non-boundary regions (1923) and (1924) outside the boundary region D.

Referring to FIG19A, the image (1901) may include boundary regions A to D and non-boundary regions (1921) to (1924) outside the boundary regions A to D.

Sub-boundary regions A1 to D1 and A2 to D2 (and boundary regions A to D) can have any suitable size (e.g., width and/or height). In the embodiment shown in Figure 19A, sub-boundary regions A1, A2, B1, and B2 have the same m×n size, where n is the width of blocks (1911) to (1914), and m is the height of sub-boundary regions A1, A2, B1, and B2. Both m and n are positive integers. In the example, m is four pixels or four samples. Therefore, boundary regions A and B have the same 2m×n size. Sub-boundary regions C1, C2, D1, and D2 have the same n×m size, where n is the height of blocks (1911) to (1914), and m is the width of sub-boundary regions C1, C2, D1, and D2. Therefore, boundary regions C and D have the same n×2m size. As mentioned above, sub-boundary regions and boundary regions can have different sizes, such as different widths, different heights, etc. For example, sub-boundary regions A1 and A2 can have different heights. In this example, sub-boundary regions C1 and C2 can have different widths. Boundary regions A and B can have different widths. Boundary regions C and D can have different heights.

In some examples, the boundary region has a shape that differs from a rectangular or square shape. The boundary region can have an irregular shape that depends on the position, shape, and/or size of the adjacent blocks.

Referring to Figures 19A and 19B, boundary region A includes m rows of samples (e.g., m columns of samples) in block (1911) starting from the first shared boundary (1941) and m rows of samples (e.g., m columns of samples) in block (1913) starting from the first shared boundary (1941). Boundary region C includes m columns of samples (e.g., m columns of samples) in block (1911) starting from the shared boundary (1943) and m columns of samples (e.g., m columns of samples) in block (1912) starting from the shared boundary (1943).

Deblocking neural networks (1930), such as those based on DNNs, CNNs, or any suitable neural network, can be used to perform deblocking on one or more of the boundary regions A through D. In the example, a CNN comprising one or more convolutional layers is used to implement the deblocking neural network (1930). The deblocking neural network (1930) may include additional layers as described in this disclosure, such as pooling layers, fully connected layers, normalization layers, etc. The layers in the deblocking neural network (1930) can be arranged in any suitable order and with any suitable architecture (e.g., feedforward architecture, recurrent architecture). In the example, convolutional layers are followed by other layers, such as pooling layers, fully connected layers, normalization layers, etc.

Deblocking can be performed on boundary regions A through D using a deblocking NN (1930). One or more of the boundary regions A through D contain artifacts. These artifacts may be caused by corresponding adjacent blocks. One or more of the boundary regions A through D can be sent to the deblocking NN (1930) to reduce artifacts. Therefore, the input to the deblocking NN (1930) includes one or more of the boundary regions A through D, and the output from the deblocking NN (1930) includes one or more of the deblocked boundary regions A through D.

Referring to Figure 19B, boundary regions A to D include artifacts caused by their respective neighboring blocks. Boundary regions A to D can be sent to the deblocking NN (1930) to reduce artifacts. The output from the deblocking NN (1930) includes the deblocked boundary regions A' to D'. In the example, the artifacts in the deblocked boundary regions A' to D' are reduced compared to the artifacts in the boundary regions A to D.

Referring to Figures 19B and 19C, the boundary regions A to D in image (1901) are updated, for example, by being replaced with deblocked boundary regions A’ to D’. Thus, image (1950) (e.g., a deblocked image) is generated, and image (1950) (e.g., a deblocked image) includes the deblocked boundary regions A’ to D’ and non-boundary regions (1921) to (1924).

In some examples, an image (1901) with boundary regions A through D is treated as a single boundary region. Image (1901) is sent to a deblocking neural network (1930) to reduce blocking artifacts. The output from the deblocking neural network (1930) includes a deblocked image (e.g., image (1950)), which may include deblocked boundary regions (e.g., A’ through D’) and non-boundary regions (1921) through (1924).

In some examples, adjacent blocks that include boundary regions (e.g., blocks (1911) to (1912) that include boundary region C) are sent to the deblocking NN (1930). The output from the deblocking NN (1930) includes deblocked blocks, which may include deblocked boundary regions (e.g., C’) and non-boundary regions (1921) to (1922).

One or more samples can exist in multiple boundary regions. When multiple boundary regions are replaced by corresponding deblocked boundary regions, the value of one shared sample among one or more shared samples can be determined using any suitable method.

Referring to Figure 19A, sample S is located in boundary regions A and C. After obtaining boundary regions A' and C', the value of sample S can be obtained using the following method. In the example, boundary region A is replaced by the deblocked boundary region A', and subsequently, boundary region C is replaced by the deblocked boundary region C'. Therefore, the value of sample S is determined by the value of sample S in the deblocked boundary region C'.

In the example, boundary region C is replaced by the deblocked boundary region C', and subsequently, boundary region A is replaced by the deblocked boundary region A'. Therefore, the value of sample S is determined by the value of sample S in the deblocked boundary region A'.

In the example, the value of sample S is determined by the average (e.g., weighted average) of the values of sample S in the deblocked boundary region A’ and the values of sample S in the deblocked boundary region C’.

Boundary regions may include samples from more than two blocks. Figure 20 illustrates an exemplary deblocking process according to an embodiment of the present disclosure, wherein boundary regions may include samples from more than two blocks. A single boundary region AB may include boundary regions A and B. Boundary region AB may include samples on both sides of a shared boundary (1941) between two adjacent reconstructed blocks (1911) and (1913), and includes samples on both sides of a shared boundary (1942) between two adjacent reconstructed blocks (1912) and (1914). A single boundary region CD may include boundary regions C and D. Boundary region CD may include samples on both sides of a shared boundary (1943) between two adjacent reconstructed blocks (1911) and (1912), and includes samples on both sides of a shared boundary (1944) between two adjacent reconstructed blocks (1913) and (1914).

A deblocking NN (such as deblocking NN (1930)) can perform deblocking on one or more boundary regions AB and CD to generate one or more deblocked boundary regions. Referring to FIG20, boundary regions AB and CD are sent to deblocking NN (1930) and deblocked boundary regions AB’ and CD’ are generated. Deblocked boundary regions AB’ and CD’ can replace boundary regions AB and CD in image (1901) to generate image (2050). Image (2050) may include deblocked boundary regions AB’ to CD’ and non-boundary regions (1921) to (1924). Image (2050) may be the same as or different from image (1950).

According to embodiments of this disclosure, a multi-model deblocking method can be used. Different deblocking models can be applied to boundary regions of different types or categories to remove artifacts. A classification module can be applied to classify the boundary regions into different categories. Any classification module can be applied. In the example, the classification module is based on a neural network (NN). In the example, the classification module is not based on an NN. The boundary region can be sent to different deblocking models according to the corresponding category.

In this implementation, the deblocking neural network (NN) comprises multiple NNs, each implemented based on a different deblocking model. For example, a classification module can determine which of the multiple NNs should be applied to the boundary region. Deblocking of the boundary region can then be performed using the determined NN. In this example, a classification module based on an NN (e.g., a classification NN)—such as a DNN, CNN, etc.—determines which of the multiple NNs to apply.

In implementation, the deblocking NN includes a single NN that implements multi-model deblocking.

Figure 21 illustrates an exemplary deblocking process (2100) based on multiple deblocking models according to an embodiment of the present disclosure. A classification module (2110) can classify boundary regions A through D into one or more categories. For example, boundary regions C through D can be classified into a first category, boundary region B into a second category, and boundary region A into a third category. Different deblocking models can be applied to boundary regions in different categories. In Figure 21, deblocking can be performed using a deblocking neural network (2130), such as multi-model deblocking based on multiple deblocking models (e.g., deblocking models 1 through L). L is a positive integer. When L is 1, the deblocking neural network (2130) includes a single deblocking model. When L is greater than 1, the deblocking neural network (2130) includes multiple deblocking models. The deblocking neural network (2130) can be implemented using a single neural network or multiple neural networks.

In the example, deblocking model 1 is applied to the boundary regions (e.g., C and D) in the first category, and deblocked boundary regions (e.g., C” and D”) are generated. Deblocking model 2 is applied to the boundary regions (e.g., B) in the second category, and deblocked boundary regions (e.g., B”) are generated. Deblocking model 3 is applied to the boundary regions (e.g., A) in the third category, and deblocked boundary regions (e.g., A”) are generated. The deblocked boundary regions A” to D” can replace the corresponding boundary regions A to D in image (1901) to generate image (2150). Image (2150) may include the deblocked boundary regions A” to D” and non-boundary regions (2121) to (2124).

Any suitable metric can be applied to classify or categorize boundary regions. In the example, boundary regions are classified based on their content. For example, boundary regions with high-frequency content (e.g., content with relatively large variance) and boundary regions with low-frequency content (e.g., content with relatively small variance) are classified into different categories corresponding to different deblocking models. Boundary regions can be classified using the intensity of artifacts within them. Multi-model deblocking methods can be applied to any suitable boundary region, such as the boundary region between two or more blocks (e.g., A, B, C, D, AB, and/or CD). The frequency of a boundary region can be determined based on the maximum difference between samples within it. In the example, the first difference is determined for samples near the first edge on the first side of a shared boundary. In the example, the second difference is determined for samples near the second edge on the second side of a shared boundary. In the example, both the first and second differences are determined.

Artifacts between blocks can be removed by applying a deblocking neural network (e.g., deblocking neural network (1930) in Figure 19B or deblocking neural network (2130) in Figure 21). In the example, samples or pixels closer to the shared boundary can be deblocked to a greater extent than samples (or pixels) farther from the shared boundary. Referring back to Figure 19A, sample S is closer to the shared boundary (1941) than sample F, therefore sample S can be deblocked to a greater extent than sample F.

Deblocking models in deblocking neural networks (e.g., deblocking NN (1930) or deblocking NN (2130)) can include one or more convolutional layers. For example, CNN-based attention mechanisms (e.g., non-local attention, squeeze-and-excitation network (SENet)), residual neural networks (ResNet) (e.g., comprising a set of CNNs or convolutional neural networks and activation functions), etc., can be used. For example, a DNN used by image super-resolution can be used by changing the output size to the same size as the input. In image super-resolution, the resolution of an image can be increased from a low resolution to a high resolution.

The above describes how to perform deblocking on boundary regions using neural networks (NNs) or other learning-based methods. In some examples, the video encoder and/or video decoder can choose between NN-based or non-NN-based deblocking methods. Selection can be made at various levels (such as slice level, image level, for a set of images, sequence level, etc.). This selection can be signaled using flags. The selection can also be inferred from the content of the boundary region.

Video encoders and/or video decoders can apply various levels of boundary strength beyond those described in the methods and implementations of this disclosure, such as adjustments to the default level of boundary strength (BS) derived from a neural network of pixels or samples. Different levels of BS can be assigned by analyzing boundary conditions and block coding features to modify (e.g., scale up or down) the default adjustment.

According to embodiments of this disclosure, one or more non-boundary regions of adjacent reconstruction blocks can be enhanced using an enhanced NN.

Deblocking neural networks (e.g., (1930) or (2130)) can be implemented using any suitable filtering method that reduces artifacts, such as those caused by blocking, and improves visual quality. Deblocking neural networks (e.g., (1930) or (2130)) can include any neural network architecture (e.g., a feedforward architecture or a recursive architecture), can include any number and any suitable combination of layers, can include one or more sub-neural networks, can include any suitable type of parameters (e.g., weights, biases, combinations of weights and biases), and can include any suitable number of parameters as described in this disclosure. In the examples, deblocking neural networks (e.g., (1930) or (2130)) are implemented using DNNs and/or CNNs.

In the example, a deblocking neural network is implemented using a CNN comprising one or more convolutional layers. The deblocking neural network may include additional layers described in this disclosure, such as pooling layers, fully connected layers, normalization layers, etc. The layers in the deblocking neural network can be arranged in any suitable order and with any suitable architecture (e.g., feedforward or recursive architecture). In the example, convolutional layers are followed by other layers, such as pooling layers, fully connected layers, normalization layers, etc. Each of the convolutional layers in the deblocking neural network may include any suitable number of channels and any suitable convolutional kernel and stride.

The deblocking neural network (NN) can be trained based on any suitable training image or training block. The training of the NN can be similar to or the same as the training of the NN described above with reference to FIG9B. In an embodiment, the training image may include a reconstructed image with artifacts due to blocking effects. In an embodiment, the training block may include a reconstructed block with artifacts due to blocking effects. In an embodiment, the NN (e.g., (1930) or (2130)) is trained using boundary regions (e.g., boundary regions A to D, AB, and/or CD) in the corresponding training block or training image that have artifacts due to blocking effects. The NN (e.g., (1930) or (2130)) can be configured with training parameters after the training process. The trained NN can perform deblocking on one or more boundary regions, as shown, for example, in FIGS. 19A to 19C, FIG. 20, FIG. 21, FIG. 22A, and FIG. 22B. Deblocking neural networks (e.g., (1930) or (2130)) can be applied to reconstruct an image (e.g., image (1901)) or reconstruct blocks (e.g., blocks (1911) to (1912)) to reduce blocking artifacts, as described, for example, with reference to Figures 19B to 19C. Deblocking neural networks (e.g., (1930) or (2130)) can be applied to boundary regions, as shown below in Figures 22A to 22B.

A deblocking neural network (e.g., (1930) or (2130)) can be trained separately to determine the parameters in the deblocking neural network. A deblocking neural network (e.g., (1930) or (2130)) can be trained as a component of the NIC framework, or trained together with the NIC framework. For example, the NIC framework (900) and the deblocking neural network (e.g., (1930) or (2130)) can be trained jointly.

In the following description, for the sake of brevity, the boundary signal may refer to a reconstructed image including a boundary region, an adjacent reconstructed block including a boundary region, or a boundary region within a reconstructed image or an adjacent reconstructed block. The boundary signal may have artifacts due to blocking effects. In this implementation, the boundary signal is not deblocked by the deblocking neural network (NN). The boundary signal may correspond to the coded signal.

Referring to Figures 9B, 19A, and 19B, the boundary signal can be based on the output (e.g., a reconstructed block) from the main decoder network (915). The encoded signal corresponding to the boundary signal can be based on the output (e.g., a encoded block) from the entropy encoder (913).

In the example, the boundary signal includes a reconstructed image containing boundary regions, such as image (1901) containing reconstructed blocks (e.g., blocks (1911) to (1914)) from the master decoder network (915), where image (1901) includes boundary regions A to D. The encoded signal corresponding to the reconstructed image may include an encoded image based on the output (e.g., encoded blocks) from the entropy encoder (913).

In the example, the boundary signal includes boundary regions (e.g., boundary regions A, B, C, D, AB, or CD) in the reconstructed image (e.g., image (1901)). The encoded signal corresponding to the boundary region may include an encoded image based on the output (e.g., a coded block) from the entropy encoder (913).

In the example, the boundary signal includes adjacent reconstructed blocks containing boundary regions, such as blocks (1911) to (1912) from the master decoder network (915). The encoded signal corresponding to the adjacent reconstructed block may include encoded blocks (e.g., adjacent encoded blocks) based on the output from the entropy encoder (913).

In the example, the boundary signals include boundary regions (e.g., boundary regions A, B, C, D, AB, or CD) in adjacent reconstructed blocks output from the master decoder network (915). The encoded signals corresponding to the boundary regions may include encoded blocks (e.g., adjacent encoded blocks) based on the output from the entropy encoder (913).

Deblocking can be applied to boundary signals (e.g., boundary regions in a reconstructed image or reconstructed block, a reconstructed image, or adjacent reconstructed blocks) to determine deblocking boundary signals (e.g., deblocking boundary regions, deblocked images, or deblocking adjacent reconstructed blocks). Deblocking can be used to reduce the R-D loss L of the deblocking boundary signals, where the R-D loss L can include the distortion loss D of the deblocking boundary signals and the rate loss R of the encoded signal corresponding to the boundary signals. In an embodiment, the rate loss R of the encoded signal remains unchanged during the deblocking process because the encoded signal (e.g., an encoded image including encoded block 931 or a encoded block including encoded block 931 in FIG. 9B) is not affected by the deblocking process. Therefore, the R-D loss L of the deblocking boundary signals can be indicated by the distortion loss D of the deblocking boundary signals, which depends on the deblocking process. Deblocking can be applied to reduce the distortion loss D of the deblocking boundary signals; for example, the distortion loss D of the deblocking boundary signals can be less than the distortion loss D of the corresponding boundary signals.

Examples of deblocking are shown with reference to Figures 9A, 9B, and 19A-19C. The input image to be encoded or compressed can be segmented into blocks (e.g., input blocks), as described with reference to Figure 9A. Referring to Figure 9B, each of the input blocks (e.g., input block x) can be processed by the NIC framework (900) to generate corresponding encoded and reconstructed blocks, respectively.

In the example, the boundary signal to be deblocked is the reconstructed image including the reconstructed block. Symbols I and II are used to represent the input image, the coded image including the coded block, and the reconstructed image including the reconstructed block, respectively. Referring to Figures 19A to 19C, in the example, the reconstructed image (e.g., image (1901)) is fed into the deblocking NN (e.g., (1930)) to obtain the deblocked image (e.g., image (1950)). According to embodiments of this disclosure, deblocking can be implemented to reduce the R-D loss shown in Equation 9.

The R-D loss corresponding to the deblocked image can include the distortion loss of the deblocked image and the rate loss of the coded image.

As mentioned above, the rate loss of the encoded image remains unchanged during deblocking. Therefore, the performance of the deblocking neural network can be determined based on the distortion loss of the deblocked image. In the example, the distortion loss of the reconstructed image is reduced to the distortion loss of the deblocked image, where the distortion loss is less than the distortion loss.

The above description may apply if the boundary signal to be deblocked includes (i) adjacent reconstructed blocks (e.g., (1911) to (1912)) or (ii) a boundary region (e.g., boundary region A).

The boundary signal to be deblocked by the deblocking neural network (NN) can correspond to the input signal (e.g., an input image or input block) to be encoded and optionally included in the encoded video bitstream. The deblocking boundary signal can be generated by the NN based on the boundary signal. If the boundary signal to be deblocked is significantly different from the training image, training block, and/or training boundary region used to train the NN, the deblocking boundary signal can correspond to a relatively poor R-D loss L or a relatively large distortion D. Therefore, aspects of this disclosure describe content-adaptive online training methods and apparatus for deblocking, for example, in block-by-block image compression. In content-adaptive online training for deblocking, one or more parameters (e.g., weights and/or biases) of the NN can be determined based on the boundary signal, which is generated based on the input signal (e.g., an input image or input block) to be compressed (e.g., encoded) and/or included in the encoded video bitstream. The input signal can include raw data or residual data. For brevity, content-adaptive online training for deblocking based on the boundary signal to be deblocked is referred to as deblocking training in this disclosure.

According to embodiments of this disclosure, deblocking information indicating one or more determined parameters of the deblocking neural network (e.g., determined weights and/or determined biases) can be encoded and optionally included in the encoded video bitstream. Therefore, the RD loss _Lp can include a rate loss or bit consumption R(p) that signals the deblocking information in the encoded video bitstream. In the example, the boundary signal is the reconstructed image and _Lp can be determined as shown in Equation 10.

The RD loss corresponding to the deblocking boundary signal can include the distortion loss of the deblocking boundary signal (e.g., the deblocked reconstructed image), the rate loss of the encoded signal (e.g., the encoded image), and the rate loss R(p). As mentioned above, the rate loss remains constant during the deblocking process, and therefore remains constant during deblocking _training . Compared to the RD loss shown in Equation 9, adding the rate loss R(p) to Equation 10 increases the RD loss, and the sum of R(p) is expressed as L_{_training}. Deblocking _training can be implemented to reduce the distortion loss of _the deblocking boundary signal such that, despite the increase in rate loss R(p), the term L_{_training} is less than λD of the corresponding reconstructed signal without a deblocking process.

According to embodiments of this disclosure, on the encoder side, the input image to be compressed or encoded can be segmented into blocks. Referring to FIG9B, for example, based on the NIC framework (900), the blocks can be encoded into encoded blocks. The encoded blocks can be processed by the NIC framework (900) to generate reconstructed blocks.

Deblocking training can be implemented, for example, on the encoder side, based on boundary signals associated with reconstructed blocks (e.g., neighboring reconstructed blocks within a reconstructed block). In the example, neighboring reconstructed blocks comprise a subset of reconstructed blocks in the reconstructed image. In the example, neighboring reconstructed blocks comprise each of the reconstructed blocks in the reconstructed image, and thus the entire reconstructed image is considered as a neighboring reconstructed block. Therefore, the boundary signals can include (i) the entire reconstructed image, such as image (1901), (ii) neighboring reconstructed blocks, such as blocks (1911) to (1912), or (iii) included in one or more boundary regions (e.g., one or more boundary regions A to D) within the reconstructed block.

According to embodiments of this disclosure, the boundary signal comprises the entire reconstructed image. Deblocking training can be implemented based on the reconstructed image, wherein one or more parameters of the deblocking neural network are determined (e.g., updated) by optimizing rate distortion performance, for example, based on an iterative update process, such as described above with reference to FIG9B. During deblocking training, the term L_{_training} can be reduced by iteratively updating one or more parameters of the deblocking neural network.

In the example, the boundary signal includes adjacent reconstructed blocks, such as blocks (1911) to (1912). Figure 22A illustrates an exemplary deblocking process according to an embodiment of the present disclosure. Adjacent reconstructed blocks (1911) to (1912) are fed into a deblocking NN (e.g., (1930)) and deblocked blocks (1911)' to (1912)' are generated, wherein the deblocked blocks (1911)' to (1912)' include deblocking boundary regions C' and non-boundary regions (1921) to (1922). In the example, the distortion loss of the reconstructed blocks (1911) to (1912) is reduced to the distortion loss of the deblocked blocks (1911)' to (1912)'. According to an embodiment of the present disclosure, deblocking training can be implemented based on adjacent reconstructed blocks, wherein one or more parameters of the deblocking NN are determined (e.g., updated) by optimizing rate distortion performance, for example, based on an iterative update process, such as described above with reference to Figure 9B. During deblocking training, _{the training} term L can be reduced by iteratively updating one or more parameters of the deblocked neural network.

In the example, the boundary signal includes one or more boundary regions, such as one or more of boundary regions A to D shown in FIG. 19A or one or more of boundary regions AB and CD shown in FIG. 20. FIG. 22B illustrates an exemplary deblocking process according to an embodiment of the present disclosure. Boundary regions A in adjacent reconstructed blocks (1911) and (1913) are fed into the deblocking NN (e.g., (1930)) to obtain deblocking boundary region A'. In the example, the distortion loss D of boundary region A is reduced to the distortion loss D of deblocking boundary region A'. According to an embodiment of the present disclosure, deblocking training can be implemented based on one or more boundary regions, wherein one or more parameters of the deblocking NN are determined (e.g., updated) by optimizing rate distortion performance, for example, based on an iterative update process, such as described above with reference to FIG. 9B. During the deblocking training process, the term L _training can be reduced by iteratively updating one or more parameters of the deblocking NN.

According to embodiments of this disclosure, the boundary signal includes a boundary region (e.g., A) in adjacent reconstructed blocks (e.g., (1911) and (1913)) within the reconstructed image (e.g., (1901)). On the encoder side, deblocking information indicating one or more parameters of the determined deblocking NN can be encoded into the video bitstream along with the corresponding coded blocks. The deblocking information may correspond to a boundary signal (e.g., a boundary region) used to determine the deblocking information in deblocking training. For brevity, unless otherwise stated, the following examples and/or embodiments use boundary regions as boundary signals. If the boundary signal includes a reconstructed image or a reconstructed block, the description may be adjusted accordingly.

On the decoder side, coded blocks can be reconstructed from the coded video bitstream. Reconstructed blocks may include adjacent reconstructed blocks (e.g., (1911) and (1913)). Deblocking information in the coded video bitstream corresponding to the boundary region can be decoded, where the deblocking information may indicate one or more parameters of the deblocking NN in the video decoder. The deblocking NN in the video decoder can be determined based on one or more parameters indicated by the deblocking information. In the example, the deblocking NN is specifically determined for the boundary region. The boundary region can be deblocked based on the determined deblocking NN corresponding to the boundary region.

Since one or more parameters of the deblocking neural network (NN) are determined based on the boundary signals (e.g., boundary regions) to be deblocked, the NN can depend on these boundary signals. Because the boundary signals are determined based on the input image or blocks to be encoded, the NN is content-adaptive to the input image or blocks to be encoded. Since deblocking training can be performed based on the boundary signals to be encoded, it can be trained online. Because deblocking training depends on and can therefore be adjusted for the boundary signals, better compression performance can be achieved by using a NN determined using deblocking training.

NN-based deblocking (e.g., DNN-based or CNN-based deblocking) can be implemented in NN-based image coding frameworks (e.g., the NIC framework (900)) or other image compression methods. Deblocking training can be used as a preprocessing step (e.g., a precoding step) to improve the compression performance of any image compression method.

Deblocking training can be independent of the image compression codec and can be implemented with any suitable type of image compression codec. The image compression codec can be based on a neural network, such as the NIC framework (900) shown in Figure 9B. The image compression codec can also be implemented without a neural network, as in some implementations shown in Figures 5 through 8.

One or more parameters may include bias terms, weight coefficients, etc. in the deblocked neural network.

In this implementation, the deblocking neural network is not configured with initial parameters, and deblocking training is implemented to generate parameters of the deblocking neural network (e.g., including one or more parameters of the deblocking neural network).

In one implementation, for example, the deblocking neural network (NN) is configured with initial parameters (e.g., initial weights and/or initial biases) before deblocking training. In another implementation, the NN is pre-trained based on a training dataset that includes training blocks, training images, and/or training boundary regions. The initial parameters may include pre-trained parameters (e.g., pre-trained weights and/or pre-trained biases). In yet another implementation, the NN is not pre-trained. The initial parameters may include randomly initialized parameters.

In implementations, one or more initial parameters in the deblocking neural network are iteratively updated during deblocking training based on boundary signals. One or more initial parameters can be updated based on one or more parameters determined during deblocking training (e.g., one or more replacement parameters). For example, one or more initial parameters are each replaced by one or more replacement parameters. In some examples, one or more parameters indicated by deblocking information are decompressed and then used to update the deblocking neural network.

In the example, the entire initial parameter set is updated using one or more replacement parameters. In the example, a subset of the initial parameters is updated by one or more replacement parameters, and the remaining subset of the initial parameters remains unchanged through deblocking training.

By using one or more determined parameters (e.g., one or more replacement parameters) in the deblocking neural network, the deblocking neural network can be applied to the boundary signal to be deblocked and achieve better distortion performance, such as lower RD loss _Lp . As described in Equation 10, R(p) represents the bit consumption of the deblocking information encoded into the video bitstream by one or more determined parameters (e.g., one or more replacement parameters).

According to embodiments of this disclosure, one or more other boundary regions (e.g., boundary region B in reconstructed blocks (1912) and (1914)) can be deblocked based on a determined deblocking neural network corresponding to the boundary region (e.g., boundary region A). For example, when the boundary region (e.g., A) and one or more other boundary regions (e.g., B) have similar pixel distributions, the same deblocking neural network can be applied to deblock the boundary region (e.g., A) and one or more other boundary regions (e.g., B), achieving relatively high coding efficiency.

The above description may be appropriately adapted when using multiple boundary signals, such as multiple boundary regions (e.g., boundary regions A to B), to determine one or more parameters of the deblocking NN—where the determined one or more parameters are shared by multiple boundary signals.

According to embodiments of this disclosure, deblocking training can be referred to as a fine-tuning process, wherein one or more initial parameters (e.g., one or more pre-trained parameters) in the deblocking neural network can be updated (e.g., fine-tuned) based on boundary signals determined based on an input image to be encoded and optionally included in a video bitstream, or an input block of the input image to be encoded and optionally included in a video bitstream. The boundary signals may differ from the training image, training block, or training boundary region used to obtain the pre-trained parameters. Therefore, the deblocking neural network can be tailored to the content of the input image or input block.

In the example, the boundary signal comprises a single boundary region (e.g., boundary region A), and deblocking training is performed using this single boundary region. The deblocking neural network (NN) is determined (e.g., trained or updated) based on this single boundary region. On the decoder side, the determined deblocking NN can be used to deblock the boundary region and optionally other boundary regions. The deblocking information can be encoded into the video bitstream along with coded blocks corresponding to reconstructed blocks including the boundary regions.

In this implementation, the boundary signal includes multiple boundary regions (e.g., boundary regions A to B), and deblocking training is performed using these multiple boundary regions. A deblocking neural network (NN) is determined (e.g., trained or updated) based on these multiple boundary regions. On the decoder side, the determined NN can be used to deblock the multiple boundary regions and optionally other boundary regions. The deblocking information can be encoded into the video bitstream along with coded blocks corresponding to reconstructed blocks that include the multiple boundary regions.

The rate loss R(p) may increase as deblocking information is signaled in the video bitstream. When the boundary signal comprises a single boundary region, deblocking information can be signaled for each boundary region, and a first increase in the rate loss R(p) indicates the increase in rate loss R(p) due to signaling deblocking information for each boundary region. When the boundary signal comprises multiple boundary regions, deblocking information can be signaled for multiple boundary regions, and the deblocking information can be shared by multiple boundary regions, and a second increase in the rate loss R(p) indicates the increase in rate loss R(p) due to signaling deblocking information for each boundary region. Because the deblocking information is shared by multiple boundary regions, the second increase in the rate loss R(p) can be less than the first increase in the rate loss R(p). Therefore, in some examples, using multiple boundary regions to determine (e.g., train or update) the deblocking neural network may be advantageous.

In one implementation, a subset of parameters of the deblocking neural network is determined during deblocking training. For example, a subset of the initial parameters in the deblocking neural network is updated during deblocking training. In another implementation, each parameter of the deblocking neural network is determined during deblocking training. For example, each initial parameter in the deblocking neural network is updated during deblocking training. In this example, the entire set of initial parameters is replaced by one or more of the determined parameters.

In one implementation, one or more initial parameters are updated in a single layer (e.g., a single convolutional layer) of the deblocked neural network. In another implementation, one or more initial parameters are updated in multiple or all layers (e.g., multiple or all convolutional layers) of the deblocked neural network.

Deblocking neural networks (NNs) can be specified by different types of parameters, such as weights and biases. NNs can be configured with appropriate initial parameters, such as weights, biases, or a combination of weights and biases. When using CNNs, weights can include elements from the convolutional kernel.

In one implementation, the one or more initial parameters to be updated are bias terms, and only the bias terms are replaced by the determined one or more parameters. In another implementation, the one or more initial parameters to be updated are weights, and only the weights are replaced by the determined one or more parameters. In yet another implementation, the one or more initial parameters to be updated include weights and bias terms, and are replaced by the determined one or more parameters.

In implementation, different types of parameters (e.g., biases or weights) can be determined (e.g., updated) for different boundary signals (e.g., boundary regions). For example, a first boundary region is used to update a first type of parameter (e.g., at least one bias) in the deblocking neural network corresponding to the first boundary region, and a second boundary region is used to update a second type of parameter (e.g., at least one weight) in the deblocking neural network corresponding to the second boundary region.

In the implementation, different parameters are updated for different boundary signals (e.g., different boundary regions).

In this implementation, multiple boundary signals (e.g., multiple boundary regions) share the same one or more parameters. In the example, all boundary regions in the reconstructed image share the same one or more parameters.

In one implementation, one or more initial parameters to be updated are selected based on characteristics of the boundary signal (e.g., the boundary region), such as the RGB variance of the boundary region. In another implementation, one or more initial parameters to be updated are selected based on the RD performance of the boundary region.

At the end of deblocking training, one or more updated parameters can be computed for the corresponding determined one or more parameters (e.g., corresponding one or more replacement parameters). Deblocking information can indicate one or more updated parameters, and therefore the determined one or more parameters can be indicated by indicating one or more updated parameters. In some implementations, one or more updated parameters can be encoded as deblocking information into the video bitstream. In implementations, one or more updated parameters are computed as the difference between the determined one or more parameters (e.g., corresponding one or more replacement parameters) and the corresponding one or more initial parameters (e.g., one or more pre-trained parameters). For example, deblocking information indicates the difference between one or more parameters and the corresponding one or more of the initial parameters. One or more parameters can be determined based on the sum of the difference and the corresponding one or more of the initial parameters.

In the implementation, one or more updated parameters are the one or more parameters that have been determined.

In implementation, how one or more updated parameters are obtained from the corresponding determined parameters depends on the boundary signals (e.g., boundary regions) used in the deblocking training. Different methods can be used for different boundary regions. In the example, the updated parameters applied to the deblocking neural network for the first boundary region are calculated as the difference between the replacement parameters obtained based on the first boundary region and the corresponding initial parameters. In the example, the updated parameters applied to the deblocking neural network for the second boundary region are the replacement parameters obtained based on the second boundary region.

In this implementation, different boundary signals (e.g., different boundary regions) have different relationships between one or more updated parameters and one or more determined parameters. For example, for a first boundary region, one or more updated parameters are calculated as the difference between one or more replacement parameters and the corresponding one or more pre-trained parameters. For a second boundary region, one or more updated parameters are each one or more replacement parameters.

In this implementation, how one or more updated parameters are obtained from the corresponding determined one or more parameters does not depend on the boundary signals (e.g., boundary regions) used in the deblocking training. In the example, all boundary regions share the same way of updating the parameters in the deblocking neural network. In this implementation, multiple boundary regions in the image (e.g., all boundary regions) have the same relationship between one or more updated parameters and one or more replacement parameters.

In one implementation, the relationship between one or more updated parameters and one or more replacement parameters is selected based on the characteristics of the boundary signal (e.g., the boundary region), such as the RGB variance of the boundary region. In another implementation, the relationship between one or more updated parameters and one or more replacement parameters is selected based on the RD performance of the boundary region.

In implementations, for example, specific linear or nonlinear transformations may be used to generate one or more updated parameters from a set of determined parameters (e.g., one or more replacement parameters), and the one or more updated parameters are representative parameters generated based on the determined one or more parameters. The determined one or more parameters are transformed into one or more updated parameters for better compression.

One or more updated parameters may or may not be compressed. In the example, one or more updated parameters are compressed, for example, using LZMA2, bzip2, etc., which are variants of the Lempel–Ziv–Markov chain algorithm (LZMA). In the example, compression is omitted for one or more updated parameters.

In some implementations, the compression method for one or more updated parameters differs for different boundary signals (e.g., different boundary regions). For example, LZMA2 is used to compress one or more updated parameters for a first boundary region, while bzip2 is used for a second boundary region. In other implementations, the same compression method is used to compress one or more updated parameters for multiple boundary regions (e.g., all boundary regions) in an image or reconstructed block. In other implementations, the compression method is selected based on characteristics of the boundary signals (e.g., boundary regions), such as the RGB variance of the boundary regions. In other implementations, the compression method is selected based on the RD performance of the boundary regions.

Deblocking training can include multiple iterations, where one or more initial parameters are updated during each iteration. Deblocking training can stop when the training loss drops to or is about to drop to a certain level. In the example, deblocking training stops when the training loss (e.g., the RD loss _Lp or the term _Ltraining ) falls below a first threshold. In the example, deblocking training stops when the difference between two consecutive training losses falls below a second threshold.

Two hyperparameters (e.g., step size and maximum number of steps) can be used together with a loss function (e.g., RD loss L _{_p} or term L_{_training}) for deblocking training. The maximum number of iterations can be used as a threshold for the maximum number of iterations to terminate deblocking training. In the example, deblocking training stops when the maximum number of iterations is reached.

The step size can indicate the learning rate of an online training process (e.g., deblocking training). The step size can be used for gradient descent algorithms or backpropagation calculations performed in deblocking training. Any suitable method can be used to determine the step size.

In deblocking training, the step size for each boundary signal (e.g., each boundary region) can be different. In implementations, different step sizes can be assigned to boundary regions in the image to achieve better compression results (e.g., better RD loss _Lp or better term L _training ).

In this implementation, different step sizes are used for boundary signals (e.g., boundary regions) with different types of content to achieve optimal results. Different types can refer to different variances. In the example, the step size is determined based on the variance of the boundary regions used to update the deblocked neural network. For example, the step size for boundary regions with high variance is greater than the step size for boundary regions with low variance, where high variance is greater than low variance.

In one implementation, the step size is selected based on the characteristics of the boundary signal (e.g., the boundary region), such as the RGB variance of the boundary region. In another implementation, the step size is selected based on the RD performance of the boundary region (e.g., RD loss _Lp or term L _training ). Parameters for multiple sets (e.g., replacement parameters for multiple sets) can be generated based on different step sizes, and sets with better compression performance (e.g., smaller RD loss _Lp or smaller term L _training ) can be selected.

In an implementation, the first step size can be used to run a certain number of iterations (e.g., 100). Then, a second step size (e.g., the first step size plus or minus a size increment) can be used to run a certain number of iterations. The results of the first and second step sizes can be compared to determine the step size to use. More than two step sizes can be tested to determine the optimal step size.

During deblocking training, the step size can be varied. The step size can have an initial value at the beginning of deblocking training, and this initial value can be reduced (e.g., halved) later in the deblocking training process to achieve finer tuning. During iterative deblocking training, the step size or learning rate can be changed by a scheduler. The scheduler can include parameter tuning methods for adjusting the step size. The scheduler can determine the value of the step size such that it can be increased, decreased, or kept constant over multiple intervals. In the example, the learning rate is changed by the scheduler at each step. A single scheduler or multiple different schedulers can be used for different boundary signals (e.g., different boundary regions). Therefore, multiple sets of parameters can be generated based on multiple schedulers, and a set of parameters with better compression performance (e.g., smaller RD loss _Lp or smaller term _Ltraining ) can be selected from the multiple sets of parameters.

In one implementation, multiple learning rate schedules are assigned to different boundary signals (e.g., different boundary regions) to achieve better compression results. In one example, all boundary regions in the image share the same learning rate schedule. In another example, a set of boundary regions shares the same learning rate schedule. In one implementation, the selection of the learning rate schedule is based on the characteristics of the boundary regions, such as their RGB variance. In yet another implementation, the selection of the learning rate schedule is based on the RD performance of the boundary regions.

In this implementation, the structure (e.g., architecture) of the deblocking neural network is identical for each boundary signal (e.g., each boundary region). The structure of the deblocking neural network can include multiple layers, how different nodes and/or layers are organized and connected, feedforward architectures, recursive architectures, DNNs, CNNs, etc. In this example, the structure involves multiple convolutional layers, and different blocks have the same number of convolutional layers.

In implementations, different structures of the deblocking neural network correspond to different boundary signals (e.g., different boundary regions). In the example, the deblocking neural network has different numbers of convolutional layers corresponding to different boundary regions.

In this implementation, whether to use a deblocking neural network (NN) to deblock boundary signals (e.g., boundary regions) is determined based on: (i) a comparison of the RD loss _Lp with deblocking training and the RD loss L without deblocking training, or (ii) a comparison of the term L _training with deblocking training and the weighted distortion loss λD without deblocking training. In this implementation, the NN with the best RD performance (e.g., the minimum RD loss _Lp ) is selected based on a comparison of different RD losses _Lp from different NNs.

According to embodiments of this disclosure, each boundary signal (e.g., each boundary region) corresponds to a deblocking neural network (NN) determined during deblocking training based on the boundary region. The deblocking NN can be updated independently of another deblocking NN. For example, the deblocking NN corresponding to a first boundary region is updated independently of the deblocking NN corresponding to a second boundary region.

According to embodiments of this disclosure, a deblocking neural network (NN) corresponding to a boundary signal (e.g., a boundary region) can be updated based on a deblocking NN corresponding to another boundary signal (e.g., another boundary region). In an example, first deblocking information in the coded bitstream corresponding to a first boundary region is decoded. The first deblocking information may indicate a first parameter of the deblocking NN in the video decoder. The deblocking NN in the video decoder corresponding to the first boundary region can be determined based on the first parameter indicated by the first deblocking information. The first boundary region can be deblocked based on the determined deblocking NN corresponding to the first boundary region.

The second adjacent reconstructed block may have a second shared boundary and include a second boundary region of samples on both sides of the second shared boundary. Parameters in the deblocking neural network corresponding to the first boundary region can be used to update the deblocking neural network corresponding to the second boundary region. For example, the pixel distribution in the boundary regions of the same image may be similar, and therefore the parameters to be updated for deblocking neural networks corresponding to different boundary signals (e.g., different boundary regions) can be reduced. Second deblocking information in the encoded bitstream corresponding to the second boundary region can be decoded. The second deblocking information may indicate second parameters. The second boundary region is different from the first boundary region. The deblocking neural network can be updated based on the first and second parameters. The updated deblocking neural network corresponds to the second boundary region and is configured by the first and second parameters. The second boundary region can be deblocked based on the updated deblocking neural network corresponding to the second boundary region.

In implementations, different deblocking neural networks (NNs) can be applied to boundary signals (e.g., boundary regions) of different sizes. Typically, the number of parameters in the deblocking NN can increase with the size of the boundary region (e.g., width, height, or area).

In the implementation, different deblocking NNs correspond to different compression quality targets being applied to the same boundary signal (e.g., the same boundary region).

The NIC framework and deblocking NN can include any type of neural network and use any neural network-based image compression method, such as contextual super-prior encoder-decoder framework, scale super-prior encoder-decoder framework, Gaussian mixture likelihood framework and its variants, RNN-based recursive compression method and its variants, etc.

The deblocking training method and apparatus of this disclosure have the following advantages: Improved encoding efficiency through adaptive online training mechanisms. A flexible and general framework that can be adapted to various types of pre-training frameworks and quality metrics. For example, some initial parameters (e.g., pre-training parameters) in the deblocking neural network can be replaced by online training utilizing boundary signals based on the blocks or images to be encoded.

Figure 23 shows a flowchart outlining an encoding process (2300) according to an embodiment of the present disclosure. The process (2300) can be used to encode input blocks, perform deblocking training, and/or encode deblocking information. In various embodiments, the process (2300) is performed by a processing circuitry, such as a processing circuitry in terminal devices (310), (320), (330), and (340), a processing circuitry performing the functions of a video encoder (e.g., (403), (603), (703), (1600A), or (1700)). In an example, the processing circuitry performs a combination of functions, such as (i) one of a video encoder (403), a video encoder (603), and a video encoder (703), and (ii) one of a video encoder (1600A) or a video encoder (1700). In some embodiments, the process (2300) is implemented in software instructions, so that the processing circuitry executes the process (2300) when the software instructions are executed. The process begins at (S2301) and continues to (S2310).

At (S2310), any suitable method can be used to generate an encoded block (e.g., ) based on an input block (e.g., X) in the input image. In the example, as described with reference to FIG9B, the encoded block is generated based on the input block X. In the example, the input block includes a subset of the blocks in the input image. In the example, the input block includes the entire set of blocks in the input image.

At (S2320), a reconstructed block can be generated based on the coded block. Referring to FIG9B, in the example, a reconstructed block is generated based on the coded block using the NIC framework (900) (e.g., ). Deblocking training can be implemented to determine one or more parameters of the deblocked NN based on boundary signals. As described in this disclosure, boundary signals can be determined based on the reconstructed block. In the example, the boundary signals include boundary regions in the reconstructed block.

Deblocking neural networks can be configured with initial parameters, and one or more of these initial parameters can be iteratively updated during deblocking training. One or more initial parameters can be replaced by one or more parameters.

At (S2330), deblocking information indicating one or more determined parameters of the deblocking NN can be encoded. The deblocking information corresponds to boundary signals (e.g., boundary regions). In some examples, the encoded blocks and deblocking information can be included in the encoded video bitstream and optionally transmitted in the encoded video bitstream. The process (2300) continues to (S2399) and terminates.

The process (2300) can be appropriately adapted to various situations, and the steps in the process (2300) can be adjusted accordingly. One or more steps in the process (2300) can be adjusted, omitted, repeated, and/or combined. The process (2300) can be implemented using any suitable order. Additional steps can be added. In the example, the boundary signal includes reconstructed blocks, and one or more parameters can be generated based on the reconstructed blocks. In the example, the boundary signal includes each reconstructed block in the reconstructed image, and one or more parameters can be generated based on the reconstructed image.

Figure 24 shows a flowchart outlining a decoding process (2400) according to an embodiment of the present disclosure. The process (2400) can be used to reconstruct and deblock boundary signals (e.g., boundary regions) based on deblocking information in the video bitstream. In various embodiments, the process (2400) is performed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video decoder (1600B), and processing circuitry performing the functions of a video decoder including deblocking NN. In an example, the processing circuitry performs a combination of functions including deblocking. In some embodiments, the process (2400) is implemented in software instructions, so that the processing circuitry executes the process (2400) when the software instructions are executed. The process begins at (S2401) and continues to (S2410).

At (S2410), blocks of an image to be reconstructed from the encoded video bitstream can be reconstructed. A reconstructed block may include first adjacent reconstructed blocks having a first shared boundary, and a first boundary region including samples on both sides of the first shared boundary.

At (S2420), the first deblocking information in the encoded video bitstream can be decoded. In the example, the first deblocking information corresponds to a first boundary region. The first deblocking information may indicate the first parameter (or first deblocking parameter) of the deblocking neural network (NN) (e.g., a deep NN or DNN) in the video decoder. The first deblocking parameter of the DNN is an updated parameter that has been previously determined (e.g., trained) through a content-adaptive training process. For example, the first deblocking parameter is determined by an online training process. In the example, the first deblocking information may include the first deblocking parameter.

The first parameter can be a bias term or weight coefficient in the deblocked neural network.

The first deblocking information can indicate one or more parameters of the deblocking neural network in various ways. In the example, the deblocking neural network is configured with initial parameters. The first deblocking information indicates the difference between the first parameter and one of the initial parameters, and the first parameter can be determined based on the sum of the difference and one of the initial parameters. In another example, the first position filtering information directly indicates one or more parameters.

In the example, the number of layers in the deblocking NN depends on the size of the first boundary region (e.g., width, height, or area).

In the example, the first boundary region also includes samples on both sides of the third shared boundary between the third two adjacent reconstruction blocks of the reconstruction block, and the first two adjacent reconstruction blocks are different from the third two adjacent reconstruction blocks.

At (S2430), the deblocking NN in the video decoder for the first boundary region can be determined based on the first parameter indicated by the first deblocking information.

In the example, the deblocking NN is configured with initial parameters, and one of the initial parameters is updated based on the first parameter. For example, one of the initial parameters is replaced with the first parameter.

At (S2440), the first boundary region can be deblocked based on the determined deblocking NN corresponding to the first boundary region.

In the example, the first adjacent reconstruction block also includes a non-boundary region outside the first boundary region, and the first boundary region in the first adjacent reconstruction block is replaced by the first boundary region of the deblock.

In the example, the second adjacent reconstruction blocks of the reconstructed block have a second shared boundary and include a second boundary region with samples on both sides of the second shared boundary. The second boundary region can be deblocked based on the determined deblocking NN corresponding to the first boundary region.

The process (2400) continues to (S2499) and terminates.

The process (2400) can be adapted appropriately to various situations, and the steps in the process (2400) can be adjusted accordingly. One or more steps in the process (2400) can be adjusted, omitted, repeated, and/or combined. The process (2400) can be implemented using any suitable order. Additional steps can be added.

In the example, the second adjacent reconstructed blocks of the reconstructed block have a second shared boundary and include a second boundary region of samples on both sides of the second shared boundary. Second deblocking information in the encoded bitstream corresponding to the second boundary region can be decoded. The second deblocking information can indicate a second parameter. The second boundary region may be different from the first boundary region. The deblocking neural network (NN) can be updated based on the first and second parameters. The updated deblocking NN corresponds to the second boundary region and is configured with at least the first and second parameters. The second boundary region can be deblocked based on the updated deblocking NN corresponding to the second boundary region.

The embodiments described in this disclosure can be used individually or in any combination in any order. Furthermore, each of the method (or embodiment), encoder, and decoder can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium.

This disclosure does not impose any restrictions on the methods used for encoders (such as NN-based encoders) or decoders (such as NN-based decoders). The neural networks used in encoders, decoders, etc., can be any suitable type of neural network, such as DNNs, CNNs, etc.

Therefore, the content-adaptive online training method of this disclosure can be adapted to different types of NIC frameworks, such as different types of encoding DNN, decoding DNN, encoding CNN, decoding CNN, etc.

The above-described techniques can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 25 illustrates a computer system (2500) suitable for implementing certain embodiments of the disclosed subject matter.

Computer software can be coded using any suitable machine code or computer language, which can be subjected to mechanisms such as assembly, compilation, and linking to create code including instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or through interpretation, microcode execution, etc.

The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

The components for the computer system (2500) shown in Figure 25 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or a combination of components shown in the exemplary embodiments of the computer system (2500).

The computer system (2500) may include certain human-machine interface input devices. Such human-machine interface input devices can respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface devices can also be used to capture certain media that are not necessarily directly related to intentional human input, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from still image capturing devices), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (2501), mouse (2502), trackpad (2503), touch screen (2510), data glove (not shown), joystick (2505), microphone (2506), scanner (2507), and camera (2508).

The computer system (2500) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback via a touchscreen (2510), data gloves (not shown), or joystick (2505), but tactile feedback devices that are not used as input devices may also exist); audio output devices (e.g., speakers (2509), headphones (not depicted)); visual output devices (e.g., screens (2510), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which may be able to output two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and ashtrays (not depicted)); and printers (not depicted).

The computer system (2500) may also include human-accessible storage devices and their associated media, such as optical media including CD/DVD ROM/RW (2520) having media such as CD/DVD (2521), thumb drives (2522), removable hard disk drives or solid-state drives (2523), conventional magnetic media such as magnetic tapes and floppy disks (not depicted), and devices based on dedicated ROM/ASIC/PLD such as security dongles (not depicted).

Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

The computer system (2500) may also include interfaces (2554) to one or more communication networks (2555). These networks may be, for example, wireless, wired, or optical. Networks may also be local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), vehicle and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of networks include LANs such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require external network interface adapters attached to certain general-purpose data ports or peripheral buses (2549) (e.g., USB ports of the computer system (2500)); other networks are typically integrated into the core of the computer system (2500) via system buses as described below (e.g., integrated into a PC computer system via an Ethernet interface, or integrated into a smartphone computer system via a cellular network interface). The computer system (2500) can communicate with other entities using any of these networks. Such communication can be one-way (receive only, e.g., broadcast television), one-way (transmit only, e.g., to a CANbus device), or bidirectional, e.g., to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

The human-machine interface device, human-accessible storage device and network interface mentioned above can be attached to the core (2540) of the computer system (2500).

The core (2540) may include one or more central processing units (CPUs) (2541), graphics processing units (GPUs) (2542), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (2543), hardware accelerators (2544) for certain tasks, graphics adapters (2550), etc. These devices, along with read-only memory (ROM) (2545), random access memory (2546), and internal mass storage devices (2547) such as internal non-user-accessible hard disk drives, SSDs, etc., may be connected via the system bus (2548). In some computer systems, the system bus (2548) may be accessed via one or more physical connectors to allow for expansion through additional CPUs, GPUs, etc. Peripheral devices may be attached directly or via a peripheral bus (2549) to the core's system bus (2548). In this example, a screen (2510) may be connected to the graphics adapter (2550). Peripheral bus architectures include PCI, USB, etc.

The CPU (2541), GPU (2542), FPGA (2543), and accelerator (2544) can execute certain instructions, which can be combined to form the computer code mentioned above. This computer code can be stored in ROM (2545) or RAM (2546). Transient data can also be stored in RAM (2546), while permanent data can be stored, for example, in an internal mass storage device (2547). Fast storage and retrieval of any storage device can be achieved by using a cache memory, which can be integrated with one or more CPUs.

(2541), GPU (2542), Mass Storage Device (2547), ROM (2545), RAM

(2546) and other closely related.

Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

By way of example and not limitation, a computer system (2500) with an architecture, particularly a core (2540), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of the core (2540) having non-transitory characteristics, such as a mass storage device (2547) or ROM (2545) within the core. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core (2540). Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause the core (2540)—particularly the processor therein (including a CPU, GPU, FPGA, etc.)—to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (2546) and modifying such data structures according to the processes defined by the software. Alternatively or as an alternative, the computer system may be functionalized by logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2544)), which may operate in place of or with the software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to a computer-readable medium may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry implementing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

Appendix A: Acronyms

JEM: Joint Development Model

VVC: Multi-functional Video Coding

BMS: Benchmark Set

MV: Motion Vector

HEVC: High-efficiency video encoding and decoding

SEI: Supplemental Enhancement Information

VUI: Video Availability Information

GOPs: Image Group

TUs: Transformation Unit

PUs: Prediction Units

CTUs: Coding Tree Units

CTBs: Coded Tree Blocks

PBs: Predicted Blocks

HRD: Hypothetical Reference Decoder

SNR: Signal-to-noise ratio

CPUs: Central Processing Unit

GPUs: Graphics Processing Units

CRT: Cathode Ray Tube

LCD: Liquid Crystal Display

OLED: Organic Light Emitting Diode

CD: Compact Disc

DVD: Digital Video Disc

ROM: Read-Only Memory

RAM: Random Access Memory

ASIC: Application-Specific Integrated Circuit

PLD: Programmable Logic Device

LAN: Local Area Network

GSM: Global System for Mobile Communications

LTE: Long Term Evolution

CANBus: Controller Area Network Bus

USB: Universal Serial Bus

PCI: Peripheral Component Interconnect

FPGA: Field Programmable Gate Domain

SSD: Solid State Drive

IC: Integrated Circuit

CU: Encoding Unit

NIC: Neural Image Compression

R-D: Rate Distortion

E2E: End-to-End

ANN: Artificial Neural Network

DNN: Deep Neural Network

CNN: Convolutional Neural Network

While this disclosure has described several exemplary embodiments, variations, substitutions, and various alternative equivalents fall within the scope of this disclosure. It will therefore be appreciated that, although not expressly shown or described herein, those skilled in the art will be able to conceive of many systems and methods that implement the principles of this disclosure and thus within its spirit and scope.

Claims (7)

1. A video decoding method, characterized in that it includes: Reconstructing blocks of an image from an encoded video bitstream; Decode the first deblocking information in the encoded video bitstream, the first deblocking information including the first deblocking parameters of the deep neural network (DNN) in the video decoder, wherein the first deblocking parameters of the DNN are updated parameters previously determined through a content adaptive training process, and the first deblocking parameters are bias terms or weight coefficients in the DNN. The DNN in the video decoder is determined based on the first deblocking parameters included in the first deblocking information, for a first boundary region including a subset of samples in the reconstructed block, wherein the number of layers in the DNN depends on the size of the first boundary region; and The first boundary region, which includes the subset of samples in the reconstructed block, is deblocked based on a DNN determined according to the first deblocking parameters. The reconstruction block includes a first adjacent reconstruction block, the first adjacent reconstruction block having a first shared boundary and including the first boundary region of the samples on both sides of the first shared boundary; The first adjacent reconstruction block also includes a non-boundary region outside the first boundary region; and Replace the first boundary region in the first adjacent reconstructed block with the first boundary region after deblocking; The reconstruction block includes a second adjacent reconstruction block, the second adjacent reconstruction block having a second shared boundary and including a second boundary region of samples on both sides of the second shared boundary; and The method further includes: Decode the second deblocking information in the encoded video bitstream corresponding to the second boundary region. The second deblocking information indicates the second deblocking parameters that have been previously determined through the content adaptive training process. The second boundary region is different from the first boundary region. The DNN is updated based on the first deblocking parameter and the second deblocking parameter. The updated DNN corresponds to the second boundary region and is configured with the first deblocking parameter and the second deblocking parameter. The second boundary region is deblocked based on the updated DNN corresponding to the second boundary region; Among them, adjacent reconstruction blocks are two adjacent reconstruction sub-blocks. 2. The method according to claim 1, characterized in that, The DNN is configured with initial parameters, and Determining the DNN includes updating one of the initial parameters based on the first deblocking parameters. 3. The method according to claim 2, characterized in that, The first deblocking information indicates the difference between the first deblocking parameter and one of the initial parameters, and The method further includes determining the first deblocking parameter based on the sum of the difference and one of the initial parameters. 4. The method according to claim 1, characterized in that, The first boundary region also includes samples on both sides of the third shared boundary, which lies between two adjacent reconstruction blocks included in the reconstruction block. The first two adjacent reconstruction blocks are different from the third two adjacent reconstruction blocks. 5. A video decoding device, characterized in that it comprises: The reconstruction module is used to reconstruct blocks of an image to be reconstructed from the encoded video bitstream; The first decoding module is used to decode the first deblocking information in the encoded video bitstream. The first deblocking information includes the first deblocking parameters of the deep neural network (DNN) in the video decoder. The first deblocking parameters of the DNN are update parameters that have been previously determined through a content adaptive training process. The first deblocking parameters are bias terms or weight coefficients in the DNN. A first determining module is configured to determine the DNN in the video decoder based on the first deblocking parameters included in the first deblocking information, for a first boundary region including a subset of samples in the reconstructed block, wherein the number of layers in the DNN depends on the size of the first boundary region; and A first deblocking module is configured to deblock the first boundary region, which includes the subset of samples in the reconstructed block, based on a determined DNN corresponding to the first deblocking parameters. The reconstruction block includes a first adjacent reconstruction block, the first adjacent reconstruction block having a first shared boundary and including the first boundary region of the samples on both sides of the first shared boundary; The first adjacent reconstruction block also includes a non-boundary region outside the first boundary region; and Replace the first boundary region in the first adjacent reconstructed block with the first boundary region after deblocking; The reconstruction block includes a second adjacent reconstruction block, the second adjacent reconstruction block having a second shared boundary and including a second boundary region of samples on both sides of the second shared boundary; and The first deblocking module is further configured to: Decode the second deblocking information in the encoded video bitstream corresponding to the second boundary region. The second deblocking information indicates the second deblocking parameters that have been previously determined through the content adaptive training process. The second boundary region is different from the first boundary region. The DNN is updated based on the first deblocking parameter and the second deblocking parameter. The updated DNN corresponds to the second boundary region and is configured with the first deblocking parameter and the second deblocking parameter. The second boundary region is deblocked based on the updated DNN corresponding to the second boundary region; Among them, adjacent reconstruction blocks are two adjacent reconstruction sub-blocks. 6. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer-readable instructions, and When the computer-readable instructions are executed by the processor, the electronic device performs the method according to any one of claims 1 to 4. 7. A non-transitory computer-readable storage medium storing a program, the program being executable by at least one processor to perform the method according to any one of claims 1 to 4.