HK40080593B - Video decoding method, video decoding apparatus and storage medium - Google Patents

Description

Video decoding methods, video decoding equipment and storage media

Incorporation

This application claims priority to U.S. Patent Application No. 17/729,978, filed April 26, 2022, entitled “BLOCK-WISE CONTENT-ADAPTIVE ONLINETRAINING IN NEURAL IMAGE,” which in turn claims priority to U.S. Provisional Application No. 63/182,366, filed April 30, 2021, entitled “Block-wise Content-Adaptive Online Training in Neural Image Compression.” The entire disclosure of the earlier applications is incorporated herein by reference.

Technical Field

This disclosure describes embodiments that generally involve video coding.

Background Technology

The background description provided herein is intended to provide a general overview of the context of this disclosure. To the extent described in this background section, the work of the currently attributed inventors and aspects of the description that may not conform to the prior art at the time of filing should not be explicitly or implicitly considered as prior art to this disclosure.

Video encoding and decoding can be performed using inter-frame picture prediction with motion compensation. Uncompressed digital images and/or video can comprise a series of pictures, each with spatial dimensions, for example, 1920×1080 luma samples and correlated chroma samples. This series of pictures can have a fixed or variable image rate (informally also called frame rate), for example, 60 pictures per second or 60Hz. Uncompressed images and/or video have specific bit rate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luma sample resolution at 60Hz frame rate) requires close to 1.5 Gbit/s of bandwidth. One hour of such video would require more than 600 GBytes of storage space.

One objective of video encoding and decoding is to reduce redundancy in the input image and/or video signal through compression. Compression helps reduce the aforementioned bandwidth and/or storage space requirements, in some cases by two orders of magnitude or more. Although the description herein uses video encoding/decoding as an illustrative example, the same techniques can be applied in a similar manner to image encoding/decoding without departing from the spirit of this disclosure. Lossless compression and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal is useful for the intended application. In the case of video, lossy compression is widely used. The amount of distortion tolerated depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can be reflected in the fact that higher tolerance/tolerable distortion can result in a higher compression ratio.

Video encoders and decoders can utilize several major categories of techniques, including motion compensation, transform, quantization, and entropy coding.

Video codec techniques can include techniques called intra-frame coding. In intra-frame coding, sample values are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture can be an intra-frame picture. Intra-frame pictures and their derived pictures (e.g., standalone decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in the encoded video bitstream and video session, or as a still image. Samples of intra-frame blocks can be transformed, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique that minimizes sample values in the pre-transform domain. In some cases, the smaller the transformed DC value, the smaller the AC coefficients, and the fewer bits are needed to represent the entropy-coded block at a given quantization step size.

For example, traditional intra-frame coding known from MPEG-2 generation coding techniques does not use intra-frame prediction. However, some newer video compression techniques include attempts to use surrounding sample data and/or metadata obtained during the encoding and/or decoding of, for example, spatially adjacent data blocks that are earlier in the decoding order. This technique is hereby referred to as "intra-frame prediction." Note that, at least in some cases, intra-frame prediction uses only reference data from the current picture in the reconstruction, and not reference data from a reference picture.

There can be many different forms of intra-prediction. When more than one such technique can be used in a given video coding technique, the technique used can be encoded in the intra-prediction mode. In some cases, the mode can have sub-modes and/or parameters, which can be encoded individually or included in the mode codeword. For a given combination of modes, sub-modes, and/or parameters, which codeword is used will affect the coding efficiency gain through intra-prediction, and therefore, the entropy coding technique used to convert the codeword into a bitstream will also have an impact.

H.264 introduced specific intra-frame prediction modes, which were improved in H.265 and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Universal Video Coding (VVC), and Baseline Set (BMS). Prediction blocks can be formed using neighboring sample values belonging to already available samples. The sample values of neighboring samples are copied into the prediction block according to the orientation. The reference to the orientation being used can be encoded in the bitstream or can be predicted itself.

Referring to Figure 1A, the lower right depicts a subset of nine known prediction directions from the 33 possible prediction directions of H.265 (corresponding to 33 angular modes of 35 intra-frame modes). The point (101) where the arrows converge represents the sample being predicted. The arrows indicate the direction of the predicted sample. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples to the upper right at a 45° angle to the horizontal line. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples to the lower left of sample (101) at a 22.5° angle to the horizontal line.

Referring again to Figure 1A, a square block (104) of 4×4 samples is depicted in the upper left (represented by a thick dashed line). The square block (104) comprises 16 samples, each labeled "S" with its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (counting from the top) and the first sample in the X dimension (counting from the left). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. Reference samples are labeled (104) with R, their Y position (e.g., row index) relative to block (104), and their X position (column index). In H.264 and H.265, predicted samples are adjacent to blocks in the reconstruction; therefore, negative values are not required.

Intra-frame image prediction works by copying reference sample values from neighboring samples, which is appropriate using the signaled prediction direction. For example, suppose the encoded video bitstream contains a signal that indicates a prediction direction consistent with arrow (102) for this block—that is, predicting multiple samples based on one or more prediction samples at the upper right, at a 45° angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Then sample S44 is predicted based on reference sample R08.

In some cases, the values of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample; especially when the direction is not divisible by 45°.

With the development of video coding technology, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), while JEM/VVC/BMS, at the time of its disclosure, could support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding have been used to represent those possible directions with a small number of bits, penalizing less likely directions. Furthermore, the direction itself can sometimes be predicted from adjacent directions used in adjacent decoded blocks.

Figure 1B shows a schematic diagram (110) depicting 65 intra-frame prediction directions according to JEM to illustrate how the number of prediction directions increases over time.

The mapping of intra-prediction direction bits representing direction in the encoded video bitstream can vary depending on the video coding technique. This mapping can include, for example, simple and straightforward mappings such as mapping from prediction direction to intra-prediction mode, or from prediction direction to codeword; it can also include complex adaptive schemes involving the most probable mode, and similar techniques. However, in all cases, some directions are statistically less likely to appear in the video content compared to some other directions. Since the goal of video compression is to reduce redundancy, in well-functioning video coding techniques, those less likely directions will be represented by more bits than the more likely directions.

Motion compensation can be a lossy compression technique and can involve techniques in which sample data blocks from a previously reconstructed image or a portion thereof (the reference image) are spatially shifted in a direction indicated by a motion vector (hereinafter referred to as MV) to predict a newly reconstructed image or image portion. In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference image in use (the latter can indirectly be a temporal dimension).

In some video compression techniques, a motion vector (MV) applicable to a region of sample data can be predicted based on other MVs, such as a MV prediction based on another region of sample data that is spatially adjacent to the region being reconstructed and precedes that MV in the decoding order. Doing so can significantly reduce the amount of data required to encode the MV, thereby eliminating redundancy and improving compression ratio. MV prediction works effectively, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistically likely probability that multiple regions larger than the region to which a single MV can be applied move in similar directions. Therefore, in some cases, similar MVs (motion vectors) derived from multiple MVs of neighboring regions can be used for prediction. This results in the MV found for a given region being similar to or identical to the MV predicted based on surrounding MVs, and after entropy encoding, this can be represented with fewer bits than when directly encoding the MV. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, such as when calculating predictions from several surrounding MVs, the MV prediction itself can be lossy due to rounding errors.

Various MV prediction mechanisms are described in H.265/HEVC (ITU-TRec.H.265, “High Efficiency Video Coding”, December 2016). In addition to the many MV prediction mechanisms provided by H.265, a technique hereinafter referred to as “spatial merging” is described here.

Referring to Figure 2, the current block (201) includes samples discovered by the encoder during motion search so that predictions can be made based on previous blocks of the same size that have been spatially shifted. Instead of directly encoding the MV, the MV can be obtained using the MV associated with any of the five surrounding samples (denoted as A0, A1 and B0, B1, B2 (202 to 206 respectively)) based on metadata associated with one or more reference images, for example, based on the most recent (in decoding order) reference image. In H.265, MV prediction can use multiple predictors that make predictions based on the same reference images being used by adjacent blocks.

Summary of the Invention

This disclosure provides video encoding and decoding methods and apparatuses. In some examples, a video decoding apparatus includes processing circuitry. This processing circuitry is configured to decode update information of a first neural network in an encoded bitstream, the update information being used by the first neural network. The first neural network is configured with a first set of pre-trained parameters. The update information of the first neural network corresponds to a first block in an image to be reconstructed and indicates a first replacement parameter corresponding to the first pre-trained parameter in the first set of pre-trained parameters. The processing circuitry can update the first neural network based on the first replacement parameter and decode the first block based on the updated first neural network, the updated first neural network being used by the first block.

In one embodiment, the first neural network update information further indicates one or more replacement parameters, which are used for one or more remaining neural networks in addition to the first neural network. The processing circuitry can update the one or more remaining neural networks based on the one or more replacement parameters.

In one embodiment, the processing circuit decodes second neural network update information in the encoded bitstream, the second neural network update information being used by the second neural network. The second neural network is configured with a second set of pre-trained parameters. The second neural network update information corresponds to a second block in the image to be reconstructed and indicates a second replacement parameter corresponding to the second pre-trained parameter in the second set of pre-trained parameters. In one example, the second neural network differs from the first neural network. The processing circuit can update the second neural network based on the second replacement parameters and decode the second block based on the updated second neural network, the updated second neural network being used by the second block.

In one embodiment, the first pre-training parameter is one of the pre-training weight coefficients and the pre-training bias term.

In one embodiment, the second pre-training parameter is another of the pre-training weight coefficients and the pre-training bias term.

In one embodiment, the processing circuit decodes a second block of the encoded bitstream based on an updated first neural network, which was used for the first block.

In one embodiment, the first neural network update information indicates the difference between the first replacement parameter and the first pre-trained parameter. The processing circuitry determines the first replacement parameter based on the difference and the sum of the first pre-trained parameters.

In one embodiment, the processing circuitry decodes the first neural network update information based on a variation of the Lempel-Ziv-Markov chain algorithm (LZMA2) and the bzip2 algorithm.

In one example, the processing circuitry updates information based on another decoding second neural network in the LZMA2 and bzip2 algorithms.

This disclosure also provides a non-transitory computer-readable storage medium storing a program executable by at least one processor to perform a video decoding method.

Attached Figure Description

Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

Figure 1A is a schematic diagram of an exemplary subset of intra-frame prediction modes;

Figure 1B is an illustration of an exemplary intra-frame prediction direction;

Figure 2 shows the current block and surrounding samples according to one embodiment;

Figure 3 is a simplified block diagram of a communication system according to one embodiment;

Figure 4 is a simplified block diagram of a communication system according to one embodiment;

Figure 5 is a simplified block diagram of a decoder according to one embodiment;

Figure 6 is a simplified block diagram of an encoder according to one embodiment;

Figure 7 shows a block diagram of an encoder according to another embodiment;

Figure 8 shows a block diagram of a decoder according to another embodiment;

Figure 9A illustrates an example of block image encoding according to an embodiment of the present disclosure;

Figure 9B illustrates an exemplary NIC framework according to an embodiment of this disclosure;

Figure 10 illustrates an exemplary convolutional neural network (CNN) of a master encoder network according to an embodiment of the present disclosure;

Figure 11 illustrates an exemplary CNN of a master decoder network according to an embodiment of the present disclosure;

Figure 12 illustrates an exemplary CNN of a hyper encoder according to an embodiment of this disclosure;

Figure 13 illustrates an exemplary CNN of a hyper decoder according to an embodiment of the present disclosure;

Figure 14 illustrates an exemplary CNN of a context model network according to an embodiment of the present disclosure;

Figure 15 illustrates an exemplary CNN of an entropy parameter network according to an embodiment of the present disclosure;

Figure 16A illustrates an exemplary video encoder according to an embodiment of the present disclosure;

Figure 16B illustrates an exemplary video decoder according to an embodiment of the present disclosure;

Figure 17 illustrates an exemplary video encoder according to an embodiment of the present disclosure;

Figure 18 illustrates an exemplary video decoder according to an embodiment of the present disclosure;

Figure 19 shows a flowchart outlining a process according to an embodiment of the present disclosure;

Figure 20 shows a flowchart outlining a process according to an embodiment of the present disclosure;

Figure 21 is a schematic diagram of a computer system according to one embodiment.

Detailed Implementation

Figure 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices capable of communicating with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, terminal device (310) may encode video data (e.g., a video image stream captured by terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. Terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover the video images, and display the video images based on the recovered video data. Unidirectional data transmission is common in media service applications, etc.

In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, such as that that may occur during a video conference. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) can encode video data (e.g., a stream of video images captured by the terminal device) for transmission via a network (350) to the other terminal device (330) and (340). Each of the terminal devices (330) and (340) can also receive encoded video data transmitted by the other terminal device (330) and (340), and can decode the encoded video data to recover the video images, and can display the video images on an accessible display device based on the recovered video data.

In the example of Figure 3, the terminal devices (310), (320), (330), and (340) can be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and/or dedicated video conferencing equipment. Network (350) refers to any number of networks that transmit encoded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired (wired) and/or wireless communication networks. The communication network (350) can exchange data in circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may be of little importance to the operation of this disclosure, unless explained below.

As an example of the application of the disclosed subject matter, Figure 4 illustrates the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter can also be applied to other video-enabled applications, including, for example, video conferencing, digital television, and storing compressed video on digital media including CDs, DVDs, Memory Sticks, etc.

The streaming system may include: a capture subsystem (413) that may include a video source (401), such as a digital camera; and a system for creating, for example, an uncompressed video picture stream (402). In one example, the video picture stream (402) includes samples captured by a digital camera. The video picture stream (402) is depicted as a thick line emphasizing the high data volume when compared to encoded video data (404) (or encoded video bitstream), and may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume when compared to the video picture stream (402), and may be stored on a streaming server (405) for future use. One or more streaming client subsystems (e.g., client subsystems (406) and (408) in Figure 4) can access a streaming server (405) to retrieve copies (407) and (409) of encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and creates an output video picture stream (411) that can be presented on a display (412) (e.g., a screen) or other presentation device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) may be encoded according to certain video coding/compression standards. Examples of such standards include ITU-T Recommendation H.265. In one example, the video coding standard under development is informally referred to as Universal Video Coding (VVC). The disclosed topics can be used in a VVC environment.

Note that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

Figure 5 shows a block diagram of a video decoder (510) according to an embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of the video decoder (410) in the example of Figure 4.

The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences can be received from a channel (501), which can be a hardware/software link to a storage device storing the encoded video data. The receiver (531) can receive encoded video data and other data, such as encoded audio data and/or auxiliary data streams, which can be forwarded to their respective user entities (not shown). The receiver (531) can separate the encoded video sequences from other data. To combat network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder/parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, it can be outside the video decoder (510) (not shown). In other cases, a buffer memory (not shown) may be present outside the video decoder (510), for example, to combat network jitter. Additionally, another buffer memory (515) may be present inside the video decoder (510), for example, to handle playback timing. The buffer memory (515) may be unnecessary or small when the receiver (531) receives data from a store/forward device with sufficient bandwidth and controllability or from an isochronous network. For use on best-effort packet networks such as the Internet, a buffer memory (515) may be required. This buffer memory may be relatively large and advantageously have an adaptive size, and may be implemented at least partially outside the video decoder (510) in an operating system or similar component (not shown).

The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (510) and information that potentially controls a presentation device (512) (e.g., a display screen), which is not part of the electronic device (530) but may be coupled to it, as shown in FIG. 5. Control information for the presentation device may be in the form of supplementary enhancement information (SEI messages) or fragments of video availability information (VUI) parameter sets (not shown). The parser (520) may perform parsing/entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters of at least one pixel subgroup from the encoded video sequence based on at least one parameter corresponding to the set. Subgroups may include group of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (Cu), blocks, transform units (TU), prediction units (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

The parser (520) can perform entropy decoding/parsing operations on the video sequence received from the buffer memory (515) in order to create symbols (521).

Depending on the type of the encoded video picture or its portions (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) can involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by the parser (520). For clarity, the flow of such subgroup control information between the parser (520) and the multiple units below is not described.

In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In practical implementations operating under commercial constraints, many of these units interact closely with each other and can be integrated at least partially with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the following functional units.

The first unit is the scaler/inverse transform unit (551). The scaler/inverse transform unit (551) receives the quantized transform coefficients and control information, including which transform to use, block size, quantization factor, quantization scaling matrix, etc., as symbols (521) from the parser (520). The scaler/inverse transform unit (551) can output blocks containing sample values, which can be input into the aggregator (555).

In some cases, the output samples of the scaler/inverse transform (551) may belong to intra-coded blocks; that is, blocks that do not use prediction information from previously reconstructed images but can use prediction information from previously reconstructed portions of the current image. This prediction information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding reconstructed information obtained from the current picture buffer (558) to generate blocks of the same size and shape as the blocks in the reconstruction. The current picture buffer (558) buffers, for example, partially reconstructed current images and/or fully reconstructed current images. In some cases, the aggregator (555) adds the prediction information already generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler/inverse transform unit (551) based on each sample.

In other cases, the output samples of the scaler/inverse transform unit (551) may belong to inter-frame coded blocks and may be motion-compensated. In this case, the motion compensation prediction unit (553) can access the reference image memory (557) to obtain samples for prediction. After motion compensation of the extracted samples according to the symbols (521) associated with the block, these samples can be added by the aggregator (555) to the output of the scaler/inverse transform unit (551) (referred to as residual samples or residual signals in this case) to generate output sample information. The addresses in the reference image memory (557) from which the motion compensation prediction unit (553) obtains the predicted samples can be controlled by motion vectors, and the motion compensation prediction unit (553) can obtain these addresses in the form of symbols (521), which may have, for example, X, Y, and reference image components. When using subsampled exact motion vectors, motion compensation may also include interpolation of sampled values obtained from the reference image memory (557), motion vector prediction mechanisms, etc.

The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression techniques may include loop filtering techniques controlled by parameters contained in the encoded video sequence (also known as the encoded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be in response to metadata obtained during the decoding of a previous (in the order of decoding) portion of the encoded image or encoded video sequence and to previously reconstructed and loop-filtered sample values.

The output of the loop filter unit (556) can be a sample stream, which can be output to the presentation device (512) and stored in the reference image memory (557) for future inter-frame image prediction.

Once fully reconstructed, certain encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and that encoded image has been identified as a reference image (e.g., by the parser (520)), the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of the next encoded image begins.

The video decoder (510) can perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T Rec.H.265. The encoded video sequence can conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence conforms to the syntax of the video compression technique or standard and the brief documented in the video compression technique or standard. Specifically, the brief may select certain tools from all available tools in the video compression technique or standard as the only tools available under that brief. Standard conformance also requires the complexity of the encoded video sequence to be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the level can be further restricted by the assumed reference decoder (HRD) specification and the metadata managed by the HRD buffer of the signaling in the encoded video sequence.

In one embodiment, the receiver (531) may receive additional (redundant) data with encoded video. This additional data may be included as part of the encoded video sequence. The video decoder (510) may use the additional data to correctly decode the data and/or more accurately reconstruct the original video data. The additional data may be, for example, in the form of temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

Figure 6 shows a block diagram of a video encoder (603) according to an embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of Figure 4.

The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of Figure 6), which can capture video images to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

The video source (601) can be provided as a digital video sample stream of a sequence of source videos to be encoded by a video encoder (603). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601YCrCb, RGB, ...), and any suitable sampling structure (e.g., YCrCb4:2:0, YCrCb4:4:4). In a media service system, the video source (601) can be a storage device storing previously prepared video. In a video conferencing system, the video source (601) can be a camera capturing local image information as a video sequence. Video data can be provided as multiple individual pictures, which, when viewed sequentially, are given motion. The pictures themselves can be organized as a spatial array of pixels, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

According to one embodiment, the video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units as described below. For clarity, coupling is not described. Parameters set by the controller (650) may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions related to the video encoder (603) optimized for a particular system design.

In some embodiments, the video encoder (603) is configured to operate within an encoding loop. As an oversimplification, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input picture to be encoded and a reference picture) and a (local) decoder (633) embedded within the video encoder (603). The decoder (633) reconstructs the symbols in a manner similar to that created by the (remote) decoder to create sample data (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream results in bit-precise results independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-precise between the local and remote encoders. In other words, when prediction is used during decoding, the encoder's prediction portion, as a reference picture sample, "sees" the exact same sample values as the decoder "sees." This basic principle of reference image synchronization (and the resulting drift if synchronization cannot be maintained, for example, due to channel errors) is also used in some related technologies.

The operation of the “local” decoder (633) can be the same as that of a “remote” decoder such as a video decoder (510), which has been described in detail above in conjunction with Figure 5. However, briefly referring to Figure 5, since symbols are available and the encoding/decoding of symbols for the encoded video sequence by the entropy encoder (645) and the parser (520) can be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

In one embodiment, the decoder techniques exist in the corresponding encoder in the same or substantially the same functional form, except for the parsing/entropy decoding present in the decoder. Therefore, the disclosed subject focuses on decoder operation. The description of the encoder techniques can be simplified because these techniques are the inverse of the fully described decoder techniques. In some areas, a more detailed description is provided below.

During operation, in some examples, the source encoder (630) may perform motion-compensated predictive coding, which predictively encodes the input image by referencing one or more previously encoded images from the video sequence designated as "reference images". In this way, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture based on symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that can be performed on the reference picture by the video decoder, and can make the reconstructed reference picture stored in a reference picture buffer (634). In this way, the video encoder (603) can locally store copies of the reconstructed reference pictures that have the same content as the reconstructed reference pictures that will be obtained by the remote video decoder (without transmission errors).

The predictor (635) can perform a prediction search on the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can be used as appropriate prediction references for the new image. The predictor (635) can operate on a sample block-by-pixel basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references extracted from multiple reference images stored in the reference image memory (634).

The controller (650) can manage the encoding operations of the source encoder (630), including, for example, the settings of parameters and subgroup parameters for encoding video data.

The outputs of all the aforementioned functional units can undergo entropy encoding (645) in the entropy encoder. The entropy encoder (645) converts the symbols generated by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, and arithmetic coding.

The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware/software link to a storage device storing the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (source not shown).

The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded picture type to each encoded picture, which can affect the encoding techniques that can be applied to the corresponding picture. For example, a picture can typically be designated as one of the following picture types:

An intra-frame picture (I-picture) can be a picture that is encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (“IDR”) pictures. Those skilled in the art will recognize those variations of I-pictures and their corresponding applications and characteristics.

A predicted image (P-image) can be an image that uses at most one motion vector and a reference index to predict the sample value of each block, and is encoded and decoded using intra-frame prediction or inter-frame prediction.

A bidirectional prediction image (B-image) can be an image that uses up to two motion vectors and a reference index to predict sample values for each block, and is encoded and decoded using intra-frame prediction or inter-frame prediction. Similarly, a multi-prediction image can reconstruct a single block using two or more reference images and associated metadata.

The source image can typically be spatially subdivided into multiple sample blocks (e.g., each sample block is 4×4, 8×8, 4×8, or 16×16 sample blocks) and encoded on a block-by-block basis. Blocks can be predictedly encoded by referencing other (already encoded) blocks determined by the encoding allocation applied to the corresponding image of the block. For example, blocks of image I can be unpredictably encoded, or they can be predictedably encoded (spatial prediction or intra-frame prediction) by referencing already encoded blocks of the same image. Pixel blocks of image P can be predictedably encoded by referencing a previously encoded reference image, either spatially or temporally. Blocks of image B can be predictedably encoded by referencing one or two previously encoded reference images, either spatially or temporally.

The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard (e.g., ITU-T Rec.H.265). In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard being used.

In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data (e.g., redundant images and slices), SEI messages, VUI parameter set fragments, etc.

Video can be captured as multiple source images (video pictures) in a time series. Intra-frame picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-frame picture prediction utilizes (temporal or other) correlations between pictures. In one example, a specific picture in the encoding/decoding process, referred to as the current picture, is segmented into blocks. When a block in the current picture resembles a reference block in a previously encoded and still buffered reference picture in the video, the block in the current picture can be encoded using a vector called a motion vector. In the case of using multiple reference pictures, the motion vector points to the reference block in the reference picture and can have a third dimension that identifies the reference picture.

In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, for example, a first reference image and a second reference image, both of which precede the current image in the video in the decoding order (but can be displayed in the past and future, respectively). A block in the current image can be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted using a combination of the first and second reference blocks.

In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

According to some embodiments of this disclosure, prediction is performed on a block-by-block basis, such as inter-frame picture prediction and intra-frame picture prediction. For example, according to the HEVC standard, pictures in a video picture sequence are segmented into Coding Tree Units (CTUs) for compression. The CTUs in the pictures have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU comprises three Coding Tree Blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-segmented into one or more Coding Units (CUs). For example, a 64×64 pixel CTU can be segmented into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine its prediction type, such as inter-frame prediction or intra-frame prediction. Based on temporal and/or spatial predictability, CUs are divided into one or more Prediction Units (PUs). Typically, each PU comprises one luma prediction block (PB) and two chroma PBs. In this embodiment, the prediction operation in the encoding (encoding/decoding) is performed on a per-prediction-block basis. A luminance prediction block is used as an example; a prediction block contains a matrix of pixel values (e.g., luminance values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

Figure 7 illustrates a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values within a current video image in a video image sequence and to encode the processing block into an encoded image as part of an encoded video sequence. In one example, the video encoder (703) is used instead of the video encoder (403) in the example of Figure 4.

In the HEVC example, the video encoder (703) receives a sample value matrix of the processed block, such as an 8×8 sample prediction block. The video encoder (703) determines the optimal encoding method for the processed block: intra-frame mode, inter-frame mode, or bidirectional prediction mode, such as rate-distortion optimized mode. When encoding the processed block in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processed block into a coded picture; and when encoding the processed block in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques to encode the processed block into a coded picture, respectively. In some video coding techniques, the merging mode may be an inter-frame picture prediction sub-mode, in which motion vectors are derived from one or more motion vector predictors without benefiting from coded motion vector components outside the predictors. In some other video coding techniques, there may be motion vector components applicable to the object block. In one example, the video encoder (703) includes other components, such as a mode determination module (not shown) for determining the mode of the processed block.

In the example of Figure 7, the video encoder (703) includes an inter-frame encoder (730), an intra-frame encoder (722), a residual encoder (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in Figure 7.

An inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare that block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images), generate inter-frame prediction information (e.g., descriptions of redundancy information based on inter-frame coding techniques, motion vectors, merging mode information), and compute inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image based on the decoded encoded video information.

The intra encoder (722) is configured to receive samples of the current block (e.g., the processing block), in some cases compare the block with blocks already encoded in the same image, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information based on one or more intra coding techniques). In one example, the intra encoder (722) also computes an intra prediction result (e.g., a prediction block) based on the intra prediction information in the same image and a reference block.

A general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the mode of a block and provides control signals to a switch (726) based on that mode. For example, when the mode is intra-frame mode, the general controller (721) controls the switch (726) to select intra-frame mode results for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; when the mode is inter-frame mode, the general controller (721) controls the switch (726) to select inter-frame prediction results for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

A residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or inter encoder (730). A residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and the decoded image can be cached in a memory circuit (not shown) and used as a reference image in some examples.

The entropy encoder (725) is configured to format the bitstream to include coded blocks. The entropy encoder (725) is configured to include various information according to a suitable standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Note that, according to the disclosed subject matter, no residual information is present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

Figure 8 illustrates a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In one example, the video decoder (810) is used instead of the video decoder (410) in the example of Figure 4.

In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together as shown in Figure 8.

The entropy decoder (871) can be configured to reconstruct certain symbols representing the syntax elements constituting the encoded image from the encoded image. Such symbols may include, for example, the mode in which the block is encoded (e.g., intra-mode, inter-mode, bidirectional prediction mode, merged sub-mode, or the latter two of another sub-mode), prediction information (e.g., intra-prediction information or inter-prediction information) that can identify specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), such as residual information in the form of quantized transform coefficients, etc. In one example, when the prediction mode is inter-mode or bidirectional prediction mode, inter-prediction information is provided to the inter-decoder (880); and when the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and be provided to the residual decoder (873).

The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

The intra-frame decoder (872) is configured to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

The residual decoder (873) is configured to perform inverse quantization to extract the dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to include quantizer parameters (QP)), and this information can be provided by the entropy decoder (871) (the data path is not shown because this may only be a small amount of control information).

The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) and the prediction results (output by the inter-frame or intra-frame prediction module, as appropriate) in the spatial domain to form a reconstruction block, which can be part of a reconstructed image, which in turn can be part of a reconstructed video. Note that other suitable operations, such as deblocking, can be performed to improve visual quality.

Note that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

This disclosure describes video coding techniques related to neural image compression and/or neural video compression, such as artificial intelligence (AI)-based neural image compression (NIC). Aspects of this disclosure include content-adaptive online training in NICs, for example, a chunked content-adaptive online training method for optimizing an image coding framework based on neural networks for end-to-end (E2E) optimization. Neural networks (NNs) can include artificial neural networks (ANNs), such as deep neural networks (DNNs), convolutional neural networks (CNNs), etc.

In one embodiment, it is difficult to optimize a hybrid video codec as a whole. For example, improvements to a single module (e.g., the encoder) in a hybrid video codec may not result in a coding gain for the overall performance. In a neural network-based video coding framework, different modules can be jointly optimized from input to output by performing a learning or training process (e.g., a machine learning process) to improve the final objective (e.g., rate-distortion performance, such as the rate-distortion loss L described in this disclosure) and thus produce an E2E optimized NIC.

An exemplary NIC framework or system can be described as follows. The NIC framework can use an input block x as input to a neural network encoder (e.g., a neural network-based encoder, such as a DNN) to compute a compressed representation (e.g., a compact representation). This compressed representation can be compact, for example, for storage and transmission purposes. A neural network decoder (e.g., a neural network-based decoder, such as a DNN) can use the compressed representation as input to reconstruct an output block (also called a reconstructed block). In various embodiments, the input block x and the reconstructed block are in a spatial domain, while the compressed representation is in a domain different from the spatial domain. In some examples, the compressed representation is quantized and entropy-encoded.

In some examples, the NIC framework can use a variable autoencoder (VAE) architecture. In a VAE architecture, the neural network encoder can directly use the entire input block x as input. The entire input block x can be passed through a set of neural network layers operating as black boxes to compute a compressed representation, which is the output of the neural network encoder. The neural network decoder can then use the entire compressed representation as input. The compressed representation can be passed through another set of neural network layers operating as another black box to compute the reconstructed block rate-distortion (R-D) loss. This loss can be optimized to achieve a trade-off between the distortion loss of the reconstructed block and the bit consumption R of the compact representation using a trade-off hyperparameter λ.

Neural networks (e.g., ANNs) can learn to perform tasks from examples without task-specific programming. An ANN can be configured with connected nodes or artificial neurons. Connections between nodes can transmit signals from a first node to a second node (e.g., a receiving node), and these signals can be modified by weights, which can be indicated by the weight coefficients of the connections. A receiving node can process signals from nodes that send signals to it (i.e., the input signals to the receiving node) and then generate an output signal by applying a function to the input signals. This function can be a linear function. In one example, the output signal is a weighted sum of the input signals. In another example, the output signal is further modified by a bias that can be indicated by a bias term, so the output signal is the sum of the bias and the weighted sum of the input signals. The function can include, for example, a nonlinear operation on the weighted sum of the input signals or the weighted sum of the bias. The output signal can be sent to nodes connected to the receiving node (downstream nodes). An ANN can be represented or configured by parameters (e.g., the weights of the connections and/or biases). The weights and/or biases can be obtained by training the ANN with examples where the weights and/or biases can be iteratively adjusted. A trained ANN with defined weights and/or defined biases can be used to perform tasks.

Nodes in an ANN can be organized using any suitable architecture. In various embodiments, nodes in an ANN are organized into layers, including an input layer that receives the input signals to the ANN and an output layer that outputs signals from the ANN. In one embodiment, the ANN also includes layers between the input and output layers, such as hidden layers. Different layers can perform different kinds of transformations on their respective inputs. Signals can be transmitted from the input layer to the output layer.

An ANN with multiple layers between the input and output layers can be called a DNN. In one embodiment, a DNN is a feedforward network, where data flows from the input layer to the output layer without loopback. In one example, a DNN is a fully connected network, where each node in one layer is connected to all nodes in the next layer. In one embodiment, a DNN is a recurrent neural network (RNN), where data can flow in any direction. In one embodiment, a DNN is a CNN.

A CNN may include an input layer, an output layer, and hidden layers between the input and output layers. Hidden layers may include convolutional layers (e.g., used in an encoder) that perform convolutions (e.g., two-dimensional (2D) convolutions). In one embodiment, the 2D convolution performed in the convolutional layer is between the convolutional kernel (also called a filter or channel, e.g., a 5×5 matrix) and the input signal of the convolutional layer (e.g., a 2D matrix, such as a 2D block, a 256×256 matrix). In various examples, the dimension of the convolutional kernel (e.g., 5×5) is smaller than the dimension of the input signal (e.g., 256×256). Therefore, the portion of the input signal (e.g., a 256×256 matrix) covered by the convolutional kernel (e.g., a 5×5 region) is smaller than the region of the input signal (e.g., a 256×256 region), and can therefore be referred to as the receptive field in the corresponding node of the next layer.

During convolution, the dot product of the convolution kernel and the corresponding receptive field in the input signal is calculated. Therefore, each element of the convolution kernel is a weight applied to the corresponding sample in the receptive field, thus the convolution kernel comprises weights. For example, a convolution kernel represented by a 5×5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolutional layer, and the output signal is based on the sum of the dot product and the bias.

The convolutional kernel can move along the input signal (e.g., a 2D matrix) by a size called stride, so the convolution operation generates feature maps or activation maps (e.g., another 2D matrix), which in turn contribute to the input of the next layer in the CNN. For example, the input signal is a 2D block with 256×256 samples, and the stride is 2 samples (e.g., stride 2). For a stride of 2, the convolutional kernel moves 2 samples along the X direction (e.g., horizontal) and/or the Y direction (e.g., vertical).

Multiple convolutional kernels can be applied to the input signal within the same convolutional layer to generate multiple feature maps, each representing a specific feature of the input signal. Generally, a convolutional layer with N channels (i.e., N convolutional kernels) (each kernel having M×M samples and a stride S) can be specified as Conv:MxM cN sS. For example, a convolutional layer with 192 channels (each kernel having 5×5 samples and a stride of 2) is specified as Conv:5x5 c192s2. Hidden layers can include deconvolutional layers that perform deconvolution (e.g., 2D deconvolution) (e.g., used in decoders). Deconvolution is the inverse operation of convolution. A deconvolutional layer with 192 channels (each deconvolutional kernel having 5×5 samples and a stride of 2) is specified as DeConv:5x5 c192 s2.

In various embodiments, CNNs offer the following advantages. The number of learnable parameters (i.e., parameters to be trained) in a CNN can be significantly smaller than the number of learnable parameters in a DNN (e.g., a feedforward DNN). In a CNN, a relatively large number of nodes can share the same filters (e.g., the same weights) and the same biases (if biases are used), thus reducing memory footprint because a single bias and a single weight vector can be used across all receptive fields sharing the same filters. For example, for an input signal with 100×100 samples, a convolutional layer with a kernel of 5×5 samples has 25 learnable parameters (e.g., weights). If biases are used, one channel uses 26 learnable parameters (e.g., 25 weights and one bias). If the convolutional layer has N channels, the total learnable parameters are 26xN. On the other hand, for a fully connected layer in a DNN, 100x100 (i.e., 10000) weights are used for each node in the next layer. If the next layer has L nodes, the total learnable parameters are 10000xL.

CNNs can also include one or more other layers, such as pooling layers, fully connected layers that connect each node in one layer to each node in another layer, normalization layers, etc. The layers in a CNN can be arranged in any suitable order and with any suitable architecture (e.g., feedforward architecture, recursive architecture). In one example, convolutional layers are followed by other layers, such as pooling layers, fully connected layers, normalization layers, etc.

Pooling layers can be used to reduce the dimensionality of data by combining the outputs of multiple nodes in one layer into a single node in the next layer. The pooling operation of a pooling layer with a feature map as input is described below. This description can be appropriately applied to other input signals. The feature map can be divided into sub-regions (e.g., rectangular sub-regions), and features within the respective sub-regions can be independently downsampled (or pooled) into single values, for example, by averaging in average pooling or maximizing in max pooling.

Pooling layers can perform pooling operations, such as local pooling, global pooling, max pooling, and average pooling. Pooling is a form of non-linear downsampling. Local pooling combines a small number of nodes in a feature map (e.g., a local cluster of nodes, such as a 2×2 node cluster). Global pooling, for example, can combine all nodes in a feature map.

Pooling layers can reduce the size of the representation, thereby reducing the number of parameters, memory footprint, and computational cost in a CNN. In one example, a pooling layer is inserted between consecutive convolutional layers in a CNN. In another example, the pooling layer is followed by an activation function, such as a Corrected Linear Unit (ReLU) layer. In yet another example, the pooling layer is omitted between consecutive convolutional layers in a CNN.

Normalization layers can be ReLU, Leaked ReLU, Generalized Division Normalization (GDN), Inverse GDN (IGDN), etc. ReLU can apply a non-saturating activation function to remove negative values from the input signal (e.g., feature map) by setting negative values to zero. For negative values, Leaked ReLU can have a small slope (e.g., 0.01) instead of a flat slope (e.g., 0). Therefore, if the value x is greater than 0, the output from Leaked ReLU is x. Otherwise, the output from Leaked ReLU is the value x multiplied by a small slope (e.g., 0.01). In one example, the slope is determined before training and therefore not learned during training.

In NN-based image compression methods, such as DNN-based or CNN-based image compression methods, instead of directly encoding the entire image, block or chunked encoding mechanisms can effectively compress images in DNN-based video coding standards (e.g., FVC). The entire image can be divided into blocks of the same (or different) size, and these blocks can be compressed individually. In one embodiment, the image can be divided into blocks of equal or unequal size. The segmented blocks, rather than the image itself, can be compressed. Figure 9A illustrates an example of chunked image encoding according to an embodiment of this disclosure. The image (980) can be divided into blocks, for example, blocks (981)-(996). For example, blocks (981)-(996) can be compressed according to the scan order. In the example shown in Figure 9A, blocks (981)-(989) have already been compressed, and blocks (990)-(996) will be compressed.

An image can be viewed as a single block. In one embodiment, the image is compressed without being divided into blocks. The entire image can be the input to an E2E NIC framework.

Figure 9B illustrates an exemplary NIC framework (900) (e.g., a NIC system) according to an embodiment of this disclosure. The NIC framework (900) may be based on a neural network, such as a DNN and/or a CNN. The NIC framework (900) may be used to compress (e.g., encode) blocks and decompress (e.g., decode or reconstruct) compressed blocks (e.g., encoded blocks). The NIC framework (900) may include two sub-neural networks implemented using a neural network, namely a first sub-neural network (951) and a second sub-NN (952).

The first sub-NN (951) can be analogous to an autoencoder and can be trained to generate compressed blocks of input block x and decompress the compressed blocks to obtain reconstructed blocks. The first sub-NN (951) can include multiple components (or modules), such as a master encoder neural network (or master encoder network) (911), a quantizer (912), an entropy encoder (913), an entropy decoder (914), and a master decoder neural network (or master encoder network) (915). Referring to Figure 9B, the master encoder network (911) can generate a latent or latent representation y from the input block x (e.g., the block to be compressed or encoded). In one example, the master encoder network (911) is implemented using a CNN. The relationship between the latent representation y and the input block x can be described using Equation 2.

Equation 2: y = _f1 (x; _θ1 )

Here, parameter _θ1 represents parameters, such as the weights and biases used in the convolution kernels of the master encoder network (911) (if biases are used in the master encoder network (911)).

A latent representation y can be quantized using a quantizer (912) to generate a quantized latent that can be compressed. For example, an entropy encoder (913) uses lossless compression to generate a compressed block (e.g., a coded block) (931), which is a compressed representation of the input block x. The entropy encoder (913) can use entropy coding techniques, such as Huffman coding, arithmetic coding, etc. In one example, the entropy encoder (913) uses arithmetic coding and is an arithmetic encoder. In one example, the coded block (931) is transmitted in the coded bitstream.

The encoded block (931) can be decompressed (e.g., entropy decoding) by the entropy decoder (914) to generate the output. The entropy decoder (914) can use an entropy coding technique corresponding to the entropy coding technique used in the entropy encoder (913), such as Huffman coding, arithmetic coding, etc. In one example, the entropy decoder (914) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (913), lossless decompression is used in the entropy decoder (914), and noise such as that generated due to the transmission of the encoded block (931) can be ignored. The output from the entropy decoder (914) is a quantized potential.

The master decoder network (915) can decode the quantized potential to generate reconstructed blocks. In one example, the master decoder network (915) is implemented using a CNN. The relationship between the reconstructed blocks (i.e., the output of the master decoder network (915)) and the quantized potential (i.e., the input of the master decoder network (915)) can be described using Equation 3.

Here, parameter _θ2 represents parameters, such as the weights and biases used in the convolutional kernels in the main decoder network (915) (if biases are used in the main decoder network (915)). Thus, the first sub-NN (951) can compress (e.g., encode) the input block x to obtain the encoded block (931) and decompress (e.g., decode) the encoded block (931) to obtain the reconstructed block. Due to the quantization loss introduced by the quantizer (912), the reconstructed block x can be different from the input block x.

The second sub-NN (952) can learn an entropy model (e.g., a prior probability model) on the quantized latent image used for entropy encoding. Thus, the entropy model can be a conditional entropy model, such as a Gaussian mixture model (GMM) or a Gaussian scaling model (GSM) dependent on the input block x. The second sub-NN (952) can include a context model NN (916), an entropy parameter NN (917), a super encoder (921), a quantizer (922), an entropy encoder (923), an entropy decoder (924), and a super decoder (925). The entropy model used in the context model NN (916) can be an autoregressive model of the latent image (e.g., a quantized latent image). In one example, the super encoder (921), quantizer (922), entropy encoder (923), entropy decoder (924), and super decoder (925) form a super neural network (e.g., a super NN). The super neural network can represent information useful for correcting context-based predictions. Data from the context model NN (916) and the super neural network can be combined using the entropy parameter NN (917). The entropy parameter NN(917) can generate parameters, such as the mean and scale parameters for entropy models like the conditional Gaussian entropy model (e.g., GMM).

Referring to Figure 9B, on the encoder side, the quantized latent image from the quantizer (912) is fed into the context model NN (916). On the decoder side, the quantized latent image from the entropy decoder (914) is fed into the context model NN (916). The context model NN (916) can be implemented using a neural network such as a CNN. The context model NN (916) can generate an output o _cm,i based on the context, which can include the quantized latent context available to the context model NN (916) and can include the previously quantized latent image on the encoder side or the previously entropy-decoded quantized latent image on the decoder side. The relationship between the output o _cm,i of the context model NN (916) and the input (e.g., ) can be described using Equation 4.

Here, parameter _θ3 represents parameters, such as the weights and biases used in the convolution kernel in the context model NN(916) (if biases are used in the context model NN(916)).

The outputs o _cm,i from the context model NN (916) and o _hc from the super decoder (925) are fed into the entropy parameter NN (917) to generate the output o _ep . The entropy parameter NN (917) can be implemented using a neural network such as a CNN. The relationship between the output o _ep of the entropy parameter NN (917) and the inputs (e.g., o _cm,i and o _hc ) can be described using Equation 5.

o _ep = f ₄ (o _cm,i ,o _hc ；θ ₄ ) Equation 5

Here, parameter _θ4 represents parameters, such as the weights and biases used in the convolutional kernel of the entropy parameter NN (917) (if biases are used in the entropy parameter NN (917)). The output _oep of the entropy parameter NN (917) can be used to determine (e.g., adjust) the entropy model, and thus the adjusted entropy model can depend on the input block x, for example, via the output _ohc from the super decoder (925). In one example, the output _oep includes parameters for adjusting the entropy model (e.g., GMM), such as the mean and scale parameters. Referring to Figure 9B, the entropy encoder (913) and entropy decoder (914) can use the entropy model (e.g., the conditional entropy model) in entropy encoding and entropy decoding, respectively.

The second sub-NN (952) can be described as follows. The latent y can be fed into the super encoder (921) to generate the super latent z. In one example, the super encoder (921) is implemented using a neural network such as a CNN. The relationship between the super latent z and the latent y can be described using Equation 6.

z = _f5 (y; _θ5 ) Equation 6

Here, parameter _θ5 represents parameters, such as the weights and biases used in the convolution kernel of the super encoder (921) (if biases are used in the super encoder (921)).

The quantizer (922) quantizes the super-latency z to generate a quantized latent that can be compressed, for example, by using lossless compression via an entropy encoder (923), to generate side information, such as coded bits (932) from the super-neural network. The entropy encoder (923) can use entropy coding techniques, such as Huffman coding, arithmetic coding, etc. In one example, the entropy encoder (923) uses arithmetic coding and is an arithmetic encoder. In one example, side information such as coded bits (932) can be transmitted in the coded bitstream, for example, along with coded blocks (931).

Side information, such as encoded bits (932), can be decompressed (e.g., entropy decoding) by an entropy decoder (924) to generate an output. The entropy decoder (924) can use entropy coding techniques, such as Huffman coding, arithmetic coding, etc. In one example, the entropy decoder (924) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (923), lossless decompression is used in the entropy decoder (924), and noise such as that caused by the transmission of side information can be ignored. The output from the entropy decoder (924) can be a quantized latent. The super decoder (925) can decode the quantized latent to generate an output to _hc . The relationship between the output to _hc and the quantized latent can be described by Equation 7.

Here, parameter _θ6 represents parameters, such as the weights and biases used in the convolution kernel in the superdecoder (925) (if biases are used in the superdecoder (925)).

As described above, compressed or encoded bits (932) can be added as side information to the encoded bitstream, which enables the entropy decoder (914) to use a conditional entropy model. Therefore, the entropy model can be block-dependent and spatially adaptive, and thus more accurate than a fixed entropy model.

The NIC framework (900) can be modified appropriately, for example, by omitting one or more components shown in FIG. 9B, modifying one or more components shown in FIG. 9B, and/or including one or more components not shown in FIG. 9B. In one example, the NIC framework using a fixed entropy model includes a first sub-NN (951) and does not include a second sub-NN (952). In one example, the NIC framework includes components in the NIC framework (900) other than the entropy encoder (923) and the entropy decoder (924).

In one embodiment, one or more components in the NIC framework (900) shown in Figure 9B are implemented using a neural network (e.g., a CNN). Each NN-based component in the NIC framework (e.g., the master encoder network (911), the master decoder network (915), the context model NN (916), the entropy parameter NN (917), the super encoder (921), or the super decoder (925)) may include any suitable architecture (e.g., with any suitable combination of layers), any suitable type of parameters (e.g., weights, biases, combinations of weights and biases, and/or the like), and any suitable number of parameters.

In one embodiment, the master encoder network (911), master decoder network (915), context model NN (916), entropy parameter NN (917), super encoder (921), and super decoder (925) are implemented using corresponding CNNs.

Figure 10 illustrates an exemplary CNN of a master encoder network (911) according to an embodiment of the present disclosure. For example, the master encoder network (911) includes four sets of layers, wherein each set of layers includes a 5x5 c192 s2 convolutional layer followed by a GDN layer. One or more layers shown in Figure 10 may be modified and/or omitted. Additional layers may be added to the master encoder network (911).

Figure 11 illustrates an exemplary CNN of a master decoder network (915) according to an embodiment of the present disclosure. For example, the master decoder network (915) includes three sets of layers, each set comprising a 5x5 c192 s2 deconvolutional layer followed by an IGDN layer. Furthermore, after the three sets of layers is a 5x5 c3 s2 deconvolutional layer, followed by an IGDN layer. One or more layers shown in Figure 11 may be modified and/or omitted. Additional layers may be added to the master decoder network (915).

Figure 12 illustrates an exemplary CNN of a super encoder (921) according to an embodiment of the present disclosure. For example, the super encoder (921) includes a 3x3 c192 s1 convolutional layer followed by a leaky ReLU, a 5x5 c192 s2 convolutional layer followed by a leaky ReLU, and another 5x5 c192 s2 convolutional layer. One or more layers shown in Figure 12 may be modified and/or omitted. Additional layers may be added to the super encoder (921).

Figure 13 illustrates an exemplary CNN of a super decoder (925) according to an embodiment of the present disclosure. For example, the super decoder (925) includes a 5x5 c192 s2 deconvolutional layer followed by a leaky ReLU, a 5x5 c288 s2 convolutional layer followed by a leaky ReLU, and a 3x3 c384 s1 convolutional layer. One or more layers shown in Figure 13 may be modified and/or omitted. Additional layers may be added to the super encoder (925).

Figure 14 illustrates an exemplary CNN of a context model NN (916) according to an embodiment of the present disclosure. For example, the context model NN (916) includes a masked convolution 5x5 c384 s1 for context prediction, so the context in Equation 4 includes a finite context (e.g., a 5×5 convolutional kernel). The convolutional layers in Figure 14 can be modified. Additional layers can be added to the context model NN (916).

Figure 15 illustrates an exemplary CNN with entropy parameter NN (917) according to an embodiment of this disclosure. For example, the entropy parameter NN (917) includes a 1x1 convolutional layer c640 s1 followed by a leaky ReLU, a 1x1 convolutional layer c512 s1 followed by a leaky ReLU, and a 1x1 convolutional layer c384 s1. One or more layers shown in Figure 15 may be modified and/or omitted. Additional layers may be added to the entropy parameter NN (917).

The NIC framework (900) can be implemented using a CNN, as described with reference to Figures 10-15. The NIC framework (900) can be appropriately adapted so that one or more components of the NIC framework (900) (e.g., (911), (915), (916), (917), (921) and/or (925)) can be implemented using any suitable type of neural network (e.g., a CNN-based or non-CNN-based neural network). One or more other components of the NIC framework (900) can be implemented using neural networks.

The NIC framework (900) including neural networks (e.g., CNNs) can be trained to learn the parameters used in the neural networks. For example, when using a CNN, the parameters represented by _θ1 - _θ6 can be learned during training, such as the weights and biases used in the convolutional kernels in the main encoder network (911) (if biases are used in the main encoder network (911)), the weights and biases used in the convolutional kernels in the main decoder network (915) (if biases are used in the main decoder network (915)), the weights and biases used in the convolutional kernels in the super encoder (921) (if biases are used in the super encoder (921)), the weights and biases used in the convolutional kernels in the super decoder (925) (if biases are used in the super decoder (925)), the weights and biases used in the convolutional kernels in the context model NN (916) (if biases are used in the context model NN (916)), and the weights and biases used in the convolutional kernels in the entropy parameter NN (917) (if biases are used in the entropy parameter NN (917)).

In one example, referring to Figure 10, the master encoder network (911) comprises four convolutional layers, each with a 5×5 convolutional kernel and 192 channels. Therefore, the number of weights used in the convolutional kernels of the master encoder network (911) is 19200 (i.e., 4x5x5x192). The parameters used in the master encoder network (911) include 19200 weights and optional biases. Additional parameters may be included when biases and/or additional neural networks are used in the master encoder network (911).

Referring to Figure 9B, the NIC framework (900) includes at least one component or module built on a neural network. This at least one component may include one or more of a master encoder network (911), a master decoder network (915), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). This at least one component can be trained individually. In one example, the training process is used to learn the parameters of each component separately. The at least one component can also be trained as a group. In one example, the training process is used to collectively learn the parameters of a subset of at least one component. In one example, the training process is used to learn the parameters of all at least one component, hence the term E2E optimization.

During the training of one or more components in the NIC framework (900), the weights (or weight coefficients) of one or more components can be initialized. In one example, the weights are initialized based on a pre-trained corresponding neural network model (e.g., a DNN model, a CNN model). In another example, the weights are initialized by setting them to random numbers.

For example, after initializing the weights, a set of training blocks can be used to train one or more components. This set of training blocks can include any suitable block with any appropriate size. In some examples, the set of training blocks includes blocks of the original image, natural image, computer-generated image, etc., in the spatial domain. In some examples, the set of training blocks includes blocks of residual blocks or residual images with residual data in the spatial domain. The residual data can be computed by a residual calculator (e.g., a residual calculator (723)). In some examples, the original image and/or the residual image including the residual data can be directly used to train the neural network in the NIC framework. Therefore, the original image, the residual image, blocks from the original image, and/or blocks from the residual image can be used to train the neural network in the NIC framework.

For the sake of brevity, the training block is used as an example to describe the training process below. This description can be appropriately applied to the training block. The training block t in this set of training blocks can be encoded using the encoding process in Figure 9B to generate a compressed representation (e.g., encoded information, such as a bitstream). The encoded information can be computed and reconstructed using the decoding process described in Figure 9B.

For the NIC framework (900), two competing objectives are balanced, such as reconstruction quality and bit consumption. A quality loss function (e.g., distortion or distortion loss) can be used to indicate reconstruction quality, such as the difference between the reconstructed block (e.g., the reconstructed block) and the original block (e.g., the training block t). A rate (or rate loss) R can be used to indicate the bit consumption of the compressed representation. In one example, the rate loss R also includes, for example, side information used when determining the context model.

For neural image compression, a differentiable approximation of quantization can be used in E2E optimization. In various examples, noise injection is used to simulate quantization during the training process of neural network-based image compression, so quantization is simulated by noise injection rather than performed by a quantizer (e.g., quantizer (912)). Therefore, training with noise injection can variably approximate the quantization error. A bit-per-pixel (BPP) estimator can be used to simulate an entropy encoder, so entropy encoding is simulated by the BPP estimator rather than performed by an entropy encoder (e.g., (913)) and an entropy decoder (e.g., (914)). Thus, for example, the rate loss R in the loss function L shown in Equation 1 during training can be estimated based on noise injection and the BPP estimator. In general, a higher rate R can achieve a lower distortion D, while a lower rate R results in a higher distortion D. The tradeoff hyperparameter λ in Equation 1 can be used to optimize the common R-D loss L, where L can be optimized as a sum of λD and R. The training process can be used to tune the parameters of one or more components (e.g., (911), (915)) in the NIC framework (900) such that the common R-D loss L is minimized or optimized. In one example, a trade-off hyperparameter λ can be used to optimize the joint rate-distortion (R-D) loss, as follows:

Here, E measures the distortion of the decoded block residual compared to the original block residual before encoding, and it acts as the regularization loss for both the residual encoding/decoding DNN and the encoding/decoding DNN. β is a hyperparameter used to balance the importance of the regularization loss.

Various models can be used to determine the distortion loss D and the rate loss R, thereby determining the common R-D loss L in Equation 1. In one example, the distortion loss is expressed as the peak signal-to-noise ratio (PSNR), which is a measure based on mean squared error, multi-scale structural similarity (MS-SSIM) quality index, a weighted combination of PSNR and MS-SSIM, etc.

In one example, the goal of the training process is to train an encoding neural network (e.g., an encoding DNN), such as a video encoder to be used on the encoder side, and to train a decoding neural network (e.g., a decoding DNN), such as a video decoder to be used on the decoder side. In one example, referring to Figure 9B, the encoding neural network may include a main encoder network (911), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The decoding neural network may include a main decoder network (915), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The video encoder and/or video decoder may include other components based on and/or not based on NNs.

The NIC framework can be trained in an E2E manner (e.g., the NIC framework (900)). In one example, the encoding neural network and the decoding neural network are jointly updated in an E2E manner based on backpropagation gradients during training.

After training the parameters of the neural network in the NIC framework (900), one or more components in the NIC framework (900) can be used to encode and/or decode blocks. In one embodiment, on the encoder side, a video encoder is configured to encode an input block x into a coded block (931) to be transmitted in a bitstream. The video encoder may include multiple components in the NIC framework (900). In one embodiment, on the decoder side, a corresponding video decoder is configured to decode the coded block (931) in the bitstream into a reconstructed block x. The video decoder may include multiple components in the NIC framework (900).

In one example, for instance, when using content-adaptive online training, the video encoder includes all components of the NIC framework (900).

Figure 16A illustrates an exemplary video encoder (1600A) according to an embodiment of the present disclosure. The video encoder (1600A) includes a master encoder network (911), a quantizer (912), an entropy encoder (913), and a second sub-NN (952) described with reference to Figure 9B, the details of which are omitted for brevity. Figure 16B illustrates an exemplary video decoder (1600B) according to an embodiment of the present disclosure. The video decoder (1600B) may correspond to the video encoder (1600A). The video decoder (1600B) may include a master decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924), and a super decoder (925). Referring to Figures 16A-16B, on the encoder side, the video encoder (1600A) may generate coded blocks (931) and coded bits (932) to be transmitted in the bitstream. On the decoder side, the video decoder (1600B) can receive and decode coded blocks (931) and coded bits (932).

Figures 17-18 illustrate exemplary video encoders (1700) and corresponding video decoders (1800) according to embodiments of the present disclosure. Referring to Figure 17, the encoder (1700) includes a master encoder network (911), a quantizer (912), and an entropy encoder (913). An example of the master encoder network (911), quantizer (912), and entropy encoder (913) is described with reference to Figure 9B. Referring to Figure 18, the video decoder (1800) includes a master decoder network (915) and an entropy decoder (914). An example of the master decoder network (915) and entropy decoder (914) is described with reference to Figure 9B. Referring to Figures 17 and 18, the video encoder (1700) can generate encoded blocks (931) that will be transmitted in a bitstream. The video decoder (1800) can receive and decode the encoded blocks (931).

As described above, the NIC framework (900), including a video encoder and a video decoder, can be trained based on images and/or blocks in the set of training images. In some examples, one or more blocks to be compressed (e.g., encoded) and/or transmitted have properties significantly different from those of the set of training blocks. Therefore, directly encoding and decoding one or more blocks using the video encoder and video decoder trained based on the set of training blocks results in relatively poor R-D loss L (e.g., relatively large distortion and/or relatively large bit rate). Therefore, aspects of this disclosure describe content-adaptive online training methods for NICs, such as a chunked content-adaptive online training method for NICs.

In the chunked content adaptive online training method, the input image can be divided into chunks, and one or more chunks can be used to update one or more parameters in a pre-trained NIC framework with one or more replacement parameters by optimizing rate-distortion performance. Neural network update information indicating one or more replacement parameters, or a subset of one or more replacement parameters, can be encoded into a bitstream along with the encoded one or more chunks. On the decoder side, the video decoder can decode the encoded one or more chunks and can achieve better compression performance by using one or more replacement parameters, or a subset of one or more replacement parameters. The chunked content adaptive online training method can be used as a preprocessing step (e.g., a pre-encoding step) to improve the compression performance of the pre-trained E2ENIC compression method.

To distinguish between the training process based on this set of training blocks and the content-adaptive online training process based on one or more blocks to be compressed (e.g., encoded) and/or transmitted, the NIC framework (900), video encoder, and video decoder trained by this set of training blocks are referred to as the pre-trained NIC framework (900), pre-trained video encoder, and pre-trained video decoder, respectively. Parameters in the pre-trained NIC framework (900), pre-trained video encoder, or pre-trained video decoder are referred to as NIC pre-training parameters, encoder pre-training parameters, and decoder pre-training parameters, respectively. In one example, the NIC pre-training parameters include both encoder pre-training parameters and decoder pre-training parameters. In one example, the encoder pre-training parameters and decoder pre-training parameters do not overlap, wherein none of the encoder pre-training parameters are included in the decoder pre-training parameters. For example, the encoder pre-training parameters in (1700) (e.g., pre-training parameters in the main encoder network (911)) and the decoder pre-training parameters in (1800) (e.g., pre-training parameters in the main decoder network (915)) do not overlap. In one example, the encoder pre-training parameters and decoder pre-training parameters overlap, wherein at least one of the encoder pre-training parameters is included in the decoder pre-training parameters. For example, the encoder pre-training parameters (e.g., pre-training parameters in the context model NN(916)) in (1600A) and the decoder pre-training parameters (e.g., pre-training parameters in the context model NN(916)) in (1600B) overlap. NIC pre-training parameters can be obtained based on blocks and/or images in this set of training blocks.

The content-adaptive online training process can be referred to as a fine-tuning process, as described below. One or more pre-trained parameters in the pre-trained NIC framework (900) can be further trained (e.g., fine-tuned) based on one or more blocks to be encoded and/or transmitted, wherein one or more blocks may differ from this set of training blocks. One or more pre-trained parameters used in the NIC pre-trained parameters can be fine-tuned by optimizing the common R-D loss L based on one or more blocks. One or more pre-trained parameters that have been fine-tuned by one or more blocks are referred to as one or more replacement parameters or one or more fine-tuning parameters. In one embodiment, after one or more pre-trained parameters in the NIC pre-trained parameters have been fine-tuned (e.g., replaced) by one or more replacement parameters, neural network update information is encoded into a bitstream to indicate one or more replacement parameters or a subset of one or more replacement parameters. In one example, the NIC framework (900) is updated (or fine-tuned) where one or more pre-trained parameters are replaced by one or more replacement parameters, respectively.

In the first case, the one or more pre-trained parameters comprise a first subset of one or more pre-trained parameters and a second subset of one or more pre-trained parameters. The one or more replacement parameters comprise a first subset of one or more replacement parameters and a second subset of one or more replacement parameters.

A first subset of one or more pre-trained parameters is used in a pre-trained video encoder and, for example, is replaced by a first subset of one or more replacement parameters during training. Therefore, through the training process, the pre-trained video encoder is updated to a newer video encoder. Neural network update information can indicate a second subset of one or more replacement parameters, which will replace the second subset of one or more replacement parameters. One or more blocks can be encoded using the updated video encoder and transmitted in a bitstream with the neural network update information.

On the decoder side, a second subset of one or more pre-trained parameters is used in the pre-trained video decoder. In one embodiment, the pre-trained video decoder receives and decodes neural network update information to determine a second subset of one or more replacement parameters. When the second subset of one or more pre-trained parameters in the pre-trained video decoder is replaced by a second subset of one or more replacement parameters, the pre-trained video decoder is updated to an updated video decoder. The updated video decoder can then be used to decode one or more coded blocks.

Figures 16A-16B illustrate examples of the first case. For example, one or more pre-trained parameters include N1 pre-trained parameters in the pre-trained context model NN (916) and N2 pre-trained parameters in the pre-trained master decoder network (915). Therefore, a first subset of the one or more pre-trained parameters includes the N1 pre-trained parameters, and a second subset of the one or more pre-trained parameters is the same as the one or more pre-trained parameters. Therefore, the N1 pre-trained parameters in the pre-trained context model NN (916) can be replaced by the replacement parameters corresponding to N1, so that the pre-trained video encoder (1600A) can be updated to the updated video encoder (1600A). The pre-trained context model NN (916) is also updated to the updated context model NN (916). On the decoder side, the N1 pre-trained parameters can be replaced by the replacement parameters corresponding to N1, and the N2 pre-trained parameters can be replaced by the replacement parameters corresponding to N2, updating the pre-trained context model NN (916) to the updated context model NN (916) and updating the pre-trained master decoder network (915) to the updated master decoder network (915). Therefore, the pre-trained video decoder (1600B) can be updated to the updated video decoder (1600B).

In the second case, one or more pre-trained parameters are not used in the pre-trained video encoder on the encoder side. Instead, one or more pre-trained parameters are used in the pre-trained video decoder on the decoder side. Therefore, the pre-trained video encoder is not updated and continues to be used after the training process. In one embodiment, neural network update information indicates one or more replacement parameters. One or more blocks can be encoded using the pre-trained video encoder and transmitted in a bitstream with neural network update information.

On the decoder side, the pre-trained video decoder can receive and decode neural network update information to determine one or more replacement parameters. When one or more pre-trained parameters in the pre-trained video decoder are replaced by one or more replacement parameters, the pre-trained video decoder is updated to the updated video decoder. The updated video decoder can then be used to decode one or more coded blocks.

Figures 16A-16B illustrate examples of the second scenario. For instance, one or more pre-trained parameters include N2 pre-trained parameters (915) in the pre-trained master decoder network. Therefore, no one or more pre-trained parameters are used in the pre-trained video encoder on the encoder side (e.g., the pre-trained video encoder (1600A)). Thus, after the training process, the pre-trained video encoder (1600A) remains a pre-trained video encoder. On the decoder side, the N2 pre-trained parameters can be replaced by the corresponding replacement parameters, which updates the pre-trained master decoder network (915) to the updated master decoder network (915). Therefore, the pre-trained video decoder (1600B) can be updated to the updated video decoder (1600B).

In the third case, one or more pre-trained parameters are used in the pre-trained video encoder and are replaced, for example, by one or more replacement parameters during training. Therefore, through the training process, the pre-trained video encoder is updated to the updated video encoder. One or more blocks can be encoded using the updated video encoder and transmitted in a bitstream. No neural network update information is encoded in the bitstream. At the decoder end, the pre-trained video decoder is not updated and remains the pre-trained video decoder. The pre-trained video decoder can be used to decode one or more coded blocks.

Figures 16A-16B illustrate an example of the third case. For example, one or more pre-trained parameters are in the pre-trained master encoder network (911). Therefore, one or more pre-trained parameters in the pre-trained master encoder network (911) can be replaced by one or more replacement parameters, such that the pre-trained video encoder (1600A) can be updated to the updated video encoder (1600A). The pre-trained master encoder network (911) is also updated to the updated master encoder network (911). On the decoder side, the pre-trained video decoder (1600B) is not updated.

In various examples, such as those described in the first, second, and third scenarios, video decoding can be performed by pre-trained decoders with different capabilities, including decoders with and without the ability to update pre-trained parameters.

In one example, compression performance can be improved by encoding one or more blocks with an updated video encoder and/or an updated video decoder compared to encoding one or more blocks with a pre-trained video encoder and a pre-trained video decoder. Therefore, content-adaptive online training methods can be used to adapt a pre-trained NIC framework (e.g., a pre-trained NIC framework (900)) to the target block content (e.g., one or more blocks to be transmitted), thereby fine-tuning the pre-trained NIC framework. Thus, the video encoder on the encoder side and/or the video decoder on the decoder side can be updated.

Content-adaptive online training methods can be used as a preprocessing step (e.g., a precoding step) to improve the compression performance of pre-trained E2E NIC compression methods.

In one embodiment, one or more blocks comprise a single input block, and a fine-tuning process is performed using this single input block. The NIC framework (900) is trained and updated (e.g., fine-tuned) based on the single input block. Updates to the video encoder on the encoder side and/or the video decoder on the decoder side can be used to encode the single input block and optional other input blocks. Neural network update information can be encoded into the bitstream along with the encoded single input block.

In one embodiment, one or more blocks include multiple input blocks, and a fine-tuning process is performed using the multiple input blocks. The NIC framework (900) is trained and updated (e.g., fine-tuned) based on the multiple input blocks. Updates on the encoder side and/or the decoder side can be used to encode the multiple input blocks and optional additional input blocks. Neural network update information can be encoded into the bitstream along with the encoded multiple input blocks.

The rate loss R can increase with signaling of neural network update information in the bitstream. When one or more blocks comprise a single input block, neural network update information is signaled for each coded block, and the first increase in rate loss R indicates the increase in rate loss R due to signaling neural network update information for each block. When one or more blocks comprise multiple input blocks, neural network update information is signaled to multiple input images and shared by multiple input blocks, and the second increase in rate loss R indicates the increase in rate loss R due to signaling neural network update information for each block. Because the neural network update information is shared by multiple input blocks, the second increase in rate loss R can be less than the first increase in rate loss R. Therefore, in some examples, using multiple input blocks to fine-tune the NIC framework may be advantageous.

In one embodiment, one or more pre-trained parameters to be updated are in a component of the pre-trained NIC framework (900). Therefore, one component of the pre-trained NIC framework (900) is updated based on one or more replacement parameters, and the other components of the pre-trained NIC framework (900) are not updated.

A component can be a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained master encoder network (911), a pre-trained master decoder network (915), a pre-trained super encoder (921), or a pre-trained super decoder (925). The pre-trained video encoder and/or pre-trained video decoder can be updated depending on which component in the pre-trained NIC framework (900) is updated.

In one example, one or more pre-trained parameters to be updated are in the pre-trained context model NN (916), so the pre-trained context model NN (916) is updated, but the remaining components (911), (915), (921), (917), and (925) are not updated. In another example, the pre-trained video encoder on the encoder side and the pre-trained video decoder on the decoder side include the pre-trained context model NN (916), so the pre-trained video encoder and the pre-trained video decoder are updated.

In one example, one or more pre-trained parameters to be updated are in the pre-trained superdecoder (925), so the pre-trained superdecoder (925) is updated, but the remaining components (911), (915), (916), (917), and (921) are not updated. Therefore, the pre-trained video encoder is not updated, but the pre-trained video decoder is updated.

In one embodiment, one or more pre-trained parameters to be updated are in multiple components of a pre-trained NIC framework (900). Therefore, multiple components of the pre-trained NIC framework (900) are updated based on one or more replacement parameters. In one example, the multiple components of the pre-trained NIC framework (900) include all components configured with neural networks (e.g., DNN, CNN). In one example, the multiple components of the pre-trained NIC framework (900) include CNN-based components: a pre-trained master encoder network (911), a pre-trained master decoder network (915), a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained super encoder (921), and a pre-trained super decoder (925).

As described above, in one example, one or more pre-trained parameters to be updated are in the pre-trained video encoder (900) of the pre-trained NIC framework. In one example, one or more pre-trained parameters to be updated are in the pre-trained video decoder of the NIC framework (900). In one example, one or more pre-trained parameters to be updated are in both the pre-trained video encoder and the pre-trained video decoder of the pre-trained NIC framework (900).

The NIC framework (900) can be based on a neural network; for example, one or more components in the NIC framework (900) may include a neural network, such as a CNN, DNN, etc. As mentioned above, the neural network can be specified by different types of parameters, such as weights, biases, etc. Each neural network-based component in the NIC framework (900) (e.g., the context model NN (916), the entropy parameter NN (917), the master encoder network (911), the master decoder network (915), the super encoder (921), or the super decoder (925)) can be configured with appropriate parameters, such as corresponding weights, biases, or combinations of weights and biases. When using a CNN, weights may include elements in the convolutional kernel. One or more types of parameters can be used to specify the neural network. In one embodiment, one or more pre-trained parameters to be updated are bias terms, and only bias terms are replaced by one or more replacement parameters. In one embodiment, one or more pre-trained parameters to be updated are weights, and only weights are replaced by one or more replacement parameters. In one embodiment, one or more pre-trained parameters to be updated include weights and bias terms, and all pre-trained parameters including weights and bias terms are replaced by one or more replacement parameters. In one embodiment, other parameters can be used to specify the neural network, and these other parameters can be fine-tuned.

The fine-tuning process can include multiple stages (e.g., iterations), during which one or more pre-trained parameters are updated. The fine-tuning process can stop when the training loss flattens out or is about to flatten out. In one example, the fine-tuning process stops when the training loss (e.g., R-D loss L) falls below a first threshold. In another example, the fine-tuning process stops when the difference between two consecutive training losses falls below a second threshold.

Two hyperparameters (e.g., step size and maximum number of steps) can be used together with a loss function (e.g., R-D loss L) in the fine-tuning process. The maximum number of iterations can be used as a threshold for the maximum number of iterations to terminate the fine-tuning process. In one example, the fine-tuning process stops when the maximum number of iterations is reached.

The step size can indicate the learning rate during an online training process (e.g., an online fine-tuning process). The step size can be used in gradient descent algorithms or backpropagation calculations performed during fine-tuning. Any suitable method can be used to determine the step size.

The stride size of each block in the image can be different. In one embodiment, different strides can be assigned to the image to achieve better compression results (e.g., better R-D loss L).

In some examples, video encoders and decoders based on NIC frameworks (e.g., NIC framework (900)) can directly encode and decode images. Therefore, the chunked content adaptive online training method can be adapted to update certain parameters in the video encoder and/or video decoder by directly using one or more images to update certain parameters in the NIC framework. Different images can have different strides to achieve optimized compression results.

In one embodiment, different step sizes are used for blocks with different types of content to achieve optimal results. Different types can refer to different differences. In one example, the step size is determined based on the differences of the blocks used to update the NIC framework. For example, the step size for blocks with high differences is greater than the step size for blocks with low differences, where high differences are greater than low differences.

In one embodiment, the step size is selected based on features of the block or image, such as the RGB difference of the block. In another embodiment, the step size is selected based on the RD performance of the block (e.g., R-D loss L). Multiple sets of replacement parameters can be generated based on different step sizes, and the set with better compression performance (e.g., lower R-D loss) can be selected.

In one embodiment, the first step size can be used to run a certain number of iterations (e.g., 100). Then, a second step size (e.g., the first step size plus or minus a size increment) can be used to run a certain number of iterations. The results of the first and second step sizes can be compared to determine the step size to use. More than two step sizes can be tested to determine the optimal step size.

During fine-tuning, the step size can vary. The step size may have an initial value at the start of the fine-tuning process and may be reduced (e.g., halved) at a later stage (e.g., after a certain number of iterations) to achieve finer tuning. During iterative online training, the step size or learning rate can be changed by the scheduler. The scheduler may include parameter tuning methods for adjusting the step size. The scheduler can determine the value of the step size such that it can be increased, decreased, or kept constant over multiple intervals. In one example, the learning rate is changed by the scheduler at each step. A single scheduler or multiple different schedulers can be used for different blocks. Therefore, multiple sets of replacement parameters can be generated based on multiple schedulers, and the set with better compression performance (e.g., lower R-D loss) can be selected from among the multiple sets of replacement parameters.

In one embodiment, multiple learning rate schedules are assigned to different blocks to achieve better compression results. In another embodiment, all blocks in the image share the same learning rate table. In yet another embodiment, the learning rate table is selected based on block characteristics, such as the RGB differences between blocks. In yet another embodiment, the learning rate table is selected based on the block's RD performance.

In one embodiment, different blocks can be used to update different parameters in different components of the NIC framework (e.g., the context model NN (916) or the super decoder (925)). For example, the first block is used to update the parameters (916) in the context model NN, and the second block is used to update the parameters (925) in the super decoder.

In one embodiment, different blocks can be used to update different types of parameters (e.g., biases or weights) in the NIC framework. For example, a first block is used to update at least one bias in one or more neural networks in the NIC framework, and a second block is used to update at least one weight in one or more neural networks in the NIC framework.

In one embodiment, multiple blocks in an image (e.g., all blocks) update the same one or more parameters.

In one embodiment, one or more parameters to be updated are selected based on block characteristics, such as the RGB differences of the block. In another embodiment, one or more parameters to be updated are selected based on the RD performance of the block.

At the end of the fine-tuning process, one or more updated parameters can be calculated for the corresponding one or more replacement parameters. In one embodiment, one or more updated parameters are calculated as the difference between one or more replacement parameters and the corresponding one or more pre-trained parameters. In another embodiment, the one or more updated parameters are one or more replacement parameters.

In one embodiment, one or more updated parameters can be generated from one or more replacement parameters, for example, using a specific linear or nonlinear transformation, and the one or more updated parameters are representative parameters generated based on the one or more replacement parameters. The one or more replacement parameters are transformed into one or more updated parameters for better compression.

One or more update parameters, a first subset, correspond to one or more replacement parameters, and one or more update parameters, a second subset, correspond to one or more replacement parameters, a second subset.

In one embodiment, different blocks have different relationships between one or more updated parameters and one or more replacement parameters. For example, for a first block, one or more updated parameters are computed as the difference between one or more replacement parameters and the corresponding one or more pre-trained parameters. For a second block, the one or more updated parameters are, respectively, one or more replacement parameters.

In one embodiment, multiple blocks in an image (e.g., all blocks) have the same relationship between one or more update parameters and one or more replacement parameters.

In one embodiment, the relationship between one or more update parameters and one or more replacement parameters is selected based on block characteristics, such as the RGB differences of the block. In another embodiment, the relationship between one or more update parameters and one or more replacement parameters is selected based on the block's RD performance.

In one example, one or more updated parameters may be compressed, for example, using algorithms such as LZMA2 or bzip2, where LZMA2 is a variant of the Lempel–Ziv–Markov chain algorithm (LZMA). In another example, compression is omitted for one or more updated parameters. In some embodiments, one or more updated parameters, or a second subset of one or more updated parameters, may be encoded into a bitstream as neural network update information, wherein the neural network update information indicates one or more replacement parameters, or a second subset of one or more replacement parameters.

In one embodiment, the compression method for one or more updated parameters is different for different blocks. For example, LZMA2 is used to compress one or more updated parameters for the first block, and bzip2 is used to compress one or more updated parameters for the second block. In one embodiment, the same compression method is used to compress one or more updated parameters for multiple blocks (e.g., all blocks) in the image. In one embodiment, the compression method is selected based on block characteristics, such as the RGB differences of the block. In one embodiment, the compression method is selected based on the RD performance of the block.

Following the fine-tuning process, in some examples, the pre-trained video encoder on the encoder side can be updated or fine-tuned based on (i) a first subset of one or more replacement parameters or (ii) one or more replacement parameters. The updated video encoder can be used to encode the input image (e.g., one or more blocks used in the fine-tuning process) into a bitstream. Therefore, the bitstream includes encoded blocks and neural network update information.

If applicable, in one example, the neural network update information is decoded (e.g., decompressed) by a pre-trained video decoder to obtain one or more updated parameters or a second subset of one or more updated parameters. In one example, one or more replacement parameters or a second subset of one or more replacement parameters can be obtained based on the relationship between one or more updated parameters and the aforementioned one or more replacement parameters. As described above, the pre-trained video decoder can be fine-tuned, and the decoded updated video can be used to decode the coded block.

The NIC framework can include any type of neural network and use any neural network-based image compression method, such as context super-optimal encoder-decoder framework (e.g., the NIC framework shown in Figure (9B)), scale super-optimal encoder-decoder framework, Gaussian mixture likelihood framework and its variants, RNN-based recursive compression method and its variants, etc.

Compared to related E2E image compression methods, the adaptive online training method and apparatus disclosed herein have the following advantages: They utilize an adaptive online training mechanism to improve NIC coding efficiency. A flexible and general framework can be used to adapt to various types of pre-trained frameworks and quality metrics. For example, certain pre-training parameters in various types of pre-trained frameworks can be replaced by online training on the blocks to be encoded and transmitted.

Figure 19 shows a flowchart outlining a process (1900) according to an embodiment of the present disclosure. The process (1900) can be used to encode blocks, such as blocks in a raw image or blocks in a residual image. In various embodiments, the process (1900) is executed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video encoder (1600A), or processing circuitry performing the functions of a video encoder (1700). In one example, the processing circuitry performs a combination of (i) one of the video encoders (403), (603), and (703) and (ii) one of the functions of the video encoder (1600A) and the video encoder (1700). In some embodiments, the process (1900) is implemented in software instructions, so that the processing circuitry executes the process (1900) when the software instructions are executed. The process begins at (S1901). In one example, the NIC framework is based on a neural network. In one example, the NIC framework is the NIC framework (900) described with reference to Figure 9B. The NIC framework can be based on a CNN, for example, as described in Figures 10-15. As mentioned above, the video encoder (e.g., (1600A) or (1700)) and the corresponding video decoder (e.g., (1600B) or (1800)) can include multiple components in the NIC framework. The neural network-based NIC framework is pre-trained, thereby pre-training the video encoder and video decoder. Process (1900) proceeds to (S1910).

In (S1910), a fine-tuning process is performed on the NIC framework based on one or more blocks (or input blocks). The input block can be any suitable block of any appropriate size. In some examples, the input block includes blocks from the original image, natural image, computer-generated image, etc., in the spatial domain.

In some examples, the input block includes residual data in the spatial domain, calculated, for example, by a residual calculator (e.g., residual calculator (723)). Components in various devices can be appropriately combined to implement (S1910) that the residual block from the residual calculator can be fed into the main encoder network (911) in the NIC framework, for example, referring to Figures 7 and 9B.

As described above, one or more parameters (e.g., one or more pre-trained parameters) in one or more neural networks (e.g., one or more pre-trained neural networks) within a NIC framework (e.g., a pre-trained NIC framework) can be updated to one or more replacement parameters, respectively. In one embodiment, during the training process described in (S1910), for example, at each step, one or more parameters in one or more neural networks are updated.

In one embodiment, at least one neural network in the video encoder (e.g., a pre-trained video encoder) is configured with a first subset of one or more pre-trained parameters, so that the at least one neural network in the video encoder can be updated based on a corresponding first subset of one or more replacement parameters. In one example, the first subset of one or more replacement parameters includes all one or more replacement parameters. In one example, at least one neural network in the video encoder is updated as the first subset of one or more pre-trained parameters is replaced by the first subset of one or more replacement parameters, respectively. In one example, at least one neural network in the video encoder is iteratively updated during fine-tuning. In one example, one or more pre-trained parameters are not included in the video encoder, so the video encoder is not updated and remains a pre-trained video encoder.

At (S1920), a video encoder having at least one updated neural network can be used to encode one of one or more blocks, wherein the video encoder is configured with a first subset of one or more replacement parameters. In one example, one of one or more blocks is encoded after updating at least one neural network in the video encoder.

Step (S1920) can be modified appropriately. For example, when one or more replacement parameters are not included in at least one neural network in the video encoder, the video encoder is not updated, so a pre-trained video encoder (e.g., a video encoder including at least one pre-trained neural network) can be used to encode one of the one or more blocks.

In step (S1930), neural network update information indicating a second subset of one or more replacement parameters can be encoded into the bitstream. In one example, the second subset of one or more replacement parameters will be used to update at least one neural network in the video decoder on the decoder side. Step (S1930) can be omitted, and for example, if the second subset of one or more replacement parameters does not include parameters and there is no signaling neural network update information in the bitstream, then the neural network in the video decoder is not updated.

In (S1940), a bitstream including coded blocks from one or more blocks and neural network update information can be transmitted. Step (S1940) can be modified appropriately. For example, if step (S1930) is omitted, the bitstream will not include neural network update information. The process (1900) proceeds to (S1999) and terminates.

The process (1900) can be adapted to various situations, and the steps in the process (1900) can be adjusted accordingly. One or more steps in the process (1900) can be modified, omitted, repeated, and/or combined. The process (1900) can be implemented using any suitable order. Additional steps can be added. For example, in addition to encoding one of the one or more blocks, one or more blocks can be encoded in (S1920) and transmitted in (S1940).

In some examples of process (1900), one of one or more blocks is encoded by an updated video encoder and transmitted in a bitstream. Because the fine-tuning process is based on one or more blocks, it is context-based, as it is based on the context to be encoded.

In some examples, the neural network update information also indicates what a second subset of one or more pre-trained parameters (or a corresponding second subset of one or more replacement parameters) is, such that the corresponding pre-trained parameters in the video decoder can be updated. The neural network update information may indicate component information (e.g., (915)), layer information (e.g., fourth layer DeConv: 5x5 c3 s2), channel information (e.g., second channel), etc., of the second subset of one or more pre-trained parameters. Thus, referring to Figure 11, the second subset of one or more replacement parameters includes the convolutional kernel (915) of the second channel DeConv: 5x5 c3 s2 in the main decoder network. Therefore, the convolutional kernel of the second channel DeConv: 5x5 c3 s2 in the pre-trained main decoder network (915) is updated. In some examples, the component information (e.g., (915)), layer information (e.g., fourth layer DeConv: 5x5 c3 s2), channel information (e.g., second channel), etc., of the second subset of one or more pre-trained parameters are predetermined and stored in the pre-trained video decoder, and therefore not signaled.

Figure 20 shows a flowchart outlining a process (2000) according to an embodiment of the present disclosure. The process (2000) can be used for the reconstruction of coded blocks. In various embodiments, the process (2000) is executed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video decoder (1600B), and processing circuitry performing the functions of a video decoder (1800). In one example, the processing circuitry performs (i) one of video decoders (410), (510), and (810) and (ii) a combination of the functions of either video decoder (1600B) or video decoder (1800). In some embodiments, the process (2000) is implemented in software instructions, so that the processing circuitry executes the process (2000) when the software instructions are executed. The process begins at (S2001). In one example, the NIC framework is based on a neural network. In one example, the NIC framework is the NIC framework (900) described with reference to Figure 9B. The NIC framework can be based on a CNN, as described with reference to Figures 10-15. As mentioned above, the video decoder (e.g., (1600B) or (1800)) can include multiple components within the NIC framework. The neural network-based NIC framework can be pre-trained. The video decoder can be pre-trained using pre-trained parameters. The process (2000) proceeds to (S2010).

In (S2010), the update information of the first neural network in the encoded bitstream can be decoded. The update information of the first neural network can be used in the first neural network in the video decoder. The first neural network can be configured with a first set of pre-trained parameters. The update information of the first neural network can correspond to a first block in the image to be reconstructed and indicate a first replacement parameter corresponding to the first pre-trained parameter in the first set of pre-trained parameters.

In one example, the first pre-trained parameter is the pre-trained bias term.

In one example, the first pre-trained parameter is the pre-trained weight coefficient.

In one embodiment, the video decoder includes multiple neural networks. First neural network update information may indicate update information for one or more remaining neural networks among the multiple neural networks. For example, the first neural network update information may also indicate one or more replacement parameters for one or more remaining neural networks among the multiple neural networks. The one or more replacement parameters correspond to one or more corresponding pre-trained parameters of the one or more remaining neural networks. In one example, each of the first pre-trained parameter and each of the one or more pre-trained parameters is a corresponding pre-trained bias term. In one example, each of the first pre-trained parameter and each of the one or more pre-trained parameters is a corresponding pre-trained weight coefficient. In one example, the first pre-trained parameter and the one or more pre-trained parameters include one or more pre-trained bias terms and one or more pre-trained weight coefficients from the multiple neural networks.

In one example, the update information of the first neural network indicates the update information of a subset of multiple neural networks, and does not update the remaining subset of multiple neural networks.

In one example, the video decoder is the video decoder (1800) shown in Figure 18. The first neural network is the master decoder network (915).

In one example, the video decoder is the video decoder (1600B) shown in Figure 16B. Multiple neural networks in the video decoder include a master decoder network (915), a context model NN (916), an entropy parameter NN (917), and a super decoder (925). A first neural network is one of the master decoder network (915), the context model NN (916), the entropy parameter NN (917), and the super decoder (925), for example, the context model NN (916). In one example, the first neural network update information also includes one or more replacement parameters of one or more remaining neural networks in the video decoder (e.g., the master decoder network (915), the entropy parameter NN (917), and/or the super decoder (925)).

At (S2020), the first replacement parameter can be determined based on the first neural network update information. In one embodiment, the updated parameter is obtained from the first neural network update information. In one example, the updated parameter can be obtained from the first neural network update information by decompression (e.g., LZMA2 or bzip2 algorithm).

In one example, the first neural network update information indicates that the updated parameter is the difference between the first replacement parameter and the first pre-trained parameter. The first replacement parameter can be calculated based on the sum of the updated parameter and the first pre-trained parameter.

In one embodiment, the first replacement parameter is determined to be the updated parameter.

In one embodiment, the updated parameters are representative parameters generated based on the first replacement parameters on the encoder side (e.g., using a linear or nonlinear transformation), and the first replacement parameters are obtained based on the representative parameters.

In (S2030), the first neural network in the video decoder can be updated (or fine-tuned) based on the first substitution parameters, for example, by replacing the first pre-trained parameters with the first substitution parameters in the first neural network. If the video decoder contains multiple neural networks, and the first neural network update information indicates update information (e.g., additional substitution parameters) for the multiple neural networks, then the multiple neural networks can be updated. For example, the first neural network update information also includes one or more substitution parameters for one or more remaining neural networks in the video decoder, and the one or more remaining neural networks can be updated based on the one or more substitution parameters.

In (S2040), the first block encoded in the bitstream can be decoded, for example, by a video decoder updated based on the updated first neural network. The output block generated in (S2040) can be any suitable block with any appropriate size. In some examples, the output block is a reconstructed block in the reconstructed image in the spatial domain.

In some examples, the output block of the video decoder includes residual data in the spatial domain, and therefore can be used for further processing to generate a reconstructed block based on the output block. For example, the reconstruction module (874) is configured to combine the residual data and prediction results (output by the inter-frame or intra-frame prediction module) in the spatial domain to form a reconstructed block that can be part of the reconstructed image. Additional appropriate operations, such as deblocking, can be performed to improve visual quality. Components in various devices can be appropriately combined to, for example, refer to Figures 8 and 9, to implement (S2040) the residual data and corresponding prediction results from the main decoder network (915) in the video decoder being fed into the reconstruction module (874) to generate the reconstructed image.

In one example, the bitstream also includes one or more coded bits for determining the context model of the decoded coded block. The video decoder may include a master decoder network (e.g., (911)), a context model network (e.g., (916)), an entropy parameter network (e.g., (917)), and a super decoder network (e.g., (925)). A neural network is one of the master decoder network, the context model network, the entropy parameter NN, and the super decoder network. The super decoder network can be used to decode one or more coded bits. The context model network and the entropy parameter network can be used to determine the entropy model (e.g., the context model) based on the decoded bits and quantized potential bits of the coded block available from the context model network. The master decoder network and the entropy model can be used to decode the coded block.

The process (2000) proceeds to (S2099) and terminates.

The process (2000) can be adapted to various situations, and the steps in the process (2000) can be adjusted accordingly. One or more steps in the process (2000) can be modified, omitted, repeated, and/or combined. The process (2000) can be implemented using any suitable order. Additional steps can be added.

In one example, at (S2040), the first neural network, updated based on the first block, decodes another block in the encoded bitstream.

In one example, at (S2010), the second neural network update information in the encoded bitstream of the second neural network in the video decoder is obtained. The second neural network is configured with a second set of pre-trained parameters. The second neural network update information corresponds to a second block in the image to be reconstructed and indicates a second replacement parameter corresponding to the second pre-trained parameter in the second set of pre-trained parameters. The second neural network (e.g., a contextual model NN (916)) may be different from the first neural network (e.g., the main decoder network (915)). At (S2030), the second neural network in the video decoder may be updated based on the second replacement parameter. At (S2040), the second block may be decoded based on the updated second neural network of the second block. In one example, the first pre-trained parameter is one of the pre-trained weight coefficients and the pre-trained bias term. In one example, the second pre-trained parameter is the other of the pre-trained weight coefficients and the pre-trained bias term.

The embodiments in this disclosure can be used individually or in any combination in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium.

This disclosure does not impose any limitations on the methods used for encoders (e.g., neural network-based encoders), decoders (e.g., neural network-based decoders). The neural network used in the encoder, decoder, etc., can be any suitable type of neural network, such as DNN, CNN, etc.

Therefore, the adaptive online training method disclosed herein can be adapted to different types of NIC frameworks, such as different types of encoding DNN, decoding DNN, encoding CNN, decoding CNN, etc.

The above-described techniques can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 21 illustrates a computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter.

Computer software can be encoded using any suitable machine code or computer language, and can be assembled, compiled, linked or similarly to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through interpretation, microcode execution, etc.

The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

The components of the computer system (2100) shown in Figure 21 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software used to implement embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement on any component or combination of components shown in the exemplary embodiments of the computer system (2100).

The computer system (2100) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (2101), mouse (2102), trackpad (2103), touch screen (2110), data glove (not shown), joystick (2105), microphone (2106), scanner (2107), and camera (2108).

The computer system (2100) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (2110), data gloves (not shown), or joystick (2105), but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (2109), headphones (not shown)), visual output devices (e.g., screens (2110), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which are capable of outputting two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

The computer system (2100) may also include human-accessible storage devices and their associated media, such as optical media including CD/DVD ROM/RW (2120) having CD/DVD or similar media (2121), thumb drives (2122), removable hard disk drives or solid-state drives (2123), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated ROM/ASIC/PLD devices such as security dongles (not shown), etc.

Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

The computer system (2100) may also include an interface (2154) to one or more communication networks (2155). The network may be, for example, wireless, wired, or optical. The network may also be local area, wide area, metropolitan, vehicular, and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet and wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television; and vehicular and industrial networks including CANBus, etc. Some networks typically require external network interface adapters to connect to certain general-purpose data ports or peripheral buses (2149) (e.g., a USB port of the computer system (2100); others are typically integrated into the core of the computer system (2100) via connection to system buses as described below (e.g., an Ethernet interface in a PC computer system or a cellular network interface in a smartphone computer system). Using any of these networks, the computer system (2100) can communicate with other entities. This communication can be unidirectional and receive-only (e.g., broadcast television), unidirectional and transmit-only (e.g., to a CANbus device), or bidirectional, such as to other computer systems using local or wide area digital networks. As mentioned above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

The aforementioned human-machine interface device, human-accessible storage device, and network interface can be attached to the core (2140) of the computer system (2100).

The core (2140) may include one or more central processing units (CPU) (2141), graphics processing units (GPUs) (2142), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (2143), task-specific hardware accelerators (2144), graphics adapters (2150), etc. These devices, along with read-only memory (ROM) (2145), random access memory (2146), and internal mass storage such as internal non-user-accessible hard disk drives (SDs) (2147), may be connected via a system bus (2148). In some computer systems, the system bus (2148) may be accessed as one or more physical connectors to allow for the expansion of additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (2148) or via a peripheral bus (2149). In one example, a screen (2110) may be connected to a graphics adapter (2150). Peripheral bus architectures include PCI, USB, etc.

The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (2145) or RAM (2146). Transient data can also be stored in RAM (2146), while permanent data can be stored, for example, in internal mass storage (2147). Fast storage and retrieval of any storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage (2147), ROM (2145), RAM (2146), etc.

Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or may be of a type known and available to those skilled in the art of computer software.

By way of example and not limitation, a computer system having an architecture (2100), particularly a core (2140), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as some memory of the non-transitory core (2140), such as internal mass storage (2147) or ROM (2145). Software implementing various embodiments of this disclosure can be stored in such a device and executed by the core (2140). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (2140), and particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (2146) and modifying such data structures according to software-defined processes. In addition, or alternatively, the computer system may provide functionality as a result of hard-wired or otherwise incorporated logic (e.g., accelerator (2144)) that may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry containing logic for execution, or both, where appropriate. This disclosure includes any suitable combination of hardware and software.

Appendix A: Abbreviations

JEM: Joint Exploration Model

VVC: Versatile Video Coding

BMS: benchmark set

MV: Motion Vector

HEVC: High Efficiency Video Coding

SEI: Supplementary Enhancement Information

VUI: Video Usability Information

GOP: Groups of Pictures

TU: Transform Units

PU: Prediction Units

CTU: Coding Tree Units

CTB: Coding Tree Blocks

PB: Prediction Blocks

HRD: Hypothetical Reference Decoder

SNR: Signal Noise Ratio

CPU: Central Processing Unit

GPU: Graphics Processing Units

CRT: Cathode Ray Tube

LCD: Liquid-Crystal Display

OLED: Organic Light-Emitting Diode

CD: Compact Disc

DVD: Digital Video Disc

ROM: Read-Only Memory

RAM: Random Access Memory

ASIC: Application-Specific Integrated Circuit

PLD: Programmable Logic Device

LAN: Local Area Network

GSM: Global System for Mobile Communications

LTE: Long-Term Evolution

CANBus: Controller Area Network Bus

USB: Universal Serial Bus

PCI: Peripheral Component Interconnect

FPGA: Field Programmable Gate Areas

SSD: Solid-state drive

IC: Integrated Circuit

CU: Coding Unit

NIC: Neural Image Compression

R-D: Rate-Distortion

E2E: End to End

ANN: Artificial Neural Network

DNN: Deep Neural Network

CNN: Convolutional Neural Network

While several exemplary embodiments have been described in this disclosure, variations, substitutions, and various alternative equivalents fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, although not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims (17)

1. A video decoding method, characterized in that it includes: Decode the update information of the first neural network in the encoded bitstream. The update information of the first neural network is used for the first neural network. The first neural network is configured with a first set of pre-trained parameters. The update information of the first neural network corresponds to the first block in the image to be reconstructed and indicates a first replacement parameter corresponding to the first pre-trained parameter in the first set of pre-trained parameters. The update information of the first neural network also indicates one or more replacement parameters. The one or more replacement parameters are used for one or more remaining neural networks in a plurality of neural networks other than the first neural network. The update further includes updating the one or more remaining neural networks based on the first replacement parameters, wherein the update is based on the first replacement parameters; and The first block is decoded based on the updated first neural network, which is used for the first block. 2. The method according to claim 1, characterized in that it further comprises: The second neural network update information in the encoded bitstream is decoded. The second neural network update information is used in the second neural network, which is configured with a second set of pre-trained parameters. The second neural network update information corresponds to a second block in the image to be reconstructed and indicates a second replacement parameter corresponding to the second pre-trained parameter in the second set of pre-trained parameters. The second neural network is different from the first neural network. Update the second neural network based on the second replacement parameters; and The second block is decoded based on the updated second neural network, which is used for the second block. 3. The method according to claim 2, wherein, The first pre-training parameter is one of the pre-training weight coefficients and the pre-training bias term. 4. The method according to claim 3, characterized in that, The second pre-training parameter is another of the pre-training weight coefficients and the pre-training bias term. 5. The method according to claim 1, characterized in that it further comprises: Based on the updated first neural network, the second block of the encoded bitstream is decoded. 6. The method according to claim 1, characterized in that, The first neural network update information indicates the difference between the first replacement parameter and the first pre-trained parameter, and The method further includes: determining the first replacement parameter based on the difference and the sum of the first pre-trained parameters. 7. The method according to claim 2, characterized in that, Decoding the update information of the first neural network includes decoding the update information of the first neural network using one of the variants of the Lempel-Ziv-Markov chain algorithm, LZMA2 and bzip2. 8. The method according to claim 7, characterized in that, Decoding the update information of the second neural network includes: decoding the update information of the second neural network based on another of the LZMA2 and bzip2 algorithms. 9. A video decoding device, characterized in that it includes: Processing circuit, the processing circuit being configured to: Decode the update information of the first neural network in the encoded bitstream. The update information of the first neural network is used for the first neural network. The first neural network is configured with a first set of pre-trained parameters. The update information of the first neural network corresponds to the first block in the image to be reconstructed and indicates a first replacement parameter corresponding to the first pre-trained parameter in the first set of pre-trained parameters. The update information of the first neural network also indicates one or more replacement parameters. The one or more replacement parameters are used for one or more remaining neural networks in a plurality of neural networks other than the first neural network. The update further includes updating the one or more remaining neural networks based on the first replacement parameters, wherein the update is based on the first replacement parameters; and The first block is decoded based on the updated first neural network, which is used for the first block. 10. The device according to claim 9, wherein the processing circuit is configured to: The second neural network update information in the encoded bitstream is decoded. The second neural network update information is used in the second neural network, which is configured with a second set of pre-trained parameters. The second neural network update information corresponds to a second block in the image to be reconstructed and indicates a second replacement parameter corresponding to the second pre-trained parameter in the second set of pre-trained parameters. The second neural network is different from the first neural network. Update the second neural network based on the second replacement parameters; and The second block is decoded based on the updated second neural network, which is used for the second block. 11. The device according to claim 10, characterized in that, The first pre-training parameter is one of the pre-training weight coefficients and the pre-training bias term. 12. The device according to claim 11, characterized in that, The second pre-training parameter is another of the pre-training weight coefficients and the pre-training bias term. 13. The device according to claim 9, wherein the processing circuit is configured to: decode a second block of the encoded bitstream based on the updated first neural network. 14. The device according to claim 9, characterized in that, The first neural network update information indicates the difference between the first replacement parameter and the first pre-trained parameter, and The processing circuit is configured to determine the first replacement parameter based on the difference and the sum of the first pre-trained parameters. 15. The device according to claim 10, wherein the processing circuit is configured to: Decoding the update information of the first neural network includes decoding the update information of the first neural network using one of the variants of the Lempel–Ziv–Markov chain algorithm, LZMA2 and bzip2. 16. The device according to claim 15, wherein the processing circuit is configured to: The update information of the second neural network is decoded based on another of the LZMA2 and bzip2 algorithms. 17. A non-transitory computer-readable storage medium, characterized in that the medium stores a program executable by at least one processor to perform the method as described in any one of claims 1-8.