HK40046974B - Method and apparatus for decoding encoded video sequence, and storage medium - Google Patents

Description

Methods, apparatus and storage media for decoding encoded video sequences

Cross-references to related applications

This application claims priority to U.S. Provisional Patent Application No. 62/704,040, filed January 2, 2019, with the United States Patent and Trademark Office, and to U.S. Patent Application No. 16/710,389, filed December 11, 2019, with the United States Patent and Trademark Office, both of which are incorporated herein by reference in their entirety.

Technical Field

The disclosed topics relate to video encoding and decoding, and more specifically to syntax elements for adaptive image resolution rescaling in high-level syntax (synatx) structures, as well as related decoding and scaling processing of image segments.

Background Technology

Video encoding and decoding using inter-frame prediction with motion compensation has been known for decades. Uncompressed digital video can consist of a series of pictures, each with spatial dimensions such as 1920×1080 luma samples and associated chroma samples. This series of pictures can have a fixed or variable picture rate (also informally called the frame rate), such as 60 pictures per second or 60Hz. Uncompressed video has significant bitrate requirements. For example, at 8 bits per sample, 1080p60 4:2:0 video (with 1920×1080 luma sample resolution at a 60Hz frame rate) requires close to 1.5 Gbit/s of bandwidth. One hour of such video would require more than 600 GB of storage space.

One objective of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations thereof can be employed. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. In the case of video, lossy compression is widely used. The amount of distortion tolerated depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can be reflected in the fact that higher permissible/acceptable distortion results in a higher compression ratio.

Video encoders and decoders can utilize techniques from several broad categories, including motion compensation, transform, quantization, and entropy coding, some of which will be discussed below.

Basic information theory states that a lower spatial resolution representation of a series of pictures of a given set of content can be compressed into a larger representation using fewer bits. Therefore, for decades, the practice of downsampling the input series of pictures before encoding and upsampling them accordingly after decoding to obtain a displayable picture has been used when bandwidth or storage is insufficient, or for cost-sensitive applications that do not require high spatial resolution. For example, at least some MPEG-2-based TV distribution/display systems may change the horizontal resolution of the pictures on a per-picture-group basis outside the encoding loop when the available bandwidth on the channel is insufficient to allow for good reproduction quality. In this regard, it should be noted that many video codecs have a “breakpoint,” also known as a “knot point” (in the rate-distortion curve), where the gradual quality degradation achieved by increasing the quantizer value (to stay within the rate envelope) is broken, and a sudden, significant quality degradation occurs. Because some video distribution systems operate very close to the breakpoint for average-complexity content, this sudden increase in activity can lead to annoying artifacts that may not be easily compensated for by post-processing techniques.

While changing the resolution outside the coding loop might seem relatively simple from a video codec implementation and specification perspective, it's not particularly efficient. This is because resolution changes can require intra-coded images, which in many cases can be many times larger than the inter-coded images most commonly found in the encoded video bitstream. Adding additional types of intra-coded images to combat what may inherently be a bandwidth shortage problem can have the opposite effect, requiring large buffers and associated large potential delays to be effective.

For applications where latency is critical, mechanisms have been designed to allow changes in the resolution of a video sequence within the coding loop without using intra-frame coded pictures. Because these techniques require resampling of the reference picture, they are commonly referred to as Reference Picture Resampling (RPR) techniques. RPR has been introduced into standardized video coding and has seen relatively widespread deployment in some video conferencing systems, as outlined in Annex P of ITU-T Recommendation H.263, issued in 1998. This technique suffers from at least the following drawbacks: 1. The syntax used to represent reference picture resampling with signals is not error-tolerant; 2. The upsampling and downsampling filters used—bilinear filters—while computationally economical, are not very beneficial for good video quality; 3. The specific techniques that allow for “distortion” may be overly rich for unnecessary and unfounded features; and 4. This technique can only be applied to the entire picture, not to a segment of the picture.

The more recent video coding technology known as AV1 also has limited support for RPR. It encounters problems similar to those mentioned in #1 and #4 above, and furthermore, the filters used are quite complex for some applications.

Summary of the Invention

According to an embodiment, a method for decoding encoded images of an encoded video sequence, executed by at least one processor, includes: decoding syntax elements related to a reference segment resolution from a first high-level syntax structure for a plurality of images; decoding syntax elements related to a decoded segment resolution from a second high-level syntax structure that varies from a first encoded image to a second encoded image; resampling samples from a reference image buffer for use by a decoder for prediction, the decoder decoding segments at a decoding resolution, and the samples from the reference image buffer being at the reference segment resolution; decoding segments at the decoded segment resolution into decoded segments at the decoded segment resolution; and storing the decoded segments in the reference image buffer.

According to an embodiment, an apparatus for decoding encoded images of an encoded video sequence includes: at least one memory configured to store computer program code; and at least one processor configured to access the at least one memory and operate according to the computer program code, the computer program code including: first decoding code configured to decode syntax elements related to a reference segment resolution from a first high-level syntax structure for a plurality of images; second decoding code configured to decode syntax elements related to a decoded segment resolution from a second high-level syntax structure varying from a first encoded image to a second encoded image; resampling code configured to resample samples from a reference image buffer for use by a decoder for prediction, the decoder decoding segments at a decoding resolution, and the samples from the reference image buffer being at the reference segment resolution; third decoding code configured to decode segments at the decoded segment resolution into decoded segments at the decoded segment resolution; and storage code configured to store the decoded segments in the reference image buffer.

According to an embodiment, a non-transitory computer-readable storage medium stores a program for decoding encoded images of an encoded video sequence. The program includes instructions that cause a processor to: decode syntax elements related to a reference segment resolution from a first high-level syntax structure for a plurality of images; decode syntax elements related to a decoded segment resolution from a second high-level syntax structure that varies from a first encoded image to a second encoded image; resample samples from a reference image buffer for use by a decoder for prediction, the decoder decoding segments at a decoding resolution, and the samples from the reference image buffer being at the reference segment resolution; decode segments at the decoded segment resolution into decoded segments at the decoded segment resolution; and store the decoded segments in the reference image buffer.

Attached Figure Description

Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

Figure 1 is a simplified block diagram of a communication system according to an embodiment.

Figure 2 is a simplified block diagram of a communication system according to an embodiment.

Figure 3 is a simplified block diagram of the decoder according to an embodiment.

Figure 4 is a simplified block diagram of the encoder according to an embodiment.

Figure 5 is a syntax diagram according to an embodiment.

Figure 6 is a simplified block diagram of a decoder with reference image resampling capability according to an embodiment.

Figure 7 is a schematic diagram of a tile layout according to an embodiment, which resamples each tile using a reference image.

Figure 8A is a flowchart illustrating a method for decoding encoded images of an encoded video sequence according to an embodiment.

Figure 8B is a simplified block diagram of an apparatus for controlling the decoding of a video sequence according to an embodiment.

Figure 9 is a schematic diagram of a computer system according to an embodiment.

Detailed Implementation

When high-activity content is present, in-loop RPR (Rapid Reduction) techniques are needed to address potential quality issues when video encoding is run near breakpoints in average content. Compared to known techniques, this technique requires efficient filters in terms of performance and computational complexity, requires fault tolerance, and needs to be applicable only to a portion of the image, i.e., (at least a rectangular) image segment.

Figure 1 shows a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. The system (100) may include at least two terminals (110-120) interconnected via a network (150). For unidirectional data transmission, a first terminal (110) may encode video data at its local location for transmission to another terminal (120) via the network (150). A second terminal (120) may receive the encoded video data from the other terminal from the network (150), decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in applications such as media services.

Figure 1 illustrates a second pair of terminals (130, 140) configured to support bidirectional transmission of encoded video, for example, during video conferencing. For bidirectional data transmission, each terminal (130, 140) can encode video data captured at its local location for transmission to the other terminal via the network (150). Each terminal (130, 140) can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

In Figure 1, the terminals (110-140) can be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited to these. Embodiments of this disclosure can be applied to laptop computers, tablet computers, media players, and/or dedicated video conferencing equipment. The network (150) refers to any number of networks that transmit encoded video data between the terminals (110-140), including, for example, wired and/or wireless communication networks. The communication network (150) can exchange data in circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and/or the Internet. For the purposes of this discussion, unless explained below, the structure and topology of the network (150) may be irrelevant to the operation of this disclosure.

As an example of an application of the disclosed subject matter, Figure 2 illustrates the arrangement of a video encoder and decoder in a streaming environment. The disclosed subject matter can be equally applied to other video-enabled applications, including, for example, video conferencing, digital television, and storing compressed video on digital media including CDs, DVDs, Memory Sticks, etc.

The streaming system may include an acquisition subsystem (213), which may include a video source (201) such as a digital camera, thereby creating, for example, an uncompressed video sample stream (202). The sample stream (202), depicted with thicker lines to emphasize its high data volume compared to the encoded video bitstream, may be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter described in more detail below. The encoded video bitstream (204), depicted with thinner lines to emphasize its lower data volume compared to the sample stream, may be stored on a streaming server (205) for future use. One or more streaming clients (206, 208) may access the streaming server (205) to retrieve a copy (207, 209) of the encoded video bitstream (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video bitstream (207) and creates an output video sample stream (211) that can be presented on a display (212) or other presentation device (not shown). In some streaming systems, the video bitstream (204, 207, 209) may be encoded according to certain video coding/compression standards. Examples of these standards include ITU-T Recommendation H.265. A video coding standard informally known as Versatile Video Coding (VVC) is under development. The disclosed topics can be used in a VVC environment.

Figure 3 may be a functional block diagram of a video decoder (210) according to an embodiment of the present disclosure.

The receiver (310) can receive one or more encoded video sequences to be decoded by the decoder (210); in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. The encoded video sequences can be received from a channel (312), which can be a hardware/software link to a storage device storing the encoded video data. The receiver (310) can receive the encoded video data as well as other data, such as encoded audio data and/or auxiliary data streams, which can be forwarded to their respective user entities (not depicted). The receiver (310) can separate the encoded video sequences from other data. To prevent network jitter, a buffer (315) can be coupled between the receiver (310) and the entropy decoder/parser (320) (hereinafter referred to as the "parser"). When the receiver (310) receives data from a storage/forwarding device with sufficient bandwidth and controllability or from a synchronization network, the buffer (315) may not be required, or the buffer (315) may be small. For use on best-effort packet networks such as the Internet, a buffer (315) may be required, which can be relatively large and can advantageously have an adaptive size.

The video decoder (210) may include a parser (320) to reconstruct symbols (321) from the entropy-encoded video sequence. These symbols may include information for managing the operation of the decoder (210) and potential information for controlling a presentation device, such as a display (212), which is not integral to the decoder but may be coupled to it, as shown in Figure 2. The control information for the presentation device may be in the form of Supplementary Enhancement Information (SEI) messages or fragments of Video Usability Information (VUI) parameter sets (not depicted). The parser (320) may perform parsing/entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well-known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (320) can extract a set of subgroup parameters from the encoded video sequence for use in the video decoder, based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder/parser can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the encoded video sequence.

The parser (320) can perform entropy decoding/parsing operations on the video sequence received from the buffer (315) to create symbols (321).

The reconstruction of symbol (321) can involve multiple different units, depending on the type of encoded video picture or a subset of encoded video pictures (e.g., inter-frame pictures and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (320). For clarity, the flow of this subgroup control information between the parser (320) and the multiple units described below is not depicted.

In addition to the functional blocks already mentioned, decoder 210 can be conceptually subdivided into multiple functional units as described below. In practical implementations operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the following functional units.

The first unit is the scaler/inverse transform unit (351). The scaler/inverse transform unit (351) receives quantization transform coefficients as symbols (321) and control information from the parser (320), including which transform mode to use, block size, quantization factor, quantization scaling matrix, etc. The scaler/inverse transform unit (351) can output a block containing sample values, which can be input into the aggregator (355).

In some cases, the output samples of the scaler/inverse transform (351) may belong to intra-coded blocks, i.e., blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. This predictive information can be provided by the intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) uses surrounding reconstructed information extracted from the current (partially reconstructed) image (356) to generate blocks with the same size and shape as the blocks being reconstructed. In some cases, the aggregator (355) adds the predictive information already generated by the intra-picture prediction unit (352) to the output sample information provided by the scaler/inverse transform unit (351) based on each sample.

In other cases, the output samples of the scaler/inverse transform unit (351) may belong to blocks of inter-frame coding and potential motion compensation. In this case, the motion compensation prediction unit (353) can access the reference image memory (357) to extract samples for prediction. After motion compensation of the extracted samples according to the symbols (321) belonging to the block, these samples can be added by the aggregator (355) to the output of the scaler/inverse transform unit (referred to as residual samples or residual signals in this case) to generate output sample information. The extraction of prediction samples from the address in the reference image memory by the motion compensation unit can be controlled by a motion vector, which can be available to the motion compensation unit in the form of symbols (321), which can have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values extracted from the reference image memory when using subsampled precise motion vectors, motion vector prediction mechanisms, etc.

The output samples of the aggregator (355) can be subjected to various loop filtering techniques in the loop filter unit (354). The video compression technique may include in-loop filtering techniques, which are controlled by parameters included in the encoded video bitstream and available to the loop filter unit (354) as symbols (321) from the parser (320). However, the video compression technique may also respond to metadata obtained during decoding of previous (in the order of decoding) portions of the encoded picture or encoded video sequence, as well as to sample values from previous reconstruction and loop filtering.

The output of the loop filter unit (354) can be a sample stream, which can be output to the presentation device (212) and stored in the reference image memory (356) for use in future inter-frame image prediction.

Once a certain encoded image is fully reconstructed, it can be used as a reference image for future predictions. Once an encoded image is fully reconstructed and the encoded image (e.g., by the parser (320)) is identified as a reference image, the current reference image (356) can become part of the reference image buffer (357), and new current image memory can be reallocated before reconstruction of subsequent encoded images begins.

The video decoder 320 can perform decoding operations according to a predetermined video compression technique that may be documented in standards such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in use; that is, the encoded video sequence in some sense adheres to the syntax of the video compression technique or standard, as specified in the video compression technique documentation or standard, particularly in its configuration documentation. For compliance, it may also be necessary that the complexity of the encoded video sequence be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented as a signal in the encoded video sequence.

In this embodiment, the receiver (310) can receive encoded video as well as additional (redundant) data. The additional data can be included as part of the encoded video sequence. The video decoder (320) can use the additional data to properly decode the data and/or more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

Figure 4 can be a functional block diagram of a video encoder (203) according to an embodiment of the present disclosure.

The encoder (203) can receive video samples from a video source (201) (not part of the encoder), which can capture video images to be encoded by the encoder (203).

A video source (201) may be provided as a digital video sample stream containing a sequence of source videos to be encoded by an encoder (203). This digital video sample stream may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (201) may be a storage device storing previously prepared video. In a video conferencing system, the video source (203) may be a camera that acquires local image information as a video sequence. The video data may be provided as multiple individual pictures that are given motion when viewed in sequence. The pictures themselves may be organized as spatial pixel arrays, where each pixel may include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following focuses on describing samples.

According to an embodiment, the encoder (203) can encode and compress images of a source video sequence into an encoded video sequence (443) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (450). The controller controls and is functionally coupled to other functional units described below. Coupling is not depicted for clarity. Parameters set by the controller may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of images (GOP) layout, maximum motion vector search range, etc. Other functions of the controller (450) can be readily identified by those skilled in the art and may belong to the video encoder (203) optimized for a particular system design.

Some video encoders operate in a manner readily recognized by those skilled in the art as an “encoding loop.” As an oversimplification, an encoding loop may include the encoding portion of an encoder (430) (hereinafter referred to as the “source encoder”) responsible for creating symbols based on the input image to be encoded and a reference image, and a (local) decoder (433) embedded in the encoder (203), which reconstructs the symbols to create sample data, which may also be created by a (remote) decoder (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). This reconstructed sample stream is input to a reference image memory (434). Since the decoding of the symbol stream results in a bit-accurate result independent of the (local or remote) decoder location, the contents of the reference image buffer also correspond bit-accurately between the local and remote encoders. In other words, the reference image samples “seen” by the encoder’s prediction portion are exactly the same sample values that the decoder can “see” when using prediction during decoding. The basic principles of this reference image synchronization (and the drift that occurs if synchronization cannot be maintained, for example, due to channel errors) are well known to those skilled in the art.

The operation of the “local” decoder (433) can be the same as that of the “remote” decoder (210) which has been described in detail above in conjunction with Figure 3. However, referring simply to Figure 3, since symbols are available and encoding/decoding symbols into an encoded video sequence via an entropy encoder (445) and a parser (320) may be lossless, the entropy decoding part of the decoder (210) (including the channel (312), receiver (310), buffer (315), and parser (320)) may not be fully implemented in the local decoder (433).

It can be observed that any decoder technique other than parsing/entropy decoding present in the decoder also needs to exist in the corresponding encoder in essentially the same functional form. Therefore, the subject matter presented here focuses on decoder operation. The description of encoder techniques can be simplified, as encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

As part of the operation of the source encoder, the source encoder (430) may perform motion-compensated predictive coding, which predictively codes the input frame with reference to one or more previously encoded frames from the video sequence designated as "reference frames". In this way, the encoding engine (432) encodes the differences between the pixel blocks of the input frame and the pixel blocks of the reference frame, which can be selected as the prediction reference for the input frame.

The local video decoder (433) can decode encoded video data of a frame that can be designated as a reference frame based on symbols created by the source encoder (430). The operation of the encoding engine (432) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder (not shown in FIG. 4), the reconstructed video sequence can typically be a copy of the source video sequence, but with some errors. The local video decoder (433) replicates the decoding process that can be performed by the video decoder on the reference frame and can make the reconstructed reference frame stored in a reference picture cache (434). In this way, the encoder (203) can locally store a copy of the reconstructed reference frame that shares common content with the reconstructed reference frame that will be obtained by the remote video decoder (without transmission errors).

The predictor (435) can perform a prediction search on the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) can search the reference image memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can be used as a suitable prediction reference for the new image. The predictor (435) can operate pixel-by-pixel based on the sample blocks to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction references obtained from multiple reference images stored in the reference image memory (434).

The controller (450) can manage the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters for encoding video data.

The outputs of all the above functional units can be entropy encoded in the entropy encoder (445). The entropy encoder converts the symbols generated by the various functional units into an encoded video sequence by lossless compression of the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, arithmetic coding, etc.).

The transmitter (440) can buffer the encoded video sequence created by the entropy encoder (445) in preparation for transmission of the encoded video sequence via a communication channel (460), which may be a hardware/software link to a storage device that will store the encoded video data. The transmitter (440) can combine the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (sources not shown).

The controller (450) can manage the operation of the encoder (203). During encoding, the controller (450) can assign a specific encoded image type to each encoded image, which may affect the encoding techniques that can be applied to the corresponding image. For example, images can typically be assigned one of the following frame types:

An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, pictures refreshed by a separate decoder. Those skilled in the art are familiar with those variations of I-pictures, as well as their respective applications and characteristics.

A predicted image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict the sample value for each block.

A bidirectional prediction image (B-image) can be an image encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and a reference index to predict the sample values for each block. Similarly, multiple prediction images can use more than two reference images and associated metadata to reconstruct a single block.

The source image can typically be spatially subdivided into multiple sample blocks (e.g., each block has 4×4, 8×8, 4×8, or 16×16 samples) and encoded block by block. A block can be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the corresponding image of the block. For example, a block of an I-image can be unpredictably coded, or the block can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. A pixel block of a P-image can be unpredictably coded with reference to a previously coded reference image via spatial prediction or temporal prediction. A block of a B-image can be unpredictably coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.

The video encoder (203) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (203) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

In this embodiment, the transmitter (440) can transmit encoded video as well as additional data. The video encoder (430) can include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data (such as redundant images and slices), supplementary enhancement information (SEI) messages, fragments of visual usability information (VUI) parameter sets, etc.

Referring to Figure 5, in an embodiment, a flag (e.g., adaptive_picture_resolution) (502) may indicate whether the spatial resolution of a picture segment (e.g., tile, tile group, CTU, CTU group) can be adaptively resampled/rescaled/unscaled (these three terms are used interchangeably throughout the text) for decoding, for predictive reference, and for display output (collectively referred to as RPR information). If the flag indicates the presence of RPR information, certain syntax elements may indicate the picture sizes of multiple reference pictures and the output picture, respectively. These syntax elements and the aforementioned flags may be in any suitable high-level syntax structure to include, for example, decoder/video/sequence/picture/slice/tile parameter sets, sequence/GOP/picture/slice/GOB/tile group/tile header, and/or SEI messages. Not all of these syntax elements need to be always present. For example, while the RPR resolution may be dynamic, the aspect ratio of the picture may be fixed in the video coding technique or standard, or this fixed aspect ratio may be indicated by a signal in a tag within an appropriate high-level syntax structure. Similarly, video coding techniques or standards can specify reference image resampling and omit output image resampling, in which case output image size information can also be omitted. In another example, the presence of output image size information can be adjusted based on its own flag (not depicted).

In a non-restrictive example, some RPR information may reside in the sequence parameter set (501). The syntax elements reference_pic_width_in_luma_samples (503) and reference_pic_height_in_luma_samples (504) may indicate the width and height of the reference picture, respectively. The syntax elements output_pic_width_in_luma_samples (505) and output_pic_height_in_luma_samples (506) may specify the output picture resolution. All of the above values may be in units of luminance samples or other units, as these are common in video compression techniques or standards. Certain restrictions on the values of syntax elements may also be imposed by video coding techniques or standards; for example, the values of one or more syntax elements may need to be powers of two (so that the picture can be easily fitted into blocks commonly used in video coding), or the relationship between horizontal dimensions may be restricted to a certain value (so that a finite set of filters optimized for certain resolution ratios, as described below).

The encoding of the above information can be in any suitable form. As shown in the figure, a simple option is to use a variable length, which is an unsigned integer value represented by ue(v). Other options are always possible, including those used to indicate the image size in traditional video coding techniques or standards (such as H.264 or H.265).

One objective of the disclosed subject matter is to allow RPR within the coding loop; that is, between different pictures in a video sequence (CVS). Therefore, the syntax elements specifying the actual decoded size of a picture may need to be in a syntax structure that allows the syntax elements to potentially change within the CVS as they move from one picture to another. In an embodiment, the syntax elements decoded_pic_width_in_luma_samples (508) and decoded_pic_height_in_luma_samples (509) reside in a suitable high-level syntax structure, here in PPS (507), and the values of the fields can change within the video sequence (CVS). Other suitable high-level syntax structures may include PPS, slice parameter sets, tile parameter sets, picture/slice/GOB/tile group/tile header, and/or SEI messages. Using SEI messages may be less desirable because RPR techniques can have a prescriptive impact on the decoding process. The above discussion applies to the encoding of these syntax elements.

In embodiments, `reference_pic_width_in_luma_samples` and `reference_pic_height_in_luma_samples` can indicate the image resolution of a reference image or a segment of a reference image in the decoded image buffer. This can mean that the reference image is always kept at full resolution, regardless of the applied resampling, and is a key difference between the techniques discussed herein and those described in Appendix P of H.263.

The above description assumes that RPR technology is applied to the entire image. Certain environments may benefit from RPR technology that can be applied to image segments, such as tile groups, tiles, slices, GOBs, etc. For example, an image can be spatially divided into semantically distinct spatial regions, often referred to as tiles. One example is security video, and another is a 360-degree video with various views, such as a cube projection (where six views corresponding to the six faces of the cube constitute a representation of the 360-degree scene). In such and similar scenarios, the semantically distinct content of each tile may require the application of multiple RPR technologies differently on a per-tile basis, as the content activity of each tile may differ. Therefore, in an embodiment, RPR technology may be applied to each tile. This requires signaling based on each tile (not depicted). These signaling technologies can be similar to those described above for signaling per image, except that they may need to include signaling for potentially multiple tiles.

In an embodiment, each tile or tile group may have different reference_tile_width_in_luma_samples and reference_tile_height_in_luma_samples values in the tile group header or header parameter set or other suitable high-level syntax structure.

In an embodiment, if the resolution of the reference image is different from the resolution of the decoded image, the decoded image can be rescaled relative to the ratio between the resolution of the reference image and the resolution of the decoded image, and then the rescaled decoded image can be stored in the decoded image buffer (DPB) as a reference image.

In an embodiment, if the vertical/horizontal resolution ratio between the decoded image resolution and the reference image resolution is explicitly represented by a signal as described above, the decoded image can be rescaled relative to the ratio represented by the signal, and the rescaled decoded image can then be stored in the decoded image buffer (DPB) as a reference image.

In this embodiment, output_pic_width_in_luma_samples and output_pic_height_in_luma_samples can indicate the image resolution of the output image or the output image segment to the video player.

In an embodiment, if the output image resolution differs from the reference image resolution, the reference image can be rescaled relative to the ratio between the output image resolution and the reference image resolution. The rescaled reference image can then be highlighted from the DPB as the output image and fed into the video player for display.

In an embodiment, if the vertical/horizontal resolution ratio between the reference image resolution and the output image resolution is explicitly represented by a signal, the reference image can be rescaled relative to the ratio between the output image resolution and the reference image resolution. The rescaled reference image can then be highlighted from the DPB as the output image and fed into the video player for display.

In an embodiment, each tile or tile group may have different output_tile_width_in_luma_samples and output_tile_height_in_luma_samples values in the tile group header or header parameter set or other suitable syntax structure.

Some video coding techniques or standards include temporal scalability in the form of temporal sublayers. In embodiments, each sublayer may have different values for reference_pic_width_in_luma_samples, reference_pic_height_in_luma_samples, output_pic_width_in_luma_samples, output_pic_height_in_luma_samples, decoded_pic_width_in_luma_samples, decoded_pic_height_in_luma_samples. Syntax elements for each sublayer may be represented by signals, for example, using SPS or any other suitable high-level syntax structure.

Referring to Figure 6, in an embodiment, the video bitstream parser (602) can parse and interpret the aforementioned syntax elements and other syntax elements from the encoded video bitstream, which is received from an encoded picture buffer (601). When receiving non-RPS-related syntax elements from the encoded video bitstream, the video decoder can reconstruct the encoded picture at a potentially undersampled resolution. To do this, reference samples may be needed, which can be received from the decoded picture buffer (604). According to an embodiment, since the decoded picture buffer (604) stores reference pictures or segments at full resolution, rescaling (605) may be necessary to provide the decoder (603) with appropriately resampled reference pictures. Rescaling (603) can be controlled by a rescaling controller (606), which can receive scaling parameters (e.g., the aforementioned syntax elements) (607) and convert the scaling parameters into appropriate information (608) for the rescaler (605); for example, calculating appropriate rescaling filter parameters. Finally, if the output resolution needs to be rescaled, the rescaling controller (606) can also provide rescaling information 609 to the rescaling mechanism (610) for display. Finally, the reconstructed video can be played by a video player (611) or otherwise processed for consumption or storage.

The filters used in the rescaling process can be specified in the video coding technique or standard. Since both filtering directions need to be "inside" the coding loop—that is, downsampling (e.g., from the decoded image buffer (604) to the video decoder (603)) and upsampling (e.g., from the video decoder (603) to the decoded image buffer (604))—these two filtering directions may be fully specified as required and should be specified by the video compression technique or standard to achieve reversibility as much as possible. As for the filter design itself, a balance may need to be struck between computational/implementation simplicity and performance. Some initial results suggest that bilinear filters, such as those suggested in Appendix P of H.263, may be suboptimal in terms of performance. On the other hand, certain adaptive filtering techniques employing neural network-based processing may be computationally too complex to be widely adopted by video coding techniques or standards within a commercially appropriate timeframe and under commercially appropriate complexity constraints. As a balance, filter designs such as those used in SHVC or various interpolation filters, such as those used in HEVC, may be appropriate and will have the added advantage of a better understanding of their characteristics.

Referring to Figure 7, in an embodiment, each image segment, such as a slice, GOB, tile, or tile group (hereinafter, tile), can be independently rescaled from a decoded tile to a reference tile, and from a reference tile to an output tile (or image) at different resolutions.

Considering that the input image (701) enters a square encoder and is divided into four square source tiles (702) (source tile 2 of the four source tiles is shown), each source tile covers 1/4 of the input image. Of course, other image geometries and tile layouts are equally possible, depending on the subject matter disclosed. Each tile is made twice the width and twice the height of W and H, respectively, hereinafter referred to as "2W" for twice the width and "2H" for twice the height (similarly for other numbers; for example, 1W means 1 times the width and 3H means 3 times the height – this convention is used throughout the figures and their description). The source tiles can be, for example, camera views of different scenes in a secure camera environment. Thus, each tile can cover content with potentially fundamentally different levels of activity, which may require selecting different RPRs for each tile.

Assuming the encoder (not depicted) creates the encoded image, the reconstruction produces four tiles with the following rescaled resolution:

Decoded blocks 0(702): 1H and 1W

Decoded Plot 1 (703): 1H and 2W

Decoded Plot 2 (704): 2H and 2W

Decoded Plot 3 (705): 2H and 1W

This produces a scaled-down representation of the decoded tile size.

Note that in some video coding techniques or standards, there may be samples in a decoded image that are not assigned to any tile. How these samples are encoded, if any, varies depending on the video coding technique. In some embodiments, samples not assigned to any depicted tile may be assigned to other tiles, and all samples of a tile may be encoded by creating a low number of coding bits, for example, in a skip mode. In other embodiments, the video coding technique or standard may not have the requirement to encode all samples of the image in some form in every video frame (which is currently common to some extent), thus avoiding any wasted bits on those samples. In yet another embodiment, certain padding techniques can be used to efficiently fill in unused samples, making their coding overhead negligible.

In this example, the reference image buffer maintains the reference image sample at full resolution, which in this case is the same as the source resolution. Therefore, the four rescaled tiles (706 to 709) used for reference can be maintained at 2H and 2W resolutions, respectively. To match the varying resolutions of the decoded tiles (702 to 705), the rescaling (710) in both directions—from the decoder to the reference image buffer and from the reference image buffer to the decoder—may be different for each tile.

If output rescaling (711) is also in use, the output of the decoded image buffer can be rescaled to the output image for display (or otherwise processed) at each tile or each image granularity (712). The resolution of the output image (712) used for display can be greater than or less than the resolution of the image in the decoded image buffer.

Figure 8A is a flowchart illustrating a method (800) for decoding encoded images of an encoded video sequence according to an embodiment. In some embodiments, one or more processing blocks of Figure 8A may be performed by a decoder (210). In some embodiments, one or more processing blocks of Figure 8A may be performed by another device or a group of devices (e.g., an encoder (203)) separate from or including the decoder (210).

Referring to Figure 8A, the method (800) includes determining whether RPR information exists (805), and if it is determined that RPR information does not exist, the method ends (855). If it is determined that RPR information exists, the method includes decoding the syntax elements related to the reference segment resolution from a first high-level syntax structure for multiple images (810).

The method (800) includes a second high-level syntax structure that varies from the first encoded image to the second encoded image, and decodes syntax elements related to the resolution of the decoded segment (820).

Method (800) includes resampling samples from a reference image buffer for use by the decoder for prediction, the decoder decoding the segment at a decoding resolution, and the samples from the reference image buffer at a reference segment resolution (830).

The method (800) includes decoding a segment at the resolution of a decoded segment into a decoded segment at the resolution of a decoded segment (840).

In addition, the method (800) includes storing the decoded segment in a reference image buffer (850).

Method (800) may further include resampling the decoded segment to the reference segment resolution.

The method (800) may further include a resampling filter for at least one of the following: resampling samples from a reference image buffer for use by the decoder for prediction; and resampling the decoded segment to the resolution of the reference segment, wherein the resampling filter is computationally more complex and not adaptive than a bilinear filter.

The method (800) may further include the resampling filter being selected from a plurality of resampling filters based on the relationship between the decoding resolution and the reference segment resolution.

Method (800) may further include, in which the segment is an image.

The method (800) may further include, wherein each of the first encoded image and the second encoded image contains a plurality of segments.

The method (800) may further include decoding the syntax elements related to the output resolution according to the third high-level syntax structure; and resampling samples of the decoded segments to the output resolution.

Method (800) may further include resampling using different resampling factors in width and height.

Although Figure 8A shows an example block of method (800), in some implementations, method (800) may include more blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted in Figure 8A. Alternatively, two or more blocks of method (800) may be executed in parallel.

Furthermore, the proposed methods can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In the example, one or more processors execute a program stored in a non-transitory computer-readable medium to perform one or more of the proposed methods.

Figure 8B is a simplified block diagram of an apparatus (860) for decoding encoded images of a video sequence according to an embodiment.

Referring to FIG8B, the device (860) includes a first decoding code (870), a second decoding code (875), a resampling code (880), a third decoding code (885), and a storage code (890).

The first decoding code (870) is configured to decode syntax elements related to the resolution of the reference segment from a first high-level syntax structure for multiple images.

The second decoding code (875) is configured as a second high-level syntax structure that varies from the first encoded image to the second encoded image, decoding syntax elements related to the resolution of the decoded segment.

The resampling code (880) is configured to resample the samples from the reference image buffer for use by the decoder for prediction, the decoder decodes the segment at the decoding resolution, and the samples from the reference image buffer are at the reference segment resolution.

The third decoding code (885) is configured to decode a segment at the resolution of a decoded segment into a decoded segment at the resolution of a decoded segment.

The storage code (890) is configured to store the decoded segment in the reference image buffer.

The above technology can be implemented as computer software, which uses computer-readable instructions and is physically stored in one or more computer-readable media.

The aforementioned techniques for adaptive image resolution rescaling can be implemented as computer software, which uses computer-readable instructions and is physically stored in one or more computer-readable media. For example, Figure 9 illustrates a computer system 900 suitable for implementing certain embodiments of the disclosed subject matter.

Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through interpretation, microcode, etc.

The instructions can be executed on various types of computers or their components, including personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things devices.

The components for computer system 900 shown in Figure 9 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. Nor should the configuration of the components be construed as having any dependencies or requirements relating to any one or a combination of components described in the embodiments of computer system 900.

Computer system 900 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images acquired from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), etc.

The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard 901, mouse 902, touchpad 903, touch screen 910, data glove 904, joystick 905, microphone 906, scanner 907, camera 908.

Computer system 900 may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell/taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback from touchscreen 910, data gloves 904, or joystick 905, but may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers 909, headphones (not depicted)), and visual output devices (e.g., screens 910 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which are capable of outputting two-dimensional or more three-dimensional visual outputs through components such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media including CD/DVD ROM/RW 920 with media such as CD/DVD 921, finger drives 922, removable hard disk drives or solid-state drives 923, conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM/ASIC/PLD such as security dongles (not depicted), etc.

Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

Computer system 900 may also include interfaces to one or more communication networks (955). Networks (955) may be, for example, wireless networks, wired networks, or optical networks. Networks (955) may further be local area networks, wide area networks, metropolitan area networks, vehicle and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of networks (955) include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cloud and the like, cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial television including CANBus, etc. Some networks (955) typically require an external network interface adapter (954) (e.g., a USB port of computer system 900) to connect to some general-purpose data port or peripheral bus (949); other network interfaces are typically integrated into the core of computer system 900 by connecting to the system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system), as described below. Computer system 900 can communicate with other entities using any of these networks (955). Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., CANbus connected to certain CANbus devices), or bidirectional, such as using a local area network or wide area network to connect to other computer systems. As mentioned above, certain protocols and protocol stacks can be used on each of those networks (955) and network interfaces (954).

The aforementioned human-machine interface devices, human-machine accessible storage devices, and network interfaces can be attached to the kernel 940 of the computer system 900.

The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) 943, hardware accelerators 944 for certain tasks, etc. These devices, along with read-only memory (ROM) 945, random access memory 946, graphics adapter 950, and internal mass storage 947 such as internal non-user-accessible hard disk drives (SDs), etc., can be connected via the system bus 948. In some computer systems, the system bus 948 can be accessed via one or more physical connectors to allow for expansion by adding CPUs, GPUs, etc. Peripheral devices can be directly connected to the core's system bus 948 or connected via a peripheral bus 949. Peripheral bus architectures include PCI, USB, etc.

The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute certain instructions, which can be combined to form the aforementioned computer code. The computer code can be stored in ROM 945 or RAM 946. Transient data can also be stored in RAM 946, while permanent data can be stored, for example, in internal mass storage 947. Fast storage and retrieval of any storage device can be achieved using a cache, which can be closely associated with one or more CPUs 941, GPUs 942, mass storage 947, ROM 945, RAM 946, etc.

Computer-readable media may have computer code thereon for performing various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

As a non-limiting example, a computer system having architecture 900, particularly kernel 940, can provide functionality by having one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as some non-transitory memory of kernel 940, such as internal mass storage 947 or ROM 945. Software implementing various embodiments of this disclosure can be stored in such devices and executed by kernel 940. Depending on specific needs, the computer-readable media may include one or more storage devices or chips. The software can cause kernel 940, particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 946 and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may provide functionality through hard-wired or otherwise embodied logic in circuitry (e.g., accelerator 944), which may replace or operate with the software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

Although several exemplary embodiments have been described in this disclosure, modifications, substitutions, and various equivalent alternatives that fall within the scope of this disclosure exist. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and thus fall within its spirit and scope.

Claims (13)

1. A method for decoding encoded images of an encoded video sequence, the method being executed by at least one processor, and the method comprising: Decode the syntax elements related to the resolution of the reference image from the first high-level syntax structure used for multiple images; A second high-level syntax structure that varies from a first encoded image to a second encoded image decodes syntax elements related to the resolution of the decoded tiles, wherein each of the first encoded image and the second encoded image contains multiple tiles; Samples from the reference image buffer are resampled for use by the decoder for prediction. The decoder decodes the current tile at the decoding resolution, and the samples from the reference image buffer are at the reference image resolution. Decode the current tile at the resolution of the decoded tile into a decoded tile at the resolution of the decoded tile; and The decoded image tiles are stored in the reference image buffer, and the decoded image tiles are resampled to the reference image resolution; In response to the existence of samples not assigned to any tile, the samples are assigned to other tiles, wherein the samples in the other tiles are encoded in a skip mode. 2. The method of claim 1, wherein the resampling filter is used for at least one of: resampling the samples from the reference image buffer for use by the decoder for prediction; and resampling the decoded patches to the reference image resolution, wherein the resampling filter is computationally more complex and not adaptive than a bilinear filter. 3. The method of claim 2, wherein the resampling filter is selected from a plurality of resampling filters based on the relationship between the decoding resolution and the reference image resolution. 4. The method according to any one of claims 1 to 3, further comprising: Based on the third-level grammatical structure, decode the grammatical elements related to the output resolution; and The samples of the decoded tiles are resampled to the output resolution. 5. The method according to any one of claims 1 to 3, wherein the resampling uses resampling factors that are different in width and height. 6. An apparatus for decoding encoded images of an encoded video sequence, the apparatus comprising: At least one memory is configured to store computer program code; and At least one processor is configured to access the at least one memory and operate according to the computer program code, the computer program code comprising: The first decoding code is configured to decode syntax elements related to the resolution of a reference image from a first high-level syntax structure used for multiple images; The second decoding code is configured to vary from the first encoded image to the second encoded image, and to decode syntax elements related to the resolution of the decoded tiles, wherein each of the first encoded image and the second encoded image contains multiple tiles; The resampling code is configured to resample samples from a reference image buffer for use by the decoder for prediction, the decoder decoding the current patch at a decoding resolution, and the samples from the reference image buffer are at the reference image resolution; The third decoding code is configured to decode the current tile at the resolution of the decoded tile into a decoded tile at the resolution of the decoded tile; and The storage code is configured to store the decoded tile in the reference image buffer and resample the decoded tile to the reference image resolution; In response to the existence of samples not assigned to any tile, the samples are assigned to other tiles, wherein the samples in the other tiles are encoded in a skip mode. 7. The apparatus of claim 6, wherein the resampling filter is used for at least one of: resampling the samples from the reference image buffer for use by the decoder for prediction; and resampling the decoded patches to the reference image resolution, wherein the resampling filter is computationally more complex and not adaptive than a bilinear filter. 8. The apparatus of claim 7, wherein the resampling filter is selected from a plurality of resampling filters based on the relationship between the decoding resolution and the reference image resolution. 9. The apparatus according to any one of claims 6 to 8, wherein the resampling uses resampling factors that are different in width and height. 10. A non-transitory computer-readable storage medium storing a program for decoding encoded images of an encoded video sequence, the program comprising instructions that cause a processor to: Decode the syntax elements related to the resolution of the reference image from the first high-level syntax structure used for multiple images; A second high-level syntax structure that varies from a first encoded image to a second encoded image decodes syntax elements related to the resolution of the decoded tiles, wherein each of the first encoded image and the second encoded image contains multiple tiles; Samples from the reference image buffer are resampled for use by the decoder for prediction. The decoder decodes the current tile at the decoding resolution, and the samples from the reference image buffer are at the reference image resolution. Decode the current tile at the resolution of the decoded tile into a decoded tile at the resolution of the decoded tile; and The decoded image tiles are stored in the reference image buffer, and the decoded image tiles are resampled to the reference image resolution; In response to the existence of samples not assigned to any tile, the samples are assigned to other tiles, wherein the samples in the other tiles are encoded in a skip mode. 11. The non-transitory computer-readable storage medium of claim 10, wherein the resampling filter is used for at least one of: resampling the samples from the reference image buffer for use by the decoder for prediction; and resampling the decoded tiles to the reference image resolution, wherein the resampling filter is computationally more complex and non-adaptive than a bilinear filter. 12. The non-transitory computer-readable storage medium of claim 11, wherein the resampling filter is selected from a plurality of resampling filters based on the relationship between the decoding resolution and the reference image resolution. 13. An apparatus for decoding encoded images of an encoded video sequence, the apparatus comprising: The first decoding module is configured to decode syntax elements related to the resolution of a reference image from a first high-level syntax structure used for multiple images; The second decoding module is configured to use a second high-level syntax structure that varies from the first encoded image to the second encoded image to decode syntax elements related to the resolution of the decoded tiles, wherein each of the first encoded image and the second encoded image contains multiple tiles; A resampling module is configured to resample samples from a reference image buffer for use by a decoder for prediction, the decoder decoding the current patch at a decoding resolution, and the samples from the reference image buffer being at the reference image resolution; The third decoding module is configured to decode the current tile at the resolution of the decoded tile into a decoded tile at the resolution of the decoded tile; and The storage module is configured to store the decoded tiles in the reference image buffer and resample the decoded tiles to the reference image resolution; In response to the existence of samples not assigned to any tile, the samples are assigned to other tiles, wherein the samples in the other tiles are encoded in a skip mode.