JP4762486B2 - Multi-resolution video encoding and decoding - Google Patents

Description

  The present invention relates to multi-resolution video coding and decoding. For example, video encoders adaptively change the video frame size to reduce blocking artifacts at low bit rates.

  Digital video consumes a large amount of storage and transmission capacity. A typical raw digital video sequence contains 15 or 30 frames per second. Each frame may contain tens of thousands or hundreds of thousands of pixels (also called pels). Each pixel represents a very small element of the image. In raw form, computers typically represent pixels in 24 bits. Thus, the number of bits or bit rate per second of a normal raw digital video sequence can be 5 million bits / second or more.

  Most computers and computer networks lack the resources to process live digital video. For this reason, engineers use compression (also called coding or encoding) to reduce the bit rate of digital video. The compression can be lossless, in which the video quality is not compromised, but the bit rate reduction is limited by the complexity of the video. Alternatively, compression can be lossy, which results in poor video quality, but the reduction in bit rate is more dramatic for the results in lossless compression. Decompression reverses compression.

  In general, video compression techniques include intra-frame (intraframe) compression and inter-frame (interframe) compression. Intra frame compression techniques compress individual frames, which are usually referred to as I-frames, key frames, or reference frames. Inter-frame compression techniques compress frames with reference to preceding and / or following frames and are commonly referred to as predicted frames, P frames, or B frames.

  Many intra-frame and inter-frame compression techniques are block-based. A video frame is divided into a plurality of blocks for encoding. For example, an I frame is divided into 8 × 8 blocks and these blocks are compressed. Alternatively, a P frame is divided into 16 × 16 macroblocks (eg, four 8 × 8 luminance blocks and two 8 × 8 chrominance blocks), and these macroblocks are compressed. Different implementations can use different block configurations.

US Pat. No. 6,499,060 US Pat. No. 6,418,166

  Standard video encoders suffer a dramatic degradation in performance when the target rate falls below a certain threshold. In block-based video compression and decompression, quantization and other irreversible processing stages generally result in blocking artifacts, ie distortion that appears as perceptible discontinuities between multiple blocks. At low bit rates, high frequency information about blocks of I frames can be severely distorted or lost completely. Similarly, high frequency information about block residuals of P frames (parts of P frames that were not predicted by motion estimation or other predictions) can be severely distorted or lost entirely. As a result, significant blocking artifacts can occur in the “low pass” region, which can cause a substantial reduction in the quality of the reconstructed video.

Some previous encoders have attempted to reduce the perceptibility of blocking artifacts by processing the reconstructed frame with a deblocking filter (deblocking filter). The deblocking filter smoothes the boundary between the blocks. A deblocking filter can improve the perceived video quality, but this has several drawbacks. For example , the filter works only at the output reconstructed at the decoder. Therefore, even when in-loop deblocking is in use, the effect of deblocking cannot be incorporated as an element in the motion estimation, motion compensation or transform coding process for the current frame. On the other hand, smoothing of the current frame with a post-processing filter (ie, out-of-loop deblocking) can be too extreme, and the smoothing process introduces unnecessary computational complexity.

  Given the critical importance of video compression and decompression to digital video, it is not surprising that video compression and decompression is a well-developed field. However, whatever the advantages of previous video compression and decompression techniques, they do not have the advantages of the following techniques and tools.

  In summary, the detailed description is directed to various techniques and tools for multi-resolution video coding. For example, video encoders adaptively change the video frame size to reduce blocking artifacts at low bit trades. Doing so reduces the blocking artifacts, but may increase blurring, which is less perceptible and less unpleasant than blocking artifacts. Various techniques and tools can be used in combination or independently.

  In one aspect, the video encoder encodes the video with any of a number of spatial resolutions. The encoder encodes at least one frame in a sequence of multiple video frames with a first spatial resolution and encodes at least one other frame with a second spatial resolution. The second spatial resolution is different from the first spatial resolution, and the encoder selects the second spatial resolution from a set of multiple spatial resolutions to reduce blocking artifacts in the sequence of video frames. To do.

  In other aspects, the encoder encodes a first portion of the frame with a first spatial resolution and encodes a second portion of the frame with a second spatial resolution. The second spatial resolution is different from the first spatial resolution.

  In other aspects, the video encoder includes a first code in the bitstream that indicates the first spatial resolution for a first frame encoded with the first spatial resolution, and the second spatial resolution. The bit stream includes a second code indicating a second spatial resolution for the second frame encoded in step (b). The second spatial resolution is different from the first spatial resolution, and the encoder selects the second spatial resolution from a set of multiple spatial resolutions to reduce blocking artifacts in the sequence of video frames. To do.

  In other aspects, the encoder includes a first signal in the bitstream indicating a first spatial resolution for a first portion of the frame and a first indicating a second spatial resolution for the second portion of the frame. 2 signals are included in the bitstream. The second spatial resolution is different from the first spatial resolution.

  In other aspects, the decoder receives the multi-resolution signal in a sequence header for a video sequence of multiple encoded frames. The multi-resolution signal indicates whether multiple frames are encoded with multiple spatial resolutions. If the multiple frames are encoded with multiple spatial resolutions, the decoder decodes the first encoded frame with the first spatial resolution and converts the second encoded frame to the second Decrypt with spatial resolution.

  In other aspects, the decoder decodes the first portion of the encoded frame with a first spatial resolution and decodes the second portion of the encoded frame with a second spatial resolution. To do. The second spatial resolution is different from the first spatial resolution.

  In other aspects, an encoder or decoder receives pixel data for a video image and adapts and modifies the spatial resolution of the video image, which resamples the resampled pixel data. , Using a 6 tap down sampling filter or a 10 tap upsampling filter.

  Additional features and advantages will become apparent from the following detailed description of various embodiments, which proceeds with reference to the accompanying figures.

  Detailed embodiments of the present invention are directed to multi-resolution video encoding and decoding. For example, video encoders adaptively change the video frame size to reduce blocking artifacts at low bit trades. In doing so, the encoder reduces blocking artifacts but may increase blurring, which is less perceptible and less desirable than blocking artifacts.

  In some embodiments, the encoder uses multi-resolution encoding techniques and tools to encode the input frame with different spatial resolutions. For example, in one implementation, the encoder encodes the frame with the largest original resolution, encodes with a resolution down-sampled by a factor of two in the horizontal direction, or halves in the vertical direction. Or with a resolution down-sampled in half in the horizontal and vertical directions. Alternatively, the encoder may reduce or increase the resolution of a frame encoded by some other factor relative to the original resolution, or encode by a factor relative to the current resolution. Decrease or increase the resolution of a given frame, or set the resolution using some other technique. The decoder decodes the encoded frame using a corresponding technique.

  In some embodiments, the encoder selects a spatial resolution for the frame on a per frame basis or some other basis. The decoder performs the corresponding adjustment.

  In some embodiments, the encoder selects a spatial resolution by evaluating certain criteria (eg, bit rate, frame content, etc.).

  Various techniques and tools can be used in combination or independently. Individual embodiments implement one or more of the techniques and tools described. Different techniques and tools can be combined, used independently, or with other techniques and tools.

I. Computing Environment FIG. 1 illustrates a general example of a suitable computing environment (100) in which the described embodiments may be implemented. The computing environment (100) is not intended to suggest any limitation as to the scope of use or functionality of the invention, as the invention can be implemented in a variety of general purpose or special purpose computing environments. It is.

  With reference to FIG. 1, the computing environment (100) includes at least one processing unit (110) and memory (120). In FIG. 1, this most basic configuration (130) is contained within a dashed line. The processing unit (110) executes computer-executable instructions and may be a real or virtual processor. In a multi-processing system, the multi-processing device executes computer-executable instructions to increase processing power. The memory (120) can be volatile memory (eg, registers, cache, RAM), non-volatile memory (eg, ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory (120) stores software (180) that implements multi-resolution encoding and / or decoding techniques.

  A computing environment may have additional features. For example, the computing environment (100) includes storage (140), one or more input devices (150), one or more output devices (160), and one or more communication connections (170). An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects multiple components of the computing environment (100). Typically, operating system software (not shown) provides an operating environment for other software executing in the computing environment (100) and coordinates the activities of the components of the computing environment (100).

  Storage (140) can be removable or non-removable, which can be used to store magnetic disks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or information, and Any other media that can be accessed within the storage environment (100) is included. Storage (140) stores instructions for software (180) that implements multi-resolution encoding and / or decoding techniques.

  Input device (150) is a touch input device such as a keyboard, mouse, pen or trackball, voice input device, scanning device, network adapter, or another device that provides input to the computing environment (100) Can be. For video, the input device (150) is a TV tuner card, camera video interface, or similar device that accepts video input in analog or digital format, or a CD-ROM / that provides video input to the computing environment. It can be a DVD reader. The output device (160) can be a display, printer, speaker, CD / DVD writer, network adapter, or another device that provides output from the computing environment (100).

  A communication connection (170) allows communication to another computing entity via a communication medium. The communication medium carries information such as computer-executable instructions, compressed video information, or other data in the modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired or wireless techniques implemented in electrical, optical, RF, infrared, acoustic or other carrier waves.

  The invention can be described in the general context of computer-readable media. Computer readable media can be any available media that can be accessed within a computing environment. By way of example, and not limitation, computer readable media within computing environment (100) include memory (120), storage (140), communication media, and combinations of any of the above.

  The invention may be described in the general context of computer-executable instructions contained in program modules that are executed within a computing environment on a real or virtual processor of interest. Generally, program modules include those that perform particular tasks or implement particular abstract data types, such as routines, programs, libraries, objects, classes, components, data structures, and the like. Program module functionality may be combined or divided among a plurality of program modules as desired in various embodiments. Computer-executable instructions for program modules may be executed within a local or distributed computing environment.

  For the sake of presentation, the detailed description uses the terms “set”, “choose”, “encode” and “decode” and uses the terms “compute” and “decode”. A computer operation in a computing environment will be described. These terms are high-level abstractions about operations performed by a computer and should not be confused with acts performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.

II. Video Encoder and Decoder Examples Techniques and tools in various embodiments may be implemented in a video encoder and / or decoder. Video encoders and decoders can include different modules within them, and multiple different modules can be related to and communicate with one another in a number of different ways. The modules and relationships described below are exemplary.

  Depending on the implementation and the type of compression desired, encoder or decoder modules can be added, omitted, split into multiple modules, combined with other modules, and / or replaced with similar modules. . In alternative embodiments, encoders or decoders having different module and / or other module configurations perform one or more of the described techniques.

  The exemplary encoder and decoder are block based and use a 4: 2: 0 macroblock format, with each macroblock having four 8 × 8 luminance blocks (sometimes one 16 × 16 macroblock). And two 2 × 8 chrominance blocks. Alternatively, the encoder and decoder are object based and use different macroblocks or block formats, or in a set of pixels of a different size or configuration than 8x8 blocks and 16x16 macroblocks. Perform the operation.

A. Example of Video Encoder An encoder receives a sequence of video frames including a current frame and generates compressed video information as an output. The encoder compresses the predicted frame and the key frame. Many of the components of the encoder are used to compress both key frames and predicted frames. The exact operations performed by these components can vary depending on the type of information being compressed.

  Predicted frames (also called P frames, B frames in bi-directional prediction, or inter-coded frames) are expressed in terms of prediction (or difference) from one or more other frames Is done. The prediction residual is the difference between what was predicted and the original frame. In contrast, key frames (also called I-frames, intra-coded frames) are compressed regardless of other frames.

  Referring to FIG. 2 in some embodiments, the encoder (200) that encodes the current frame (205) includes a resolution converter (210) for multi-resolution encoding. The resolution converter (210) receives the current frame (205) as input, and outputs the multi-resolution parameter (215) as well as the converted frame. If the current frame (205) is a predicted frame, the resolution converter (230) receives the reference frame (225) for the current frame (205) as input and outputs the converted reference frame. .

  The resolution converters (210) and (230) are in communication with other encoding modules (240), and the other encoding modules (240) were provided by the resolution converters (210) and (230). Generate output (245) (eg, pixel block data, motion vectors, residuals, etc.) based in part on multi-resolution encoding information (eg, multi-resolution parameters (215)).

  Other encoder modules (240) may include, for example, a motion estimator, motion compensator, frequency converter, quantizer, frame store, and entropy encoder.

  If the current frame (205) is a forward prediction frame, the motion estimator estimates the motion of the macroblock or other set of pixels of the current frame (205) relative to the reference frame (225). The reference frame in this case is the previous reconstructed frame buffered in the frame store. In an alternative embodiment, the reference frame (225) is a later frame, or the current frame (205) is predicted bi-directionally. The motion compensator applies motion information to the reconstructed previous frame to form a motion compensated current frame. However, the prediction is rarely complete and the difference between the motion compensated current frame and the original current frame (205) is the prediction residual. Alternatively, the motion estimator and motion compensator apply another type of motion estimation / compensation.

  The frequency converter converts spatial domain video information into frequency domain (ie, spectrum) data. For block-based video frames, the frequency transformer applies a discrete cosine transform (DCT) to the block of pixel data or predicted residual data to generate a block of DCT coefficients. Alternatively, the frequency transformer applies another conventional frequency transform, such as a Fourier transform, or uses wavelet or sub-band analysis.

  The quantizer then quantizes the block of spectral data coefficients. The quantizer applies uniform scalar quantization to the spectral data using a step size that varies on a frame-by-frame basis or other basis. Alternatively, the quantizer can apply another type of quantization to the spectral data coefficients, such as non-uniform, vector, or non-adaptive quantization, or it can be spatially used in an encoder system that does not use frequency transforms. Quantize region data directly. In addition to adaptive quantization, the encoder (200) can use frame drops, adaptive filtering, or other techniques for rate control.

  When the reconstructed current frame is needed for subsequent motion estimation / compensation, the encoder (200) module reconstructs the current frame (205), which typically encodes the frame. Perform the reverse of the technique used to The frame store buffers the reconstructed current frame for use in predicting the next frame.

  The entropy encoder compresses the quantizer output as well as some sub-information (eg, motion information, quantization step size, etc.). Typical entropy coding techniques include arithmetic coding, differential coding, Huffman coding, run length coding, LZ coding, dictionary coding, and combinations or variations of the above. Entropy encoders typically use different encoding techniques for different types of information and can choose from a number of code tables within a particular encoding technique.

  For additional details on other encoder modules (240) in some embodiments, see US patent application Ser. No. 09 / 849,502, filed May 3, 2001, entitled “DYNAMIC FILTERING FOR LOSSY COMPRESSION”. U.S. patent application Ser. No. 09 / 201,278, entitled “EFFICIENT MOTION VECTOR CODING FOR VIDEO COMPRESSION”, filed Nov. 30, 1998, U.S. Pat. Each of these disclosures is incorporated herein by reference.

B. Video Decoder Embodiment Referring to FIG. 3, the decoder (300) receives information about a sequence of compressed video frames and generates an output that includes the reconstructed frames. The decoder (300) decompresses the predicted frame and the key frame. Many of the components of the decoder (300) are used to decompress both key frames and predicted frames. The exact operations performed by these components can vary depending on the type of information being compressed.

  In some embodiments, the decoder (300) that reconstructs the current frame (305) includes a resolution converter (310) for multi-resolution decoding. The resolution converter (310) takes the decoded frame (315) as input and outputs a reconstructed current frame (305).

  If the current frame (305) is a predicted frame, the resolution converter (330) receives as input the multi-resolution parameter (315) and the reference frame (325) for the current frame (305). To do. The resolution converter (330) outputs the reference frame information to the other decoder module (340). Another decoder module (340) uses the reference frame information along with the motion vectors, residuals, etc. (345) received from the encoder to decode the current frame (305).

  Other decoder modules 340 may include, for example, buffers, entropy decoders, motion compensators, frame stores, inverse quantizers, and inverse frequency converters.

  The buffer receives information (345) about the compressed video sequence and makes the received information available to the entropy decoder. The buffer typically includes a jitter buffer for receiving information at a rate that is fairly constant over time and smoothing short-term variations in bandwidth or transmission. Buffers can also include playback buffers and other buffers. Alternatively, the buffer receives information at a changing rate.

  The entropy decoder decodes the entropy-encoded quantized data, as well as the entropy-encoded sub-information (eg, motion information, quantization step size, etc.), and usually entropy performed in the encoder Apply the opposite of encoding. Entropy decoders often use different decoding techniques for different types of information and can choose from a number of code tables within a particular decoding technique.

  If the frame to be reconstructed is a forward predicted frame, the motion compensator applies motion information to the reference frame to form a prediction of the frame that is being reconstructed. For example, the motion compensator uses the macroblock motion vector to find the macroblock in the reference frame. The frame buffer stores the previous reconstructed frame for use as a reference frame. Alternatively, the motion compensator applies another type of motion compensation. The prediction by the motion compensator is rarely complete, so the decoder also reconstructs the prediction residual.

  When the decoder needs a reconstructed frame for subsequent motion compensation, the frame store buffers the reconstructed frame for use in predicting the next frame.

  The inverse quantizer performs inverse quantization on the entropy-decoded data. In general, an inverse quantizer applies uniform scalar inverse quantization to entropy decoded data using a step size that varies on a frame-by-frame basis or other basis. Alternatively, the inverse quantizer does not apply another type of inverse quantization to the data, such as non-uniform, vector, or non-adaptive inverse quantization, or does not use inverse frequency transforms on spatial domain data Direct dequantization in the decoder system.

  The inverse frequency converter converts the quantized frequency domain data into spatial domain video information. For block-based video frames, the inverse frequency transformer applies inverse DCT (IDCT) to the block of DCT coefficients to generate pixel data or prediction residual data for the key frame or prediction frame, respectively. . Alternatively, the frequency transformer applies another conventional inverse frequency transform, such as a Fourier transform, or uses wavelets or sub-band synthesis.

  For additional details about other decoder modules (340) in some embodiments, see US patent application Ser. No. 09 / 849,502, filed May 3, 2001, entitled “DYNAMIC FILTERING FOR LOSSY COMPRESSION”. U.S. patent application Ser. No. 09 / 201,278, entitled “EFFICIENT MOTION VECTOR CODING FOR VIDEO COMPRESSION”, filed Nov. 30, 1998, U.S. Pat.

III. Multi-resolution video encoding and decoding In multi-resolution encoding, an encoder encodes an input frame with different spatial resolutions. The encoder selects the spatial resolution for the frame on a frame-by-frame basis or on another basis. In some embodiments, the encoder selects a spatial resolution based on the following observations.
1. As the bit rate decreases, the benefits of encoding with lower spatial resolution increase.
2. As the quantization step size increases, the benefits of coding with lower spatial resolution increase.
3. Because down-sampling discards high frequency information, down-sampling is sometimes not well suited for frames with perceptually important high frequency content (eg, “strong edges”, text, etc.).
4). Downsampling may be appropriate if the frame has low frequency characteristics, or if the frame has high frequency content such as noise.

  In some embodiments, the encoder uses the current frame bit rate, quantizer step size, and orientation / magnitude for high frequency energy to select spatial resolution. For example, if the magnitude of the horizontal high frequency component of the current frame is large but the magnitude of the vertical high frequency component is small, the encoder selects vertical down-sampling. In other embodiments, the encoder uses information from the reference frame (instead of or in combination with information from the current frame) to select a spatial resolution. Alternatively, the encoder may omit some or all of the above criteria, use other criteria instead of some of the above criteria, or use additional criteria to select spatial resolution. can do.

  After the encoder selects a spatial resolution for the current frame, the encoder resamples the original frame to the desired resolution before encoding it. If the current frame is a predicted frame, the encoder also resamples the reference frame for the predicted frame to match the new resolution of the current frame. The encoder then sends a resolution selection to the decoder. In one implementation, a 6-tap filter is used for down-sampling, a 10-tap filter is used for up-sampling, and these filters combine to increase the quality of the reconstructed video Designed as such. Alternatively, other filters are used.

  FIG. 4 shows a technique (400) for multi-resolution encoding of frames. An encoder, such as the encoder (200) of FIG. 2, sets up a resolution for the frame (410). For example, the encoder considers the criteria listed above or other criteria.

  The encoder then encodes the frame (420) with that resolution. If the encoding is complete (430), the encoder ends. Otherwise, the encoder sets a resolution for the next frame (410) and continues encoding. Alternatively, the encoder sets the resolution to some level other than the frame level.

  In some embodiments, the encoder encodes predicted frames as well as intra frames. FIG. 6 shows a technique (600) for multi-resolution encoding of intra frames and predicted frames.

  Initially, the encoder checks whether the current frame to be encoded is an I frame or a P frame (610). If the current frame is an I frame, the encoder sets a resolution for the current frame (620). If the frame is a P-frame, the encoder sets a resolution for the reference frame (630) before setting a resolution for the current frame (620).

  After setting the resolution for the current frame (620), the encoder encodes the current frame with that resolution (640). If the encoding is complete (650), the encoder ends. Otherwise, the encoder continues to encode.

  In some implementations, the encoder selectively encodes the frame with one of the following resolutions: 1) maximum resolution, 2) resolution down-sampled by a factor of 2 in the horizontal direction, 3) resolution down-sampled by a factor of 2 in the vertical direction, or 4) horizontal and vertical direction. Resolution downsampled by a factor of 2. Alternatively, the encoder may reduce or increase the resolution by some other factor (eg, not a power of 2), have an additional usable resolution, or use some other technique. Use to set the resolution. The encoder sets the resolution for each frame to the original image size. Alternatively, the encoder sets the resolution for the frame relative to the resolution of the previous frame or the previous resolution setting, i.e. the encoder progressively advances the resolution relative to the previous resolution. Change progressively.

  The decoder decodes the encoded frame and, if necessary, up-samples the frame before displaying it. Like the resolution of the encoded frame, the resolution of the decoded frame can be adjusted in a number of different ways.

  FIG. 5 shows a technique (500) for multi-resolution decoding of a frame. A decoder, such as the decoder (300) of FIG. 3, sets up a resolution for the frame (510). For example, the decoder obtains resolution information from the encoder.

  The decoder then decodes the frame with the resolution (520). If decoding is complete (530), the decoder ends. Otherwise, the decoder sets a resolution for the next frame (510) and continues decoding. Alternatively, the decoder sets the resolution to some level other than the frame level.

  In some embodiments, the decoder decodes prediction frames as well as intra frames. FIG. 7 shows a technique (700) for multi-resolution decoding of intra frames and predicted frames.

  Initially, the decoder checks 710 whether the current frame to be decoded is an I frame or a P frame. If the current frame is an I frame, the decoder sets a resolution for the current frame (720). If the frame is a P frame, the decoder sets the resolution for the reference frame (730) before setting the resolution for the current frame (720).

  After setting the resolution for the reference frame (720), the decoder decodes the current frame with that resolution (740). If the decoding is complete (750), the decoder ends. Otherwise, the decoder continues decoding.

  The decoder typically decodes the frame with the resolution used by the encoder, eg, one of the above-described resolutions. Alternatively, the resolution available at the decoder is not exactly the same as that used at the encoder.

A. Signaling In order to provide the decoder with enough information to decode the multi-resolution encoded frame, the encoder uses bit stream signaling. For example, the encoder may indicate whether a sequence of frames is encoded using multi-resolution encoding and / or a signal indicating resolution of frames encoded in the sequence 1 It can be sent in the form of one or more flags or codes. Alternatively, the encoder enables / disables multi-resolution encoding at some level other than the sequence level and / or sets the resolution at some level other than the frame level.

  FIG. 8 shows a technique (800) for transmitting a signal when encoding a sequence of frames with multi-resolution encoding. The encoder takes the next sequence header as input (810) and determines whether multi-resolution encoding should be enabled for this sequence (820).

  If the encoder is not using multi-resolution encoding for this sequence, the encoder sets the signal for that sequence accordingly (830) and encodes the frames in that sequence (840). ).

  If the encoder is using multi-resolution encoding, the encoder sets the signal of the sequence accordingly (850). The encoder then encodes the frames in the sequence according to a signal indicating a scaling factor for the horizontal and / or vertical resolution of the frame (eg, as described above with reference to FIG. 4 or FIG. 6). Turn into.

  If the encoding is complete (870), the encoder ends. Otherwise, the encoder encodes the next sequence.

  In some embodiments, the encoder transmits one bit that indicates whether multi-resolution encoding is enabled for the sequence of frames. Then, for the frames in the sequence, the code in the field specified for each of the I and P frames specifies the scaling factor for the resolution of that frame relative to the maximum resolution frame. In one implementation, the code is a fixed length code. Table 1 shows how the scaling factor is encoded in the field labeled RESPIC FLC.

  Alternatively, the encoder uses another method (eg, a variable length code) that signals an adjustment to frame resolution. Depending on the implementation and the number of possible resolutions, the encoder can use additional signal codes or fewer signal codes, or can use different codes for horizontal and vertical resolution. . Further, depending on the relative probability of possible resolution, the encoder can adjust the length of the code (eg, assign a shorter code to the most likely resolution). In addition, the encoder can use the signal for other purposes. For example, the encoder can use a signal (eg, a fixed or variable length code) to indicate which filter should be used for resampling in situations where multiple filters are available. . The encoder can use such a signal to indicate whether an available predefined filter or a custom filter should be used in resampling.

  By transmitting the signal, the encoder provides the decoder with useful information for decoding the multi-resolution encoded frame. The decoder parses the signal to determine how the encoded frame should be decoded. For example, the decoder interprets the code transmitted by the encoder to determine whether the sequence of frames is encoded using multi-resolution encoding and / or the code in the sequence The resolution of the normalized frames can be determined.

  FIG. 9 shows a technique (900) for receiving and interpreting signals when decoding a sequence of encoded frames with multi-resolution decoding. The decoder takes the next encoded sequence header as input (910), checks the signal associated with this sequence, and whether the encoder used multi-resolution encoding for this sequence A decision is made (920).

  If the encoder did not use multi-resolution encoding, the decoder decodes (930) the frames in the sequence. On the other hand, if the encoder used multi-resolution encoding, the decoder parses a signal indicating the horizontal and / or vertical resolution scaling factors for that frame (940). The decoder then decodes (950) the frame accordingly (eg, as described above with reference to FIG. 5 or FIG. 7).

  If the decoding is complete (960), the decoder ends. Otherwise, the decoder decodes the next sequence.

B. Downsampling and Upsampling The following sections describe the upsampling and downsampling processes in some implementations. Other implementations use different up-sampling, down-sampling or filtering techniques. For example, in an alternative embodiment, the encoder may encode the frame using a non-linear filter or a spatially varying filter bank.

  Table 2 shows the variable definitions used by the encoder and / or decoder for down-sampling / up-sampling of the frame. These definitions will be used in the following description of pseudo code for the down-sampling and up-sampling embodiments.

Table 2: Variable definitions for downsampling / upsampling in some implementations
N _u = number of samples in the up-sampled (maximum resolution) line
N _d = number of samples in the down-sampled (semi-resolution) line
X _u [n] = sample value up-sampled at position n, where n = 0, 1, 2,... N _u −1
X _d [n] = sampled value down-sampled at position n, where n = 0, 1, 2,... N _d −1

  The term “line” refers to a group of samples in horizontal-row or vertical-column in the Y, Cr or Cb component plane. In the following example, up-sampling and down-sampling operations are equal for both rows and columns, and are therefore illustrated using a one-dimensional line sample. When vertical and horizontal up-sampling or down-sampling is performed, the horizontal line is resampled first, followed by the vertical line. Alternatively, horizontal and vertical filtering is performed simultaneously on the block of pixels using different filters.

  Table 3 shows the pseudo code for luminance line resampling, and Table 4 shows the pseudo code for chrominance line resampling.

Table 3: Pseudo code for luminance line resampling in some implementations
N _d = N _u / 2 (where N _u is the number of samples in the maximum resolution luminance line)
if ((N _d & 15)! = 0)
N _d = N _d + 16- (N _d & 15)

Table 4: Pseudo code for chrominance line resampling in some embodiments
N _d = N _u / 2, where N _u is the number of samples in the chrominance line for maximum resolution
if ((N _d & 7)! = 0)
N _d = N _d + 8- (N _d & 7)

  Resampling sets the number of samples for the down-sampled line. The resampling then adjusts the number of samples in the line (in encoders operating with 4: 2: 0 or similar macroblocks) and this line is a multiple of the macroblock for the luminance line (ie a multiple of 16) ), Or multiples of blocks for chrominance lines (ie, multiples of 8).

1. Down-sampling filter Line down-sampling produces an output according to the pseudo code in Table 5.

Table 5: Pseudocode for line downsampling in some implementations
if (N _d ! = (N _u / 2))
{
for (i = N _u ; i <N _d * 2; i ++)
x _u [i] = x _u [N _u -1]
}
downsamplefilter_line (x _u [])
for (i = 0; i <N _d ; i ++)
x _d [i] = x _u [i * 2]
The code for the 6-tap filter used in downsample ef filter_line () is shown in FIG.

  In FIG. 10, RND_DOWN is set to the value 64 when the image is filtered in the horizontal direction, and set to the value 63 when the image is filtered in the vertical direction.

2. Up-sampling filter Line up-sampling produces an output according to the pseudo code in Table 6.

Table 6: Pseudocode for line upsampling in some implementations
for (i = 0; i <N _u ; i ++)
{
x _u [i] = x _d [i * 2]
x _u [i + 1] = 0
}
upsamplefilter_line (x _u [])

An example code for a 10 tap filter used in upsampl ef filter_line () is shown in FIG. In FIG. 11, RND_UP is set to 15 when the image is filtered in the horizontal direction, and is set to 16 when the image is filtered in the vertical direction.

  Other filter pairs can also be used for resampling. The filter pair can be adjusted to the video content and / or target bit rate. The encoder can send the filter selection as side information to the decoder.

C. Calculation of New Frame Dimensions The pseudo code in Table 7 illustrates how the encoder calculates new frame dimensions for the down-sampled frames.

Table 7: Pseudocode for calculating new frame dimensions after down-sampling
X = horizontal dimension, number of samples in original resolution
Y = vertical dimension, number of samples in the original resolution
x = new horizontal resolution
y = new vertical resolution
hscale = horizontal scaling factor (0 = maximum resolution, 1 = half resolution)
vscale = vertical scaling factor (0 = maximum resolution, 1 = half resolution)
x = X
y = Y
if (hscale == 1)
{
x = X / 2
if ((x & 15)! = 0)
x = x + 16- (x & 15)
}
if (vscale == 1)
{
y = Y / 2
if ((y & 15)! = 0)
y = y + 16- (y & 15)
}

  In an implementation using the technique shown in Table 7, the encoder calculates a new frame dimension, which is down-sampling the original dimension by a factor of 2, and then the new dimension is a macroblock The fraction is rounded up to be an integer multiple of size (a multiple of 16). For chrominance lines, the encoder rounds the dimension up to be an integer multiple of block size (a multiple of 8). With the new dimension round-up, the down-sampled frame can be encoded by a video encoder / decoder using 4: 2: 0 or similar macroblock format.

D. Alternatives
With or in addition to the various alternatives described above, the encoder and decoder may operate as follows.

  The multi-resolution framework can be extended to several levels of down-sampling for individual frames or series of frames. By using several levels of down-sampling, the quality of the reconstructed frame can be improved when the encoder encodes a high resolution frame at a relatively low bit rate.

  The encoder can use a multi-rate filtering technique that resamples the frame to a resolution other than that achieved by adjusting the resolution relative to the original resolution by a factor of two. For example, fractional-rate sampling can provide a smoother tradeoff between maintaining high frequency details and reducing blocking artifacts at the cost of increased complexity. it can.

  The encoder can apply different levels of resampling to different parts of the frame. For example, an encoder can encode an area of a frame that has little high frequency content with down-sampled resolution, while an encoder can identify areas of the frame that have strong high frequency content. It can be encoded with the original resolution. In addition, the encoder can apply different filters to resample different portions of the frame, or for vertical and horizontal resampling. The encoder may use signaling to indicate different resampling levels and / or different filters used to resample different portions of the frame.

  Although the principles of the present invention have been described and illustrated with reference to various above-described embodiments, it will be understood that the above-described embodiments can be modified in arrangement and detail without departing from such principles. It should be understood that the programs, processes, or methods described herein are not related or limited to any particular type of computing environment, unless specifically stated otherwise. Various types of general purpose or special purpose computing environments may be used in conjunction with, or may perform, the operations according to the teachings described herein. Elements of the above-described embodiments shown in software can be implemented in hardware and vice versa.

  In view of the many possible embodiments to which the principles of the present invention may be applied, the present invention includes all such claims that may fall within the scope and spirit of the appended claims and their equivalents. Claim embodiment.

1 is a block diagram of a suitable computing environment in which the described embodiments can be implemented. FIG. 3 is a block diagram of a video encoder in which the described embodiments can be implemented. FIG. 3 is a block diagram of a video decoder in which the described embodiments can be implemented. Figure 3 is a flow diagram illustrating a generalized technique for multi-resolution encoding of frames. Fig. 3 is a flow diagram illustrating a generalized technique for multi-resolution decoding of frames. 2 is a flow diagram illustrating a technique for multi-resolution encoding of intra frames and predicted frames. 3 is a flow diagram illustrating a technique for multi-resolution decoding of intra and predicted frames. Fig. 4 is a flow diagram illustrating a technique for transmitting a signal when encoding a sequence of frames with multi-resolution encoding. FIG. 5 is a flow diagram illustrating a technique for receiving and interpreting signals when decoding a sequence of encoded frames with multi-resolution decoding. FIG. 4 is a pseudo code listing for a down-sampling filter in one implementation. FIG. 4 is a pseudo code listing for an up-sampling filter in one implementation.

Explanation of symbols

200 encoder 205 current frame 210 resolution converter 215 multi resolution parameter 225 reference frame 230 resolution converter 240 other encoder module 245 motion vector, residual etc. 300 decoder 305 current frame 310 resolution Transformer 315 Multi-resolution parameter, decoded frame 325 Reference frame 330 Resolution converter 340 Other decoder modules 345 Motion vector, residual, etc.

Claims (39)

A method for decoding a bitstream of a sequence of video frames, comprising:
Receiving first information about the sequence, the first information indicating whether second information is present in the bitstream ;
Determining that the second information is present in the bitstream based on the first information, and then determining the sequence based on a scaling factor of spatial resolution in response to the determination. For each of a plurality of frames of the sequence, the second information at the frame level in the bitstream, wherein one or more spatial resolution scalings for the frame Receiving second information indicative of a factor;
Multi-resolution resolution decoding the frames using the scaling factor;
The plurality of frames include at least one I frame and at least one P frame, and each of the at least one P frame has a reference I frame among the at least one I frame, and the at least one P frame. For each of the at least one P frame, the scaling factor of the one or more spatial resolutions for the P frame is the 1 or the reference I frame for the reference I frame of the P frame. A method characterized in that it is constrained to be the same as the scaling factor of multiple spatial resolutions.   The method of claim 1, wherein the scaling factor of the one or more spatial resolutions is adaptively determined based at least in part on a bit rate criterion.   The method of claim 1, wherein the scaling factor of the one or more spatial resolutions is adaptively determined based at least in part on a high frequency content criterion.   The method of claim 1, wherein the scaling factor of the one or more spatial resolutions is adaptively determined based at least in part on a quantization step size criterion.   The method of claim 1, wherein the second information is a fixed length code.   6. The method of claim 5, wherein the fixed length code is a 2-bit code that represents four possible states of the scaling factor of the one or more spatial resolutions.   The method of claim 1, wherein the second information is a variable length code.   The method of claim 1, wherein the first information is signaled in a sequence header.   9. The method of claim 8, wherein the first information is a 1-bit code in the sequence header.   The method of claim 1, further comprising receiving third information in the bitstream, the third information indicating a selected resampling filter.   The method of claim 1, wherein the one or more spatial resolution scaling factors comprise a vertical spatial resolution scaling factor and a horizontal spatial resolution scaling factor.   The method of claim 11, wherein the scaling factor of the vertical spatial resolution is different from the scaling factor of the horizontal spatial resolution.   12. The method of claim 11, wherein the vertical spatial resolution scaling factor is selected from a set of vertical spatial resolutions including full resolution and half resolution.   12. The method of claim 11, wherein the horizontal spatial resolution scaling factor is selected from a set of horizontal spatial resolutions including full resolution and half resolution. Decoding the plurality of frames with multi-spatial resolution decoding according to a scaling factor of the spatial resolution indicated by the second information;
The method of claim 1, further comprising displaying the plurality of frames. Decoding with the multi-spatial resolution decoding
Decoding the plurality of frames of current frames encoded with reduced spatial resolution;
After decoding the current frame, up-sampling the current frame, wherein the up-sampling comprises producing a decoded frame of full resolution. The method of claim 15.   The method of claim 16, wherein the up-sampling comprises applying a 10-tap filter to the decoded current frame.   The method of claim 16, wherein the displayed current frame with the reduced spatial resolution comprises reduced blocking artifacts. A method of encoding a bitstream of a sequence of video frames, the bitstream having multiple levels,
Outputting first information about the sequence, the first information indicating whether second information is present in the bitstream ;
Based on a scaling factor of spatial resolution in response to determining that the second information is present in the bitstream and thereafter determining the first information as indicated by the first information. Second information at a frame level in the bitstream for each of the plurality of frames of the sequence, wherein one or more spatial resolutions for the frame are encoded. Outputting second information indicating a scaling factor of
Outputting multi-spatial resolution encoded data for the plurality of frames using the scaling factor;
The plurality of frames include at least one I frame and at least one P frame, and each of the at least one P frame has a reference I frame among the at least one I frame, and the at least one P frame. For each of the at least one P frame, the scaling factor of the one or more spatial resolutions for the P frame is the 1 or the reference I frame for the reference I frame of the P frame. A method characterized in that it is constrained to be the same as the scaling factor of multiple spatial resolutions.   The method of claim 19, wherein the second information is a fixed length code.   21. The method of claim 20, wherein the fixed length code is a 2-bit code that represents four possible states of the scaling factor of the one or more spatial resolutions.   The method of claim 19, wherein the second information is a variable length code.   The method of claim 19, wherein the first information is signaled in a sequence header.   24. The method of claim 23, wherein the first information is a 1-bit code in the sequence header.   The method of claim 19, further comprising outputting third information in the bitstream that is indicative of a selected resampling filter.   The method of claim 19, wherein the one or more spatial resolution scaling factors comprise a vertical spatial resolution scaling factor and a horizontal spatial resolution scaling factor.   27. The method of claim 26, wherein the vertical spatial resolution scaling factor is different from the horizontal spatial resolution scaling factor.   27. The method of claim 26, wherein the vertical spatial resolution scaling factor is selected from a set of vertical spatial resolutions including a full resolution and a half resolution.   27. The method of claim 26, wherein the horizontal spatial resolution scaling factor is selected from a set of horizontal spatial resolutions including a full resolution and a half resolution.   The method of claim 19, wherein the scaling factor of the one or more spatial resolutions is adaptively determined based at least in part on a bit rate criterion.   The method of claim 19, wherein the scaling factor of the one or more spatial resolutions is adaptively determined based at least in part on a high frequency content criterion.   The method of claim 19, wherein the scaling factor of the one or more spatial resolutions is adaptively determined based at least in part on a quantization step size criterion.   The method of claim 19, further comprising encoding the plurality of frames with multi-spatial resolution encoding according to a scaling factor of the spatial resolution indicated by the second information.   34. The method of claim 33, further comprising down-sampling the plurality of current frames, wherein the down-sampling further results in a reduced resolution frame.   The method of claim 34, wherein the down-sampling comprises applying a 6-tap filter to the current frame.   35. The method of claim 34, wherein the down-sampling comprises down-sampling in a horizontal direction prior to down-sampling in a vertical direction.   The current frame of the plurality of frames includes a plurality of lines, and the multi-space resolution encoding for the current frame is performed so that the number of samples in each of the plurality of lines is a multiple of a macroblock. 20. The method of claim 19, comprising adjusting to: Means for receiving first information in a bitstream of a sequence of video frames, the first information indicating whether second information is present in the bitstream ;
If the first information indicates that the second information is present in the bitstream, second information at a frame level in the bitstream for each of a plurality of frames of the sequence; Means for receiving second information indicative of a scaling factor of one or more spatial resolutions and multi-spatial resolution decoding the plurality of frames using the scaling factor, the plurality of frames comprising: At least one I frame and at least one P frame, each of the at least one P frame having a reference I frame of the at least one I frame, and for each of the at least one P frame, The scaling factor for one or more spatial resolutions is System characterized in that it comprises a means which is constrained to be identical to the scaling factor of the one or more spatial Rizorushon for a reference I-frame of the P-frame. Means for outputting first information in a bitstream of a sequence of video frames, the first information indicating whether second information is present in the bitstream ;
If the first information indicates that the second information is present in the bitstream, for each of a plurality of frames of the sequence, second information at a frame level in the bitstream; Outputting second information indicating a scaling factor of one or more spatial resolutions for the frame and outputting multi-spatial resolution encoded data for the plurality of frames using the scaling factor The plurality of frames comprises at least one I frame and at least one P frame, each of the at least one P frame having a reference I frame of the at least one I frame, For each P frame, the one or more empty frames Scaling factor Rizorushon the system characterized in that it comprises a means which is constrained to be identical to the scaling factor of the one or more spatial Rizorushon for a reference I-frame of the P-frame.

Images