HK1226568B - Decoding method for multi-layered frame-compatible video delivery - Google Patents

Description

Decoding method for multi-layer frame-compatible video transmission

This invention application is a divisional application of invention patent application No. 201180035724.4, whose application date is July 20, 2011 and which entered the Chinese national phase on January 21, 2013 and whose invention name is “System and method for multi-layer frame-compatible video transmission”.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 61/366,512, filed July 21, 2010, which is hereby incorporated by reference in its entirety.

Technical Field

The present disclosure relates to image processing and video compression. More particularly, embodiments of the present invention relate to encoding and decoding systems and methods for multi-layer frame-compatible video transmission.

Background Art

Recently, there has been considerable interest and traction in the industry for stereoscopic (3D) video transmission. High-grossing films have made 3D stereoscopic video mainstream, and major sporting events are also being produced and broadcast in 3D. Specifically, animated films are increasingly being produced and presented in stereoscopic format.

However, while there is a large enough installed base of 3D-capable cinema screens, the same is not true for consumer 3D applications. Efforts in this area are still in their early stages, but several industry parties are investing considerable effort in the development and marketing of consumer 3D-capable displays [Reference 1].

Stereoscopic display technology and stereoscopic content creation are issues that must be properly addressed to ensure a sufficiently high-quality experience. The delivery of 3D content is equally critical. Content delivery involves several components, including compression. Stereoscopic delivery is challenging because stereoscopic delivery systems process twice as much information as 2D systems. Furthermore, computational and storage throughput requirements increase significantly.

Typically, there are two main distribution channels through which stereoscopic content can be delivered to consumers: fixed media such as Blu-ray Discs, and streaming solutions where the content is first delivered to a set-top box and then to a PC.

Most currently deployed Blu-ray players and set-top boxes only support codecs such as those based on the specifications of the ITU-T/ISO/IEC H.264/14496-10 [Reference 2] prior art video coding standard (also known as MPEG-4 Part 10 AVC) and Annex A of the SMPTE VC-1 standard [Reference 3].

Each of these codec solutions enables service providers to transmit a single HD (High Definition) image sequence at a resolution of 1920×1080 pixels. However, transmitting stereoscopic content involves sending information for both the left and right sequences. The straightforward approach is to encode two separate bitstreams, one for each viewpoint, an approach also known as simulcast.

First, simulcast or similar methods have low compression efficiency. They also use a high bandwidth to maintain an acceptable quality level. This is because the left view sequence and the right view sequence are encoded independently, even though they are related.

Secondly, the two separate bit streams are demultiplexed and decoded in parallel in two appropriately synchronized decoders. To implement such a decoder, two existing non-specially designed decoders can be used. In addition, parallel decoding is suitable for graphics processing unit architecture.

A codec supporting multiple layers can provide high compression efficiency for stereoscopic video while maintaining backward compatibility.

A multi-layer or scalable bitstream consists of multiple layers characterized by predefined dependencies. One or more of these layers is a so-called base layer that is decoded before any other layer and is independently decodable.

The other layers are often referred to as enhancement layers, as their function is to improve the content obtained by parsing and decoding one or more base layers. These enhancement layers are also dependent layers, as they rely on the base layer. The enhancement layers use some form of inter-layer prediction, and one or more of the enhancement layers may also typically rely on the decoding of other, higher-priority enhancement layers. Consequently, decoding may also terminate at one of the intermediate layers.

Multi-layer or scalable bitstreams enable scalability in terms of quality/signal-to-noise ratio (SNR), spatial resolution and/or temporal resolution and/or even the availability of additional viewpoints. For example, temporally scalable bitstreams can be generated using codecs based on the Annex A specifications of H.264/MPEG-4 Part 10, VC-1, or VP8.

The first base layer, if decoded, may provide a version of the image sequence at 15 frames per second (fps), while the second enhancement layer, if decoded, may provide the same image sequence at 30 fps in conjunction with the already decoded base layer.

SNR and spatial scalability are also possible. For example, when the Scalable Video Coding (SVC) extension of the H.264/MPEG-4 Part 10 AVC video coding standard (Annex G) is adopted, the base layer (encoded according to Appendix A) generates a lower quality version of the image sequence. One or more enhancement layers can provide additional increases in visual quality. Similarly, the base layer can provide a lower resolution version of the image sequence. The resolution can be increased by decoding additional enhancement layers, spatially and/or temporally. Scalable or multi-layered bitstreams are also useful for providing multi-viewpoint scalability.

Recently, the stereoscopic high-resolution specification of the Multiview Coding (MVC) extension (Annex H) of H.264/AVC has been completed and has been adopted as the video codec for the next generation Blu-ray Disc (Blu-ray 3D) featuring stereoscopic content. This encoding method attempts to address the high bit rate requirements for stereoscopic video streams to some extent.

Stereo High Profile utilizes a base layer that compresses one of the views (typically the left view) that conforms to the High Profile of Annex A of H.264/AVC and is called the base view. The enhancement layer then compresses the other view, called the view-dependent view (dependentview). While the base layer is a valid H.264/AVC bitstream in itself and is decodable independently of the enhancement layer, this may not be the case, and is not usually the case, for the enhancement layer. This is because the enhancement layer can utilize decoded pictures from the base layer as motion-compensated prediction references. Therefore, the view-dependent view (enhancement layer) can benefit from inter-view prediction, and compression can be significantly improved for scenes with high inter-view correlation (i.e., low stereo disparity). Therefore, the MVC extension method attempts to address the problem of increased bandwidth by utilizing stereo disparity.

However, this approach may not provide compatibility with existing set-top boxes and Blu-ray player infrastructure. Although existing H.264 decoders can decode and display the base viewpoint, they simply discard and ignore the relevant (right) viewpoint. Therefore, existing decoders do not have the ability to decode and display 3D content encoded using MVC. Therefore, although MVC retains 2D compatibility, MVC cannot transmit 3D content in traditional devices. The lack of backward compatibility is another obstacle to the rapid adoption of consumer 3D stereoscopic video.

According to one aspect of the present application, a decoding method for multi-layer frame-compatible video transmission is provided, comprising: a) performing base layer processing on multiple base layer bit stream signals through a base layer, comprising: i) providing at least one frame-compatible base layer decoded image or video frame; and b) performing enhancement layer processing on multiple enhancement bit stream signals through one or more enhancement layers, comprising: ii) providing at least one enhancement layer decoded image or video frame for multiple viewpoints; iii) performing reference processing on at least one frame-compatible base layer decoded image or video frame from the base layer or at least one decoded image or video frame from different enhancement layers; and iv) performing disparity compensation, wherein all multiple viewpoints are decoded and processed in the same enhancement layer.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more embodiments of the present disclosure and, together with the description of example embodiments, serve to explain the principles and implementations of the present disclosure.

Figure 1 depicts a checkerboard interleaving arrangement for the transmission of stereoscopic material.

FIG2 depicts a horizontal sample/column interleaving arrangement for transmission of stereoscopic material.

FIG3 depicts a vertical sampling/row interleaving arrangement for transmission of stereoscopic material.

FIG4 depicts a horizontal sampling/side-by-side arrangement for transmission of stereoscopic material.

FIG5 depicts a vertical sampling/over-under arrangement for transmission of stereoscopic material.

FIG6 depicts a quincunx sampling/side-by-side arrangement for transmission of stereoscopic material.

FIG7 depicts a frame-compatible full-resolution 3D stereoscopic scalable video coding system with reference processing for inter-layer prediction.

FIG8 depicts a frame-compatible full-resolution 3D stereoscopic scalable video decoding system with reference processing for inter-layer prediction.

FIG9 depicts a scalable video coding system with a reference processing unit for inter-layer prediction.

Figure 10 depicts the reconstruction module for a frame-compatible full-resolution two-layer transmission system.

11 illustrates a multi-layer resolution scalable 3D stereoscopic video encoder according to an embodiment of the present disclosure, where the enhancement layer maintains each of two reference picture buffers at an enhanced resolution and performs motion/disparity compensation at some reduced resolution (frame compatible).

12 depicts a multi-layer resolution scalable 3D stereoscopic video decoder according to an embodiment of the present disclosure, where the enhancement layer maintains each of two reference picture buffers at an enhanced resolution and performs motion/disparity compensation at some reduced resolution (frame compatible).

13 illustrates a multi-layer resolution scalable 3D stereoscopic video encoder according to an embodiment of the present disclosure, wherein the enhancement layer maintains each of two reference picture buffers at an enhanced resolution and performs motion/disparity compensation at the enhanced resolution.

14 illustrates a multi-layer resolution scalable 3D stereoscopic video decoder according to an embodiment of the present disclosure, wherein the enhancement layer maintains each of two reference picture buffers at an enhanced resolution and performs motion/disparity compensation at the enhanced resolution.

Figure 15 depicts a multi-layer resolution scalable 3D stereoscopic video encoder according to an embodiment of the present disclosure, where a base layer encodes a frame-compatible version of the data and two enhancement layers encode each of the enhanced resolution data categories (each viewpoint of the 3D stereoscopic video transmission).

Figure 16 depicts a multi-layer resolution scalable 3D stereoscopic video decoder according to an embodiment of the present disclosure, where a base layer encodes a frame-compatible version of the data and two enhancement layers encode each of the enhanced resolution data categories (each viewpoint of the 3D stereoscopic video transmission).

Figure 17 depicts a multi-layer resolution scalable 3D stereoscopic video encoder according to an embodiment of the present disclosure, where the enhancement layer encodes the residual and maintains each of the two reference picture buffers at an enhanced resolution, and performs motion/disparity compensation at some reduced resolution (frame compatible).

Figure 18 depicts a multi-layer resolution scalable 3D stereoscopic video decoder according to an embodiment of the present disclosure, where the enhancement layer encodes the residual and maintains each of the two reference picture buffers at an enhanced resolution, and performs motion/disparity compensation at some reduced resolution (frame compatible).

Figure 19 depicts a multi-layer resolution scalable video encoder according to an embodiment of the present disclosure, where a base layer encodes a frame-compatible version of the data and two enhancement layers encode the residual for each of the enhanced resolution data categories (each viewpoint for 3D stereoscopic video transmission).

Figure 20 depicts a multi-layer resolution scalable video decoder according to an embodiment of the present disclosure, where a base layer encodes a frame-compatible version of the data and two enhancement layers encode the residual for each of the enhanced resolution data categories (each viewpoint for 3D stereoscopic video transmission).

DETAILED DESCRIPTION

According to a first aspect of the present disclosure, a coding method for multi-layer frame-compatible video transmission is provided, the coding method comprising: a) performing base layer processing of images or video frames of multiple data categories through a base layer, comprising: i) providing a base layer frame-compatible representation of the images or video frames of the multiple data categories; and b) performing enhancement layer processing of images or video frames of multiple data categories through one or more enhancement layers, comprising: i) providing an enhancement layer frame-compatible representation of the images or video frames of the multiple data categories, ii) maintaining at least one enhancement layer reference picture buffer, iii) performing reference processing for at least one dependency on the base layer or a different enhancement layer; and iv) performing motion or disparity compensation, wherein each of the one or more enhancement layers processes all of the multiple data categories.

According to a second aspect of the present disclosure, there is provided a coding method for multi-layer frame-compatible video transmission, comprising: a) performing base layer processing of images or video frames of multiple data categories through a base layer, comprising: i) providing a base layer frame-compatible representation of the images or video frames of the multiple data categories; and b) performing enhancement layer processing of images or video frames of the multiple data categories through one or more enhancement layers, wherein each of the multiple data categories is processed separately in a separate enhancement layer, and each of the one or more enhancement layers comprises: i) providing an enhancement layer representation of an image or video for one of the multiple data categories, ii) maintaining an enhancement layer reference picture buffer in each enhancement layer, iii) performing reference processing for at least one dependency on the base layer or a different enhancement layer, and iv) performing motion or disparity compensation.

According to a third aspect of the present disclosure, a decoding method for multi-layer frame-compatible video transmission is provided, the decoding method comprising: a) performing base layer processing on multiple base layer bit stream signals through a base layer, comprising: i) providing at least one frame-compatible base layer decoded image or video frame; and b) performing enhancement layer processing on multiple enhancement bit stream signals through one or more enhancement layers, comprising: i) providing at least one enhancement layer decoded image or video frame for multiple data categories, ii) maintaining at least one enhancement layer reference picture buffer, iii) performing reference processing for at least one dependency on the base layer or different enhancement layers, and iv) performing disparity compensation, wherein all of the multiple data categories are decoded and processed in the same enhancement layer.

According to a fourth aspect of the present disclosure, a decoding method for multi-layer frame-compatible video transmission is provided, the decoding method comprising: a) performing base layer processing on multiple base layer bit stream signals passing through the base layer through the base layer, comprising: i) providing at least one frame-compatible base layer decoded image or video frame; and b) performing enhancement layer processing on multiple enhancement bit stream signals passing through one or more enhancement layers for multiple data categories through one or more enhancement layers, wherein each of the multiple data categories is processed separately in a separate enhancement layer, and each of the one or more enhancement layers comprises: i) providing at least one enhancement layer decoded image or video frame for one of the multiple data categories, ii) maintaining at least one enhancement layer reference picture buffer, iii) performing reference processing for at least one dependency on the base layer or a different enhancement layer, and iv) performing disparity compensation, wherein all of the multiple data categories are decoded and processed in the same enhancement layer.

Given the lack of backward compatibility with existing codecs, leveraging the installed base of set-top boxes, Blu-ray players, and HDTVs can accelerate consumer 3D deployment. Most display manufacturers offer HDTVs that support 3D stereoscopic display. These TVs include models from all of the professional display technologies: LCD, plasma, and DLP [Reference 1]. The key is to provide a display with defined content that contains two viewpoints but still fits into a single frame, while also leveraging existing and deployed codecs such as VC-1 and H.264/AVC. Such an approach is the so-called frame-compatible approach, which formats the stereoscopic content so that it fits into a single picture or frame. The size of the frame-compatible representation does not need to be the same as the size of the original viewpoint frame.

Similar to the MVC extension of H.264, Dolby's stereoscopic 3D consumer delivery system [Reference 4] features a base layer and an enhancement layer. In contrast to the MVC approach, views can be multiplexed into two layers to provide consumers with: a base layer that is frame-compatible by carrying subsampled versions of the two views; and an enhancement layer that, when combined with the base layer, results in full-resolution reconstruction of both views.

A backward compatible 3D video transmission system can provide 3D video to homes or other places through existing or traditional 2D video hardware and systems. A frame compatible 3D video system provides such a backward compatible transmission architecture. In this case, a layered approach can be used, in which the base layer provides low-resolution versions of the left eye and right eye arranged in a "frame compatible" format. Frame compatible formats include side-by-side, top-and-bottom, and plum blossom/checkerboard interleaving. Figures 1 to 6 show some schematic examples. In addition, there can be an additional pre-processing stage that predicts the enhancement layer frames taking into account the base layer decoded frames before using them as motion compensation references for predicting the enhancement layer. Figures 7 and 8 show an encoder and decoder, respectively, for the system disclosed in [Reference 4].

Even non-frame-compatible coding arrangements such as MVC can be enhanced using a pre-processor (e.g., a reference processing unit (RPU)/predictor) that improves the reference taken from the base view before using it as a reference for view prediction of the dependent view. This architecture is also disclosed in [Reference 4] and is shown in Figure 9.

The frame-compatible techniques of [Reference 4] ensure a frame-compatible base layer. By using pre-processor/RPU components, these techniques reduce the overhead of achieving full-resolution reconstruction of stereo viewpoints. Figure 10 illustrates the full-resolution reconstruction process.

Based on the availability of enhancement layers, there are several methods for obtaining the final reconstructed viewpoint. Some of these methods may consider encoding the actual pixel data in the enhancement layer, or may consider encoding residual data or data that is generally different from the base layer (e.g., high frequency vs. low frequency) data, which, if combined in a certain form, can enable a higher quality/resolution representation of the reconstructed signal. Any resolution can be used for these methods, for example: some of them may be at half resolution, while some of them may be at full resolution or even lower, higher, or somewhere in between. The embodiments of the present disclosure can be directed to any resolution. They can be interpolated from the frame-compatible output of the base layer (V _FC,BL (1002) of Figure 10) and optionally post-processed to produce V _0,BL,out (1004) and V _1,BL,out (1006). Alternatively, they can be multiplexed with appropriate samples of the enhancement layer to produce a higher representation of each viewpoint, reconstructing V _0,FR,out (1008) and V _1,FR,out (1010). The resulting reconstructed viewpoint can have the same resolution in both cases. However, in the second case, information is encoded for all samples, while in the first case, half of the information for the reconstructed viewpoint is obtained by interpolation using an intelligent algorithm, as disclosed in [Reference 4]. As can be observed in Figure 10, following the decoding of the base layer and enhancement layers, additional and potentially memory-intensive and bandwidth-intensive operations are used to obtain the final full-resolution reconstructed viewpoint.

The present disclosure provides techniques that enable frame-compatible 3D video systems to achieve full-resolution 3D transmission. The present disclosure also provides methods for improving the accuracy of internal predictions in enhancement layers by performing motion and stereo disparity compensation in some higher representation/resolution sample domains. These domains can have a higher spatial or frequency resolution than the samples represented in a frame-compatible representation. In some embodiments, these domains can have a resolution equal to the full resolution, that is, the original resolution of the frames of each category before they are filtered, sampled, and multiplexed into a frame-compatible representation. Additional methods for processing data compressed by these arrangements can be found in [Reference 5]. Throughout this specification, the term "data category" or "category" refers to a data group. Different data categories can refer to different data groups that may or may not have a relationship between the groups. For embodiments of the present disclosure related to 3D or stereoscopic image or video transmission, the term "data category" or "category" refers to a single viewpoint of a 3D image or video.

FIG11 illustrates a multi-layer resolution scalable 3D stereoscopic video encoder according to an embodiment of the present disclosure, wherein the enhancement layer maintains each of the two reference picture buffers at an enhanced resolution and performs motion/disparity compensation at a reduced resolution (e.g., half the horizontal or vertical resolution). FIG12 illustrates a decoder corresponding to the encoder shown in FIG11 according to an embodiment of the present disclosure. According to this embodiment, a multi-layer codec is provided for compressing a video sequence consisting of frames belonging to multiple data categories at a given time instance.

According to an embodiment of the present disclosure, the base layer (1102) of Figure 11 provides a frame-compatible representation of multiple data categories. Here, frame-compatible representation refers to sampling and multiplexing different data categories into a single frame. This single frame can have a different size than the frame comprising the original category. According to another embodiment of the present disclosure, the base layer (1102) of Figure 11 can be implemented and encoded using any available or future video codec (such as H.264/AVC, VP8, and VC-1).

Continuing with FIG11 , the data is sampled by a sampler (1106) and multiplexed by a multiplexer (1108) before being sent to a base layer encoder (1104). In another embodiment, sampling may also include filtering. Furthermore, filtering may be asymmetric between different data categories. For example, in yet another embodiment, one category may be filtered and sampled such that less than half of the information (e.g., frequency content) is retained. Another category may be filtered and sampled such that more than half of the information is retained. FIG1 through FIG6 illustrate schematic sampling and multiplexing arrangements for two categories of image data.

According to the embodiment shown in FIG11 , additional enhancement layers (1152) are provided. According to other embodiments, the number of additional enhancement layers depends on the number of categories of data and frame data that have been sampled and interleaved within the base layer. The data that is sampled and interleaved in the enhancement layers is selected so that when combined with the data already present in the base layer, the sampled and interleaved data results in an efficient representation and reconstruction of most categories of data. According to the embodiment shown in FIG11 , in which two categories of data are included, one enhancement layer (1152) is used to encode all of the original data. According to this embodiment, the base layer (1102) can carry half of the samples for each category, and the enhancement layer (1152) can provide the other half of the missing samples for each data category.

According to another embodiment of the present disclosure, the base layer compresses one-third of the samples of one category, and the remaining two-thirds of the samples are stored in the enhancement layer. The opposite situation is also possible. Similarly, as with the base layer, the data content of each category in the enhancement layer can be different from the content of another data category. This can be achieved by using different types of filtering or different numbers and arrangements of samples (for example, quincunx versus row-based subsampling). According to this embodiment, the sampling operation obtains samples for enhancement layer processing, and the sampling operation may include filtering of these samples.

According to the embodiment shown in FIG11 , the enhancement layer ( 1152 ) employs a hybrid video coding model, which can be found in modern video codecs such as VC-1 and H.264/AVC. The input data can be predicted from adjacent samples in the same picture or frame (using intra-frame prediction) or from samples from previously decoded frames belonging to the same layer and buffered in a so-called reference picture buffer as a reference for motion-compensated prediction (inter-frame prediction). Inter-layer prediction is also possible if decoded information from a lower-priority layer (such as the base layer) is available for the enhancement layer. One way to access this information is by using decoded pictures from the lower-priority layer as a reference for motion compensation. After prediction, the prediction residual ( 1154 ) is transformed ( 1156 ) and quantized ( 1158 ), and the quantized coefficients are then encoded using entropy coding ( 1160 ). The enhancement layer ( 1252 ) of the decoder shown in FIG12 reverses this process.

Unlike the base layer (1102) of FIG11 , which has a single reference picture buffer (1110) containing previously decoded pictures/frames, the enhancement layer (1152) maintains multiple internal reference picture buffers (1162), one for each data category. In the embodiment of FIG11 , the generation of the reference pictures stored in these buffers is achieved using a demultiplexer and RPU processor (1164). The demultiplexer and RPU processor (1164) processes the sum of the prediction residual and the predicted frame (which is obtained by intra-frame or inter-frame prediction).

The demultiplexer (1164) of Figure 11 also performs upsampling and interpolation of lost samples for each category. Each reference picture buffer (1162) includes only frames belonging to the same data category. The buffer (1162) stores images or frames at a higher resolution than the resolution of the samples input to the enhancement layer (1152), or optionally stores images or frames at an enhanced resolution. In addition, the resolution used to store frames in each reference picture buffer can be different from each other. One buffer can store pictures at one resolution, while a second picture buffer can store pictures at another resolution. Before performing disparity compensation (i.e., motion compensation or intra-frame prediction) (1168), the selected references from each picture buffer (1162) are downsampled using a sampler (1170) and multiplexed using a multiplexer (1172) to generate a single reference picture that can now be formatted in a frame-compatible arrangement. According to further embodiments, the operations of downsampling and multiplexing to a frame-compatible format can include more complex operations, such as linearly or nonlinearly combining two references into a final frame-compatible reference picture. According to yet another embodiment, the resolution of the frames in the internal buffer may match the enhanced resolution.

According to the embodiment shown in FIG11 , inter-frame prediction within the enhancement layer (1152) is performed after sampling (1170) and multiplexing (1172) reference pictures provided by an internal multiple (optionally enhanced resolution) reference picture buffer. Thus, inter-frame prediction is performed in a "frame-compatible" domain, but not necessarily in the same domain as the base layer. According to another embodiment of stereoscopic video, the base layer frame-compatible format may include even columns from the left view and odd columns from the right view, while at the enhancement layer, the frame-compatible format may include odd columns from the left view and even columns from the right view. Similar arrangements are possible for other interleaved arrangements (e.g., top-and-bottom, side-by-side, etc.), with the appropriate arrangement being selected such that the combination of samples encoded in the frame-compatible base layer picture and samples encoded in one or more enhancement layers should produce an enhanced resolution reconstruction of the data class. According to another embodiment of the present disclosure, such techniques can be extended to any number of layers or viewpoints. Furthermore, the inter-frame prediction process includes estimating a set of motion parameters for each enhancement layer, which are encoded and transmitted to the decoder.

According to the embodiment shown in Figure 11, the enhancement layer reference picture buffer (1162) includes pictures that are not limited to decoded pictures of the enhancement layer (1152) that have been demultiplexed and upsampled (1164). The base layer to enhancement layer reference processing unit (RPU) / pre-processor module (BL to EL RPU) (1166) takes as input the frame-compatible decoded pictures from the reference picture buffer (1110) of the base layer (1102) and then multiplexes and upsamples the frame data to estimate a higher representation (optionally, enhanced resolution) frame belonging to a different data category.

According to another embodiment, the BL to EL RPU (1166) processing may include filtering, upscaling, insertion of lost samples, and recovery or estimation of frequency content. The recovery or estimation of frequency content is used when, for example, the base layer encodes low frequencies and the enhancement layer encodes high frequencies. These BL to EL RPU (1166) processed images are then placed and used as additional motion compensated prediction references in the reference picture buffer (1162) of the higher representation (optionally enhanced resolution) of the enhancement layer (1152). The BL to EL RPU module (1166) at the encoder generates information for the prediction/upsampling process and transmits this information ("RPU bitstream") (1174) to the same BL to EL RPU (1254) of Figure 12, which is located at the decoder module shown in Figure 12. In this way, the encoder prediction operations can be replicated at the decoder. The interpolation and prediction using this RPU module may include the techniques disclosed in [Reference 6].

According to another embodiment of the present disclosure, an internal buffer stores frames at an enhanced resolution, and the decoder internally reconstructs the enhanced resolution frames for each data category and stores them in a reference picture buffer. Therefore, in order to display the enhanced resolution data, the processing module of Figure 10 is not used in this embodiment. Instead, according to this embodiment, the enhanced resolution reconstructed frames can be directly extracted and displayed from the reference picture buffer of the enhancement layer. According to another embodiment of the present disclosure, the enhanced resolution of each category is unequal. In this embodiment, the encoder and decoder rescale the pictures in the buffer of the enhancement layer to a certain same enhanced resolution.

In a further embodiment, the size of the frames encoded in the base layer may be the same as the size of the frames belonging to each category.In a further embodiment, the reference picture buffer in the enhancement layer contains frames at the resolution of the original frame (full resolution).

According to an embodiment of the present disclosure, the frame-compatible domain to which motion compensation is applied may be the same as the domain of the base layer. In another embodiment, the base layer may be interleaved in a side-by-side format, and the enhancement layer may also encode frames interleaved in the same side-by-side format.

According to an embodiment of the present disclosure, the base layer frame-compatible format may include even columns from the first viewpoint and odd columns from the second viewpoint, while in the enhancement layer, the frame-compatible format may include odd columns from the first viewpoint and even columns from the second viewpoint.

According to an embodiment of the present disclosure, layers can encode frames at different spatial resolutions, in which case a system with spatial scalability can be established. According to another embodiment, the codec system has a 1280×720 side-by-side frame-compatible base layer and an enhancement layer that can reconstruct both viewpoints at 1920×1080. In this embodiment, the BL to EL RPU first demultiplexes the frame-compatible data into different categories and then can perform one of the following operations. In one scheme, the BL to EL RPU first interpolates the lost samples of the category and then rescales the resulting frame (1280×720) to the desired spatial resolution (1920×1080) before storing it in the corresponding reference picture buffer in the enhancement layer. In the second scheme, the available and demultiplexed samples are rescaled from a lower resolution to a higher resolution, for example, from 640×720 to 960×1080. Then, additional interpolation operations determine the lost columns and optionally also filter the existing samples to derive the full-resolution frame.

According to an embodiment of the present disclosure, multiple reference picture buffers in an enhancement layer can also be controlled via memory management control operations (MMCOs), such as those disclosed in [Reference 2]. MMCO operations control how reference pictures are added to and removed from the buffers. According to another embodiment, MMCO operations occur for the enhancement layer. In this embodiment, the set of MMCOs for each reference picture buffer is the same, or the set of MMCO operations is signaled. This applies to both reference picture buffers. Thus, picture buffer operations remain synchronized. Yet another embodiment may utilize similar methods for reference picture list modification/reordering signaling, including but not limited to the method disclosed in [Reference 2].

Signaling information controls the generation of reference picture lists in the buffer. These lists are then used in motion-compensated prediction. According to another embodiment, the correction information is the same for the reference picture lists of each data class. In yet another embodiment, a single set of correction information is generated and applied to all lists in the enhancement layer. According to another embodiment of the present disclosure, a similar approach is used in a codec that utilizes signaling to control the contents of the reference picture buffer and the initialization and modification of its reference picture lists.

According to an embodiment of the present disclosure, such a demultiplexer and upsampler RPU may be implemented as a reference processing unit as disclosed in [Reference 6], which stores frames into a reference picture buffer after obtaining a single category frame of enhanced resolution.

According to further embodiments of the present disclosure, a base layer may encode a representation using a first range of frequency content, while an additional enhancement layer may provide a second range of frequency content. Their outputs may also be combined at the decoder to provide a better representation of the original data class.

FIG13 shows a multi-layer resolution scalable 3D stereoscopic video encoder according to an embodiment of the present disclosure, wherein the enhancement layer maintains each of the two reference picture buffers at an enhanced resolution and performs motion/disparity compensation at the enhanced resolution. FIG14 shows a decoder corresponding to the encoder shown in FIG13 according to an embodiment of the present disclosure.

The embodiments shown in Figures 13 and 14 are similar to those shown in Figures 11 and 12. However, unlike the embodiments shown in Figures 11 and 12, according to the embodiment shown in Figure 13, the sampler (1354) and multiplexer (1356) are placed after the disparity compensation module (1358). In fact, there are as many disparity compensation modules (1358) as there are data classes. According to the embodiment of stereoscopic video transmission shown in Figure 13, there are two disparity compensation modules (1358), one for each viewpoint. After disparity compensation is performed for each higher representation reference, the resulting frame is passed to the sampler (1354) which performs a downsampling process, which can be performed before or after filtering (in the case of two views, it will retain half the samples). The downsampled data from each data class is then fed to the multiplexer (1356) which generates a frame-compatible picture (1360). This frame-compatible picture (1360) is then used as a prediction for the prediction residual (1362) added to the enhancement layer hybrid video coding loop.

According to the embodiment shown in FIG13 , the disparity compensation module ( 1358 ) makes more samples available (higher representation pictures) and is able to produce better predictions. In addition, the disparity compensation module can have more spatially accurate partitions in motion compensation. For example, when using the side-by-side format, a 4×4 partition size in a frame-compatible reference picture is equivalent to an 8×4 partition size in a full-resolution picture. Similarly, a 16×16 partition is actually a 32×16 partition in a full-resolution picture. Therefore, in this embodiment, the disparity compensation module ( 1358 ) can have larger and more accurate partitions.

According to the embodiment shown in FIG13 , disparity estimation and compensation may be performed multiple times (e.g., twice) at the enhancement layer, thereby increasing system complexity. Furthermore, the benefit gained from increased spatial accuracy of motion compensation depends on how a higher representation reference picture is upsampled from a frame-compatible picture obtained by adding the prediction residual to the frame-compatible predicted picture V _FC,PRED (1362) of FIG13 . Furthermore, according to this embodiment, the enhancement layer compresses the amount of motion vector information twice. According to another embodiment, the reference picture buffer uses an enhanced resolution, and a final reconstruction of each data class is performed as part of generating the reference stored in the reference picture buffer of the enhancement layer. Therefore, unlike the decoder shown in the figure, this embodiment does not further process the outputs of the base layer and enhancement layer.

According to further embodiments of the present disclosure, a multi-layer codec may take into account spatial scalability, similar to the additional embodiments of the first approach. Similar to the embodiments shown in Figures 11 and 12, yet another embodiment provides reference picture list modification and MMCO operation signaled to the decoder.

According to another embodiment of the present disclosure, since there is sufficient correlation among the motion parameters used in multiple disparity/motion estimators and compensation modules, these motion parameters are selected so that the parameters of one module can be efficiently predicted based on the parameters from other modules. In another embodiment, the motion parameters are selected to be the same, and only one set of parameters is sent for each enhancement layer. In another embodiment, the set of parameters for each module is signaled. Motion parameter prediction can also use information from adjacent or side-by-side parameters signaled from higher priority disparity estimation/compensation modules.

According to another embodiment of the present disclosure, the size of the frames encoded in the base layer may be the same as the size of the frames belonging to each category. According to another embodiment, the reference picture buffer in the enhancement layer contains frames at the resolution of the original frame (full resolution). According to yet another embodiment, the demultiplexer and upsampler of the single category frames that obtain the enhanced resolution before being stored in the reference picture buffer may be implemented as a reference processing unit as disclosed in [Reference 6]. In yet another embodiment, the base layer may encode the representation with a first range of frequency content, while the additional enhancement layer may provide a second range of frequency content. Their outputs may also be combined at the decoder to provide a better representation of the original data category.

FIG15 shows a multi-layer resolution scalable video encoder according to an embodiment of the present disclosure, wherein a base layer (1502) encodes a frame-compatible version of the data, and two of the multiple enhancement layers (1532, 1562) encode one of two enhanced resolution data categories (one for each viewpoint in the case of 3D stereoscopic video transmission). FIG16 shows a corresponding decoder according to an embodiment of the present disclosure.

According to the embodiment shown in FIG15 , the structure of the base layer (1502) is identical to the structure of the embodiment shown in FIG11 to FIG14 . According to the embodiment shown in FIG15 , the base layer (1502) can encode frame-compatible versions of multiple data categories. In this embodiment, an enhancement layer (1532, 1562) is provided for each data category. According to another embodiment for stereoscopic video transmission, each enhancement layer (1532, 1562) provides enhanced resolution reconstruction for each data category. According to this embodiment, each enhancement layer (1532, 1562) contains a single reference picture buffer (1534, 1564) and uses a very similar structure to the base layer (1502). In this embodiment, the enhancement layers (1532, 1562) directly receive enhanced (e.g., full) resolution frames for each category. In contrast, according to the embodiments shown in FIG11 to FIG14 , the input to the enhancement layers consists of frame-compatible representations of all data categories.

According to the embodiment shown in FIG15 , the reference picture buffer (1534, 1564) of each layer (1532, 1562) stores references that can be used for motion compensated prediction (1536, 1566). These references include previously decoded frames of the same layer. According to another embodiment, additional references in the enhancement layer (1532, 1562) can be inserted from the base layer (1502), as is done with the MVC extension of H.264. In this embodiment, these references are processed using an RPU/preprocessor (1538, 1568) before being inserted to obtain processed references corresponding to frames stored in the target reference picture buffer. According to another embodiment for stereoscopic video transmission, the base layer frame-compatible pictures are demultiplexed into samples belonging to different classes. The samples are then upsampled to the enhanced (e.g., full) resolution within the RPU/preprocessor (1538, 1568) before being stored in the reference picture buffer (1534, 1564) of each enhancement layer (1532, 1562). According to this embodiment, the prediction, interpolation, and upsampling processes within the RPU/preprocessor (1538, 1568) may employ the techniques disclosed in [Reference 6].

According to the embodiment shown in Figure 15, a separate RPU (1538, 1568) can be implemented to generate each reference to be stored in each of the reference picture buffers (1534, 1564) of the enhancement layer. According to another embodiment, a single module can be provided to jointly optimize and perform demultiplexing and upsampling of the base layer decoded frame-compatible picture into multiple full reference pictures, one full reference picture for each enhancement layer.

According to another embodiment of the present disclosure, additional dependencies of the enhancement layer in addition to the dependency on the base layer are provided. In this embodiment, the enhancement layer can rely on the parsing and decoding of another enhancement layer. Continuing with reference to Figure 15, in addition to the base layer (1502), the decoding process of enhancement layer 1 (1562) can also rely on enhancement layer 0 (1532). The picture (1572) for display stored in the reference picture buffer (1534) of enhancement layer 0 (1532) is fed into the additional RPU/preprocessor module (1570). The additional RPU/preprocessor (1570) processes the fed reference input (1572) into a format similar to that of enhancement layer 1 (1562). The processed result (1574) is then stored in the reference picture buffer (1564) of enhancement layer 1 (1562) and can be used for motion compensated prediction (1566). According to another embodiment for stereoscopic video transmission, each enhancement layer encodes one of the views, and the RPU processes one view using motion and spatial processing to produce a reference picture that is closer to the other view. According to yet another embodiment, the motion processing can include high-order motion models, such as affine and perspective motion models.

According to another embodiment, similar to the embodiments shown in Figures 11 to 14, a multi-layer codec can take into account spatial scalability. In this embodiment, the pre-processor module that performs prediction of the enhancement layer based on the base layer (e.g., 1538 and 1568 of Figure 15) can also include rescaling to the target layer resolution. If the enhancement layers do not have the same spatial resolution, the pre-processor module that predicts one enhancement layer based on a second enhancement layer (e.g., 1570 of Figure 15) can also include rescaling.

FIG17 shows a multi-layer resolution scalable 3D stereo video encoder according to an embodiment of the present disclosure, wherein the enhancement layer encodes the residual and maintains each of the two reference picture buffers at an enhanced resolution, and performs motion/disparity compensation at a reduced resolution (frame compatible). FIG18 shows a corresponding decoder according to an embodiment of the present disclosure.

According to the embodiment shown in FIG17 , the base layer ( 1702 ) encodes a frame-compatible signal that can be further improved (particularly in terms of resolution or spatial frequency content) when one or more enhancement layers ( 1752 ) are decoded and combined with the output of the base layer ( 1702 ). According to this embodiment, the enhancement layer ( 1752 ) encodes a filtered, sampled, and multiplexed residual ( 1754 ) [ Reference 7 ] that is the result of subtracting a prediction ( 1756 ) of the original full-resolution data class frame. This prediction ( 1756 ) is the result of using an RPU processor ( 1758 ) that takes as input a decoded picture ( 1760 ) from the frame-compatible base layer ( 1702 ) and outputs a prediction ( 1756 ) of the original frame class at the original (full) resolution. In other embodiments, the RPU ( 1758 ) can use techniques such as those disclosed in [ Reference 6 ] , including filtering, interpolation, rescaling, etc. According to the embodiment shown in FIG. 17 , the internal picture buffer ( 1762 ) of the enhancement layer ( 1752 ) does not receive processed references from the base layer buffer ( 1704 ) through the RPU.

At the decoder shown in FIG18 , a similar RPU ( 1854 ) takes as input the decoded base layer pictures from the picture buffer ( 1804 ) of the base layer ( 1802 ), processes them to their original (full) resolution to derive full resolution frames ( 1856 ) for each category, and then adds these frames ( 1856 ) to the frames ( 1858 ) already decoded in the enhancement layer reference picture buffer ( 1860 ) to produce the final reconstructed frames ( 1862 ) for each data category.

All further embodiments according to the embodiment shown in Figures 11 and 12 that do not conflict with the differences between the embodiment shown in Figures 11 and 12 and the embodiment shown in Figures 17 and 18 also apply to the further embodiments according to the embodiment shown in Figures 17 and 18. According to further embodiments, the resolution of the base layer, the resolution of the enhancement layer, and the internal reference picture buffer of the enhancement layer may be different.

In another embodiment according to the embodiment shown in Figures 17 and 18 , similar to the embodiment shown in Figures 13 and 14 , the enhancement layer provides multiple disparity compensation modules, one for each reference picture buffer of each data category. All applicable alternative embodiments according to the embodiment shown in Figures 13 and 14 also apply.

Figure 19 shows a multi-layer resolution scalable video encoder according to an embodiment of the present disclosure, where the base layer encodes a frame-compatible version of the data and two enhancement layers encode the residual for each enhanced resolution data category (each viewpoint for 3D stereoscopic video transmission). Figure 20 shows a corresponding decoder according to an embodiment of the present disclosure.

According to the embodiment shown in FIG19 , similar to the embodiments shown in FIG17 and FIG18 , the enhancement layers (1932, 1962) employ residual coding. According to this embodiment, each enhancement layer (1932, 1962) corresponds to each data class and encodes a residual (1934, 1964) that is the result of subtracting a prediction (1936, 1966) of the original full-resolution data class frame [Reference 7]. This prediction is the result of using an RPU processor (1938, 1968) for each enhancement layer (1932, 1962), which takes as input a decoded picture from the frame-compatible base layer (1902) and outputs a prediction (1936, 1966) of the original frame class at the original (full) resolution of the given layer. According to another embodiment, the RPU (1938, 1968) may use techniques such as those disclosed in [Reference 6], including filtering, interpolation, rescaling, etc. According to the embodiment shown in FIG. 19 , the internal picture buffers ( 1940 , 1970 ) of the enhancement layer ( 1932 , 1962 ) do not receive processed references from the base layer buffer ( 1904 ) through the RPU.

At the decoder shown in Figure 20, for each enhancement layer (2032, 2062), a similar RPU (2034, 2064) takes as input the decoded base layer picture from the picture buffer (2004) of the base layer (2002), processes it to the original (full) resolution to obtain a full-resolution frame (2036, 2066) of the given category, and then adds this frame (2036, 2066) to the frames (2038, 2068) already decoded in the enhancement layer reference picture buffer (2040, 2070) to produce the final reconstructed frame (2042, 2072) of the given data category.

All further embodiments according to the embodiment shown in Figures 15 and 16 that do not conflict with the differences between the embodiment shown in Figures 15 and 16 and the embodiment shown in Figures 19 and 20 also apply to the further embodiments according to the embodiment shown in Figures 19 and 20. According to further embodiments, the resolution of the base layer and the resolution of the enhancement layer and the internal reference picture buffer of the enhancement layer can be different. Furthermore, the resolution of each enhancement layer can be different.

The method and system described in the present disclosure can be implemented with hardware, software, firmware or a combination thereof. The features described in blocks, modules or components can be implemented together (e.g., in a logic device such as an integrated logic device) or separately (e.g., a separately connected logic device). The software portion of the method disclosed herein may include a computer-readable medium comprising instructions for at least partially executing the described method when executed. The above-mentioned computer-readable medium may include random access memory (RAM) and/or read-only memory (ROM). The above-mentioned instructions may be executed by a processor (e.g., a digital signal processor (DSP), an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)).

Thus, embodiments of the invention have been described that relate to one or more of the example embodiments listed sequentially and directly below.

Thus, the present invention may be embodied in any form described herein, including but not limited to the following enumerated example embodiments (EEEs) which describe the structure, features, and functionality of certain portions of the present invention:

EEE1. A coding method for multi-layer frame-compatible video transmission, comprising:

a) performing base layer processing on images or video frames of multiple data categories through a base layer, including:

i) providing a base layer frame compatible representation of images or video frames of multiple data categories; and

b) performing enhancement layer processing on the image or video frame of the plurality of data categories through one or more enhancement layers, comprising:

i) providing enhancement layer frame-compatible representations of images or video frames of multiple data categories;

ii) maintaining at least one enhancement layer reference picture buffer;

iii) performing reference processing for at least one dependency on a base layer or a different enhancement layer; and

iv) performing motion or parallax compensation,

Each of the one or more enhancement layers processes all of the multiple data categories.

EEE2. The encoding method of enumerated example embodiment 1, wherein the plurality of data categories comprises a plurality of viewpoints for a stereoscopic image or video.

EEE3. The encoding method of Enumerated Example Embodiment 1 or 2, wherein providing a base layer frame-compatible representation of an image or video frame of multiple data categories comprises:

Multiplex image or video frame base layer samples and base layers of multiple data categories into a single frame.

EEE4. An encoding method according to any one of the enumerated example embodiments 1 to 3, wherein the base layer processing complies with one of multiple existing video codecs.

EEE5. The encoding method according to enumerated example embodiment 4, wherein the plurality of existing video codecs include H.264/AVC, VP8, and VC-1.

EEE6. An encoding method according to any one of the enumerated example embodiments 3 to 5, wherein the sampling of the base layer processing includes symmetrical or asymmetrical filtering of the images or video frames of the multiple data categories.

EEE7. The encoding method according to any one of enumerated example embodiments 3 to 6, wherein the base layer sampling includes horizontal sampling, vertical sampling or quincunx sampling.

EEE8. An encoding method according to any one of the enumerated example embodiments 3 to 7, wherein the base layer multiplexing uses any one of the frame-compatible packing arrangements of checkerboard interleaving arrangement, column interleaving arrangement, row interleaving arrangement, side-by-side arrangement and top-bottom arrangement.

EEE9. The encoding method according to any one of the enumerated example embodiments 2 to 8, wherein the base layer processing further comprises: processing equal or unequal amounts of samples of images or video frames of multiple viewpoints.

EEE10. An encoding method according to any one of enumerated example embodiments 2 to 9, wherein enhancement layer processing adopts a hybrid video coding model that conforms to any one of multiple existing codecs.

EEE11. The encoding method of enumerated example embodiment 10, wherein the plurality of existing codecs include VC-1 and H.264/AVC.

EEE12. The encoding method of any one of Enumerated Example Embodiments 2 to 11, wherein enhancement layer processing further comprises:

generating at least one reference picture or video frame by predicting at least one predicted picture from samples in the same picture or frame or from samples from a previously decoded frame in the same enhancement layer, and

At least one reference image or video frame is stored in at least one enhancement layer reference picture buffer.

EEE13. The encoding method according to enumerated example embodiment 12, wherein generating at least one reference image or video frame further comprises: demultiplexing and reference processing the sum of at least one prediction residual image and at least one prediction image.

EEE14. The encoding method according to enumerated example embodiment 13, wherein the demultiplexing further comprises: upsampling and interpolating the sum of at least one prediction residual image and at least one prediction image.

EEEl5. An encoding method according to any one of enumerated example embodiments 2 to 14, wherein one of the at least one enhancement layer reference picture buffers stores images or video frames at a different resolution than the remaining buffers in the at least one enhancement layer reference picture buffer.

EEE16. The encoding method of any one of Enumerated Example Embodiments 1 to 15, wherein enhancement layer processing further comprises:

selecting, for each of a plurality of data categories, at least one reference picture from at least one enhancement layer reference picture buffer;

multiplexing the selected at least one reference picture enhancement layer sample and the enhancement layer into at least one frame-compatible picture; and

Disparity compensation or inter-frame prediction is performed based on at least one frame-compatible image.

EEE17. The encoding method according to enumerated example embodiment 16, wherein:

Sampling the enhancement layer of the selected at least one reference picture includes any one of horizontal sampling, vertical sampling, or quincunx sampling; and

The enhancement layer of the selected at least one reference image is multiplexed using any one of a checkerboard interleaving arrangement, a column interleaving arrangement, a row interleaving arrangement, a side-by-side arrangement, and a top-and-bottom arrangement for frame-compatible packing.

EEE18. The encoding method of Enumerated Example Embodiment 16 or 17, wherein the enhancement layer processing further comprises:

performing base layer to enhancement layer (BL to EL) processing on at least one base layer decoded frame-compatible picture from a reference buffer in the base layer;

The at least one base layer decoded frame-compatible picture processed by BL to EL is stored in at least one enhancement layer reference picture buffer.

EEE19. The encoding method according to enumerated example embodiment 18, wherein BL to EL processing includes reference processing, demultiplexing and upsampling of at least one base layer decoded frame-compatible image.

EEE20. The encoding method according to enumerated example embodiment 19, wherein the BL to EL processing further comprises: filtering, upscaling, and interpolating at least one base layer decoded frame-compatible image.

EEE21. The encoding method according to any one of enumerated example embodiments 18 to 20, wherein the enhancement layer processing further comprises: providing information related to the BL to EL processing to a corresponding decoder.

EEE22. An encoding method according to any one of enumerated example embodiments 1 to 21, wherein, for each data category of a plurality of data categories, at least one enhancement layer reference picture buffer stores images at the same or different enhancement resolutions.

EEE23. An encoding method according to any one of enumerated example embodiments 1 to 22, wherein at least one enhancement layer reference picture buffer stores images at the same resolution as the original input image.

EEE24. An encoding method according to any one of enumerated example embodiments 1 to 23, wherein base layer multiplexing and enhancement layer multiplexing use the same frame-compatible packing arrangement.

EEE25. An encoding method according to any one of 18 to 24, wherein the base layer processing and the enhancement layer processing process the image at different spatial resolutions.

EEE26. The encoding method of Enumerated Example Embodiment 25, wherein the BL to EL processing comprises:

demultiplexing at least one base layer decoded frame compatible picture;

interpolating the demultiplexed at least one base layer decoded frame-compatible image; and

The interpolated and demultiplexed at least one base layer decoded frame-compatible image is rescaled to a target spatial resolution.

EEE27. The encoding method of Enumerated Example Embodiment 25, wherein the BL to EL processing comprises:

demultiplexing at least one base layer decoded frame-compatible picture;

rescaling the demultiplexed at least one base layer decoded frame-compatible picture to a different spatial resolution; and

The demultiplexed and rescaled at least one base layer decoded frame-compatible image is interpolated to a target spatial resolution.

EEE28. The encoding method according to enumerated example embodiment 27 further includes: filtering the demultiplexed and rescaled at least one base layer decoded frame-compatible image to a target spatial resolution.

EEE29. The encoding method according to any one of the enumerated example embodiments 1 to 28, wherein the enhancement layer processing further comprises: controlling at least one enhancement layer reference picture buffer through a memory management control operation (MMCO).

EEE30. The encoding method of Enumerated Example Embodiment 29, wherein each of the at least one enhancement layer reference picture buffer is synchronized according to one or more sets of MMCOs. EEE31.

EEE31. The encoding method according to enumerated example embodiments 29 or 30, wherein the enhancement layer processing further comprises: providing information related to MMCO to a corresponding decoder.

EEE32. An encoding method according to any one of the enumerated example embodiments 1 to 31, wherein the base layer processing encodes the image content within a first frequency range, and the enhancement layer processing encodes the image content within a second frequency range.

EEE33. The encoding method of any one of Enumerated Example Embodiments 1 to 15, wherein the enhancement layer processing further comprises:

selecting, for each of a plurality of data categories, at least one reference picture from at least one enhancement layer reference picture buffer;

acquiring at least one compensated image for each of a plurality of data categories by performing disparity compensation or inter-frame prediction based on at least one reference image; and

At least one compensated image enhancement layer sample and the enhancement layer are multiplexed into at least one frame-compatible image.

EEE34. The encoding method according to enumerated example embodiment 33, wherein the enhancement layer processing further comprises: controlling at least one enhancement layer reference picture buffer through a memory management control operation (MMCO).

EEE35. The encoding method according to enumerated example embodiment 34, wherein the enhancement layer processing further comprises: providing information related to MMCO to a corresponding decoder.

EEE36. An encoding method according to any one of enumerated example embodiments 33 to 35, wherein at least one enhancement layer reference picture buffer stores images at the same or different enhanced resolutions for each of a plurality of data categories.

EEE37. An encoding method according to any one of enumerated example embodiments 33 to 36, wherein at least one enhancement layer reference picture buffer stores images at the same resolution as the original input image.

EEE38. An encoding method according to any one of enumerated example embodiments 33 to 37, wherein the base layer processing encodes image content within a first frequency range and the enhancement layer processing encodes image content within a second frequency range.

EEE39. A coding method for multi-layer frame-compatible video transmission, comprising:

a) performing base layer processing on images or video frames of multiple data categories through a base layer, including:

i) providing a base layer frame compatible representation of images or video frames of multiple data categories; and

b) performing enhancement layer processing on images or video frames of the plurality of data categories via one or more enhancement layers, wherein each of the plurality of data categories is processed separately in a separate enhancement layer, each of the one or more enhancement layers comprising:

i) providing an enhancement layer representation of an image or video for one of a plurality of data categories;

ii) maintaining an enhancement layer reference picture buffer in each enhancement layer;

iii) performing reference processing for at least one dependency on the base layer or a different enhancement layer; and

iv) Perform motion or parallax compensation.

EEE40. The encoding method of enumerated example embodiment 39, wherein the plurality of data categories comprises a plurality of viewpoints for a stereoscopic image or video.

EEE41. The encoding method of Enumerated Example Embodiment 39 or 40, wherein providing a base layer frame-compatible representation of an image or video frame of multiple data categories comprises:

Multiplex image or video frame base layer samples and base layers of multiple data categories into a single frame.

EEE42. An encoding method according to any one of enumerated example embodiments 39 to 41, wherein the base layer processing complies with one of multiple existing video codecs.

EEE43. The encoding method of enumerated example embodiment 42, wherein the plurality of existing video codecs include H.264/AVC, VP8, and VC-1.

EEE44. An encoding method according to any one of the enumerated example embodiments 41 to 43, wherein the sampling of the base layer processing includes: symmetrical or asymmetrical filtering of the images or video frames of the multiple data categories.

EEE45. The encoding method according to any one of enumerated example embodiments 41 to 44, wherein the base layer samples include horizontal samples, vertical samples or quincunx samples.

EEE46. An encoding method according to any one of the enumerated example embodiments 41 to 45, wherein the base layer multiplexing uses any one of the frame-compatible packing arrangements of checkerboard interleaving arrangement, column interleaving arrangement, row interleaving arrangement, side-by-side arrangement and top-bottom arrangement.

EEE47. The encoding method according to any one of the enumerated example embodiments 40 to 46, wherein the base layer processing further comprises: processing equal or unequal amounts of samples of images or video frames of multiple viewpoints.

EE48. The encoding method according to any one of enumerated example embodiments 40 to 47, wherein enhancement layer processing adopts a hybrid video coding model that conforms to any one of a plurality of existing codecs.

EEE49. The encoding method of enumerated example embodiment 48, wherein the plurality of existing codecs include VC-1 and H.264/AVC.

EEE50. The encoding method of any one of enumerated example embodiments 39 to 49, wherein motion or disparity compensation is performed based on at least one image stored in an enhancement layer reference picture buffer.

EEE51. The encoding method of any one of Enumerated Example Embodiments 39 to 49, wherein enhancement layer processing further comprises:

performing base layer to enhancement layer (BL to EL) processing on at least one base layer decoded frame-compatible picture from a reference buffer in the base layer;

The at least one base layer decoded frame-compatible picture processed by BL to EL is stored in at least one enhancement layer reference picture buffer.

EEE52. The encoding method of enumerated example embodiment 51, wherein BL to EL processing comprises reference processing, demultiplexing, and upsampling of at least one base layer decoded frame-compatible image.

EEE53. An encoding method according to enumerated example embodiment 52, wherein BL to EL processing for each of one or more enhancement layers is handled by a single processing unit that jointly performs and optimizes reference processing, demultiplexing and upsampling of at least one base layer decoded frame-compatible image.

EEE54. The encoding method of any one of Enumerated Example Embodiments 39 to 53, wherein enhancement layer processing further comprises:

performing enhancement layer to enhancement layer (EL to EL) processing on at least one reference picture stored in an enhancement layer reference picture buffer in a different enhancement layer;

The at least one EL-to-EL processed enhancement layer reference picture is stored in at least one enhancement layer reference picture buffer of the at least one enhancement layer reference picture.

EEE55. An encoding method according to any one of enumerated example embodiments 51 to 54, wherein the BL to EL processing further comprises: rescaling at least one base layer decoded frame compatible image processed by the BL to EL to a target spatial resolution.

EEE56. The encoding method according to enumerated example embodiments 54 or 55, wherein the EL to EL processing further comprises: rescaling at least one enhancement layer reference image processed by EL to EL to the target spatial resolution.

EEE57. The encoding method of Enumerated Example Embodiment 1, wherein providing enhancement layer frame-compatible representations of image or video frames of multiple data categories comprises:

Differential images of original full-resolution images and at least one base layer to enhancement layer (BL to EL) prediction of multiple data classes are filtered, sampled, and multiplexed.

EEE58. The encoding method of enumerated example embodiment 57, wherein the BL to EL prediction is obtained by demultiplexing and referencing at least one frame-compatible decoded picture from the base layer. EEE59. The encoding method of enumerated example embodiment 57, wherein the BL to EL prediction is obtained by demultiplexing and referencing at least one frame-compatible decoded picture from the base layer.

EEE59. The encoding method according to enumerated example embodiment 58, wherein the demultiplexing and reference processing further comprises: filtering, interpolation or rescaling.

EEE60. The encoding method of enumerated example embodiment 57, wherein the motion or disparity compensation is performed for each of the plurality of data categories. EEE61.

EEE61. An encoding method according to enumerated example embodiment 39, wherein providing an enhancement layer representation of an image or video for one of multiple data categories includes: obtaining at least one differential image and at least one base layer to enhancement layer (BL to EL) prediction of at least one original full-resolution image for one of the multiple data categories.

EEE62. The encoding method of enumerated example embodiment 61, wherein the at least one BL to EL prediction is obtained by demultiplexing and referencing at least one frame-compatible decoded picture from a base layer. EEE63.

EEE63. The encoding method according to enumerated example embodiment 62, wherein demultiplexing and reference processing further comprises filtering, interpolation or rescaling.

EEE64. A decoding method for multi-layer frame-compatible video transmission, comprising:

a) performing base layer processing on a plurality of base layer bit stream signals through a base layer, comprising:

i) providing at least one frame-compatible base layer decoded picture or video frame; and

b) performing enhancement layer processing on the plurality of enhanced bitstream signals through one or more enhancement layers, comprising:

i) providing at least one enhancement layer decoded picture or video frame for a plurality of data categories;

ii) maintaining at least one enhancement layer reference picture buffer;

iii) performing reference processing for at least one dependency on a base layer or a different enhancement layer; and

iv) performing parallax compensation,

Therein, all multiple data categories are decoded and processed in the same enhancement layer.

EEE65. The decoding method of enumerated example embodiment 64, wherein the plurality of data categories comprises a plurality of viewpoints for a stereoscopic image or video.

EEE66. A decoding method according to enumerated example embodiments 64 or 65, wherein the base layer processing complies with one of multiple existing video codecs.

EEE67. The decoding method of enumerated example embodiment 66, wherein the plurality of existing video codecs include H.264/AVC, VP8, and VC-1.

EEE68. A decoding method according to any one of enumerated example embodiments 64 to 67, wherein enhancement layer processing adopts a hybrid video coding model that conforms to any one of multiple existing codecs.

EEE69. The decoding method of enumerated example embodiment 68, wherein the plurality of existing codecs include VC-1 and H.264/AVC.

EEE70. The decoding method of any one of Enumerated Example Embodiments 64 to 69, wherein enhancement layer processing further comprises:

generating at least one reference picture or video frame by predicting at least one predicted picture from samples in the same picture or frame or from samples of a previously decoded frame in the same enhancement layer, and

At least one reference image or video frame is stored in at least one enhancement layer reference picture buffer.

EEE71. The decoding method according to enumerated example embodiment 70, wherein generating at least one reference image or video frame further comprises: demultiplexing and reference processing the sum of at least one image decoded from multiple enhanced bitstream signals and at least one predicted image.

EEE72. The decoding method according to enumerated example embodiment 71, wherein the demultiplexing further comprises: upsampling and interpolating the sum of at least one prediction residual image and at least one prediction image.

EEE73. A decoding method according to any one of enumerated example embodiments 65 to 72, wherein one of the at least one enhancement layer reference picture buffers stores images or video frames at a different resolution than the remaining buffers in the at least one enhancement layer reference picture buffer.

EEE74. The decoding method of any one of Enumerated Example Embodiments 64 to 73, wherein the enhancement layer processing further comprises:

For each of the plurality of data categories, at least one reference picture is selected from at least one enhancement layer reference picture buffer;

multiplexing the selected at least one reference picture enhancement layer sample and the enhancement layer into at least one frame-compatible picture; and

Disparity compensation or inter-frame prediction is performed based on at least one frame-compatible image.

EEE75. The decoding method of Enumerated Example Embodiment 73 or 74, wherein the enhancement layer processing further comprises:

performing base layer to enhancement layer (BL to EL) processing on at least one base layer decoded frame-compatible picture from a reference buffer in the base layer;

The at least one base layer decoded frame-compatible picture processed by BL to EL is stored in at least one enhancement layer reference picture buffer.

EEE76. The decoding method of enumerated example embodiment 75, wherein BL to EL processing comprises reference processing, demultiplexing, and upsampling of at least one base layer decoded frame-compatible image.

EEE77. The decoding method of enumerated example embodiment 76, wherein the BL to EL processing further comprises filtering, upscaling, and interpolating at least one base layer decoded frame-compatible image.

EEE78. The decoding method of any one of enumerated example embodiments 75 to 77, wherein the enhancement layer processing further comprises receiving information related to BL to EL processing from a corresponding encoder. ...

EEE79. A decoding method according to any one of enumerated example embodiments 75 to 78, wherein the base layer processing and the enhancement layer processing process the image at different spatial resolutions.

EEE80. The decoding method of Enumerated Example Embodiment 79, wherein the BL to EL processing comprises:

demultiplexing at least one base layer decoded frame-compatible picture;

interpolating the demultiplexed at least one base layer decoded frame-compatible image; and

The interpolated and demultiplexed at least one base layer decoded frame-compatible image is rescaled to a target spatial resolution.

EEE81. The decoding method of Enumerated Example Embodiment 79, wherein the BL to EL processing comprises:

demultiplexing at least one base layer decoded frame-compatible picture;

rescaling the demultiplexed at least one base layer decoded frame-compatible picture to a different spatial resolution; and

The demultiplexed and rescaled at least one base layer decoded frame-compatible image is interpolated to a target spatial resolution.

EEE82. The decoding method according to enumerated example embodiment 81 further includes: filtering the demultiplexed and rescaled at least one base layer decoded frame-compatible image to a target spatial resolution.

EEE83. The decoding method of any one of enumerated example embodiments 64 to 82, wherein enhancement layer processing further comprises controlling at least one enhancement layer reference picture buffer via a memory management control operation (MMCO).

EEE84. The decoding method of Enumerated Example Embodiment 83, wherein each of the at least one enhancement layer reference picture buffers is synchronized according to one or more sets of MMCOs. EEE85. The decoding method of Enumerated Example Embodiment 83, wherein each of the at least one enhancement layer reference picture buffers is synchronized according to one or more sets of MMCOs.

EEE85. The decoding method of enumerated example embodiment 83 or 84, wherein enhancement layer processing further comprises: receiving information related to the MMCO from a corresponding encoder.

EEE86. The decoding method of any one of Enumerated Example Embodiments 64 to 67, wherein enhancement layer processing further comprises:

selecting, for each of a plurality of data categories, at least one reference picture from at least one enhancement layer reference picture buffer;

acquiring at least one compensated image for each of a plurality of data categories by performing disparity compensation or inter-frame prediction based on at least one reference image; and

The at least one compensated image enhancement layer sample and the enhancement layer are multiplexed into at least one frame-compatible image.

EEE87. The decoding method of enumerated example embodiment 86, wherein the enhancement layer processing further comprises controlling at least one enhancement layer reference picture buffer via a memory management control operation (MMCO).

EEE88. The decoding method of enumerated example embodiment 87, wherein enhancement layer processing further comprises receiving information related to MMCO from a corresponding encoder. ...

EEE89. A decoding method for multi-layer frame-compatible video transmission, comprising:

a) performing base layer processing on a plurality of base layer bit stream signals passing through the base layer, comprising:

i) providing at least one frame-compatible base layer decoded picture or video frame; and

b) performing enhancement layer processing on a plurality of enhancement bitstream signals of a plurality of data categories through the one or more enhancement layers, wherein each of the plurality of data categories is processed separately in a separate enhancement layer, each of the one or more enhancement layers comprising:

i) providing at least one enhancement layer decoded picture or video frame for one of a plurality of data categories;

ii) maintaining at least one enhancement layer reference picture buffer;

iii) referencing at least one dependency on a base layer or a different enhancement layer; and

iv) performing disparity compensation, wherein all of the plurality of data categories are decoded and processed in the same enhancement layer.

EEE90. The decoding method of enumerated example embodiment 89, wherein the plurality of data categories comprises a plurality of viewpoints for a stereoscopic image or video.

EEE91. A decoding method according to enumerated example embodiments 89 or 90, wherein the base layer processing complies with one of multiple existing video codecs.

EEE92. The decoding method of enumerated example embodiment 91, wherein the plurality of existing video codecs include H.264/AVC, VP8, and VC-1.

EEE93. A decoding method according to any one of the enumerated example embodiments 89 to 92, wherein the coding layer processing adopts a hybrid video coding model that conforms to any one of multiple existing codecs.

EEE94. The decoding method of enumerated example embodiment 93, wherein the plurality of existing codecs include VC-1 and H.264/AVC.

EEE95. A decoding method according to any one of enumerated example embodiments 89 to 94, wherein motion or disparity compensation is performed based on at least one image stored in an enhancement layer reference picture buffer.

EEE96. The decoding method of any one of Enumerated Example Embodiments 89 to 95, wherein enhancement layer processing further comprises:

performing base layer to enhancement layer (BL to EL) processing on at least one base layer decoded frame-compatible picture from a reference buffer in the base layer;

The at least one base layer decoded frame-compatible picture processed by BL to EL is stored in at least one enhancement layer reference picture buffer.

EEE97. The decoding method of enumerated example embodiment 96, wherein BL to EL processing comprises reference processing, demultiplexing, and upsampling of at least one base layer decoded frame-compatible image.

EEE98. A decoding method according to enumerated example embodiment 97, wherein BL to EL processing for each of one or more enhancement layers is handled by a single processing unit that jointly performs and optimizes reference processing, demultiplexing and upsampling of at least one base layer decoded frame-compatible image.

EEE99. The decoding method of any one of Enumerated Example Embodiments 89 to 98, wherein enhancement layer processing further comprises:

performing enhancement layer to enhancement layer (EL to EL) processing on at least one reference picture stored in an enhancement layer reference picture buffer in a different enhancement layer;

The at least one EL-to-EL processed enhancement layer reference picture is stored in at least one enhancement layer reference picture buffer of the at least one enhancement layer reference picture.

EEE100. The decoding method of any one of enumerated example embodiments 89 to 99, wherein the BL to EL processing further comprises: rescaling the at least one base layer decoded frame-compatible image processed by the BL to EL process to a target spatial resolution.

EEE101. The decoding method of Enumerated Example Embodiment 99 or 100, wherein the EL-to-EL processing further comprises: rescaling the at least one EL-to-EL processed enhancement layer reference picture to a target spatial resolution.

EEE102. The decoding method of Enumerated Example Embodiment 89, wherein providing at least one enhancement layer decoded picture or video frame for multiple data categories comprises:

A reference picture stored in at least one enhancement layer reference picture buffer is added to at least one base layer to enhancement layer (BL to EL) prediction.

EEE103. The decoding method of enumerated example embodiment 102, wherein the at least one BL to EL prediction is obtained by demultiplexing and referencing at least one frame-compatible decoded picture stored in a reference picture buffer of a base layer. EEE104.

EEE104. The decoding method of enumerated example embodiment 102, wherein disparity compensation is performed for each of a plurality of data categories. EEE105.

EEE105. A decoding method according to enumerated example embodiment 89, wherein providing at least one enhancement layer decoded image or video frame for one of a plurality of data categories includes: adding a reference image stored in at least one enhancement layer reference picture buffer to at least one base layer to enhancement layer (BL to EL) prediction.

EEE106. A decoding method according to enumerated example embodiment 105, wherein at least one BL to EL prediction is obtained by demultiplexing and referencing at least one frame-compatible decoded picture stored in a reference picture buffer of a base layer.

EEE107. The decoding method according to enumerated example embodiment 106, wherein demultiplexing and reference processing further comprises filtering, interpolation or rescaling. EEE107.

EEE108. An encoder for encoding at least one image or video frame according to the method described in one or more of Enumerated Example Embodiments 1 to 63 and 117.

EEE109. An apparatus for encoding at least one image or video frame according to the method described in one or more of Enumerated Example Embodiments 1 to 63 and 117.

EEE110. A system for encoding at least one image or video frame according to the method described in one or more of Enumerated Example Embodiments 1 to 63 and 117.

EEE111. A decoder for decoding at least one image or video frame according to the method described in one or more of enumerated example embodiments 64 to 107 and 118. EEE111.

EEE112. An apparatus for decoding at least one image or video frame according to the method described in one or more of Enumerated Example Embodiments 64 to 107 and 118.

EEE113. A system for decoding at least one image or video frame according to the method described in one or more of enumerated example embodiments 64 to 107 and 118.

EEE114. A computer-readable medium comprising a set of instructions, the set of instructions causing a computer to perform a method according to one or more of enumerated example embodiments 1 to 107, 117, and 118. EEE114.

EEE115. Use of a method according to one or more of enumerated example embodiments 1 to 63 and 117 for encoding at least one image or video frame. EEE115.

EEE116. Use of a method according to one or more of enumerated example embodiments 64 to 107 and 118 for decoding at least one image or video frame. EEE116.

EEE117. The encoding method according to any one of enumerated example embodiments 1 to 38, wherein the at least one enhancement layer reference picture buffer includes at least two enhancement layer reference buffers.

EEE118. The decoding method of any one of enumerated example embodiments 64 to 88, wherein the at least one enhancement layer reference picture buffer comprises at least two enhancement layer reference buffers. ...

All patents and publications mentioned in this specification are indicative of the levels of ordinary skill in the art to which this disclosure relates. All references cited in this disclosure are incorporated by reference to the same extent as if each reference were individually incorporated by reference in its entirety.

The examples set forth above are provided to give those of ordinary skill in the art a complete disclosure and description of how to make and use the embodiments of the present disclosure for multi-layer frame-compatible video transmission, and are not intended to limit the scope of the inventors' consideration of their disclosure. Modifications of the above-described modes for carrying out the present disclosure may be used by those of ordinary skill in the art, and such modifications are intended to be within the scope of the appended claims. All patents and publications mentioned in this specification are indicative of the levels of ordinary skill in the art relevant to the present disclosure. All references cited in this disclosure are incorporated by reference to the same extent as if each reference were individually incorporated by reference in its entirety.

It should be understood that the present disclosure is not limited to specific methods or systems, which can of course vary. It should also be understood that the terms used herein are for the purpose of describing specific embodiments only and are not intended to be limiting. As used in this specification and the appended claims, unless the content clearly indicates otherwise, the singular forms "a," "an," and "the" include multiple referents. Unless the content clearly indicates otherwise, the term "multiple" includes two or more objects. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art relevant to the present disclosure.

A number of embodiments of the present disclosure have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the appended claims.

References

[1]D.C.Hutchison, "Introducing DLP 3-D TV",

http://www.dlp.com/downloads/Introducing DLP 3D HDTV Whitepaper.pdf

[2]Advanced video coding for generic audiovisual services,

http://www.itu.int/rec/recommendation.asp?type=folders&lang=e&parent=T-REC-H.264, March 2010.

[3]SMPTE 421M, "VC-1Compressed Video Bitstream Format and DecodingProcess", April 2006.

[4]A. Tourapis, P. Pahalawatta, A. Leontaris, K. Stec, and W. Husak,

"Encoding and Decoding Architecture for Format Compatible 3D Video Delivery," U.S. Provisional Patent Application No. 61/223,027, July 2009.

[5] A.Leontaris, A.Tourapis, and P.Pahalawatta, "Enhancement Methods for Sampled and Multiplexed Image and Video Data," U.S. Provisional Patent Application No. 61/365,743, July 2010.

[6] A. Tourapis, A. Leontaris, P. Pahalawatta, and K. Stec, "Directed Interpolation/Postprocessing methods for video encoded data," U.S. Provisional Patent Application No. 61/170,995, April 2009.

[7] P. Pahalawatta, A. Tourapis, W. Husak, "Systems and Methods for Multi-Layered Image and Video Delivery Using Reference Processing Signals", U.S. Provisional Patent Application No. 61/362,661, July 2010.

Claims (12)

1. A decoding method for multi-layer frame-compatible video transmission, comprising: a) Performing basic layer processing on multiple basic layer bitstream signals through the basic layer, including: i) Provide at least one frame-compatible base layer decoded image or video frame; and b) Enhancement layer processing of multiple enhanced bitstream signals through one or more enhancement layers, including: ii) Provide at least one enhancement layer for decoded image or video frames for multiple viewpoints; iii) performing reference processing on at least one frame-compatible base layer decoded image or video frame from the base layer or at least one decoded image or video frame from a different enhancement layer; and iv) Implement parallax compensation, In this process, all of the multiple viewpoints are decoded and processed within the same enhancement layer. The enhancement layer processing further includes: At least one reference image or video frame is generated by predicting at least one prediction image based on samples from the same image or frame or based on samples from previously decoded frames in the same enhancement layer. The at least one reference image or video frame is stored in at least one enhancement layer reference image buffer. 2. The method of claim 1, wherein the plurality of viewpoints includes a plurality of viewpoints for stereoscopic images or videos. 3. The method of claim 1, wherein providing at least one frame-compatible base layer decoded image or video frame from the plurality of viewpoints comprises: Basic layer sampling, wherein the basic layer sampling includes horizontal sampling, vertical sampling, or quincunx sampling; and multiplexing the basic layers of image or video frames from multiple viewpoints into a single frame. 4. The method according to claim 3, wherein the sampling of the base layer processing includes: performing symmetric or asymmetric filtering on the image or video frames of the plurality of viewpoints. 5. The method according to claim 3, wherein the basic layer multiplexing uses any one of the following frame-compatible packing arrangements: checkerboard interleaving, column interleaving, row interleaving, side-by-side arrangement, and top-bottom arrangement. 6. The method according to claim 2, wherein the basic layer processing further comprises: processing equal or unequal amounts of samples of image or video frames from the plurality of viewpoints. 7. The method according to claim 1, wherein generating the at least one reference image or video frame further comprises: demultiplexing and referencing the sum of the at least one predicted residual image and the at least one predicted image. 8. The method of claim 7, wherein the demultiplexing further comprises: upsampling and interpolating the sum of the at least one predicted residual image and the at least one predicted image. 9. The method of claim 8, wherein one of the at least one enhancement layer reference image buffers stores an image or video frame at a different resolution than the other buffers in the at least one enhancement layer reference image buffer. 10. The method of claim 1, wherein the enhancement layer processing further comprises: For each of the plurality of viewpoints, at least one reference image is selected from the at least one enhancement layer reference image buffer; Enhancement layer sampling and enhancement layer multiplexing of at least one selected reference image into at least one frame-compatible image; and Perform disparity compensation or inter-frame prediction based on the at least one frame-compatible image. 11. The method of claim 1, wherein the enhancement layer processing further comprises: At least one base layer-to-enhancement layer processing is performed on a base layer-decoded frame-compatible image from a reference buffer in the base layer. At least one base layer decoded frame-compatible image processed from the base layer to the enhancement layer is stored in the reference image buffer of the at least one enhancement layer. 12. The method of claim 11, wherein the base layer to enhancement layer process comprises one or more of the following: The at least one base layer decoded frame-compatible image is subjected to reference processing, demultiplexing, and upsampling; or The at least one base layer decoded frame compatible image is filtered, upscaled, and interpolated.