HK1218196B - Coding and decoding methods of a picture block, corresponding devices and data stream - Google Patents

Description

Picture block encoding and decoding method, corresponding equipment and data flow

Technical Field

A method for decoding a picture block from a specially reconstructed reference picture is disclosed. Also disclosed are a corresponding encoding method and corresponding encoding and decoding devices.

Background Art

During video streaming, available bandwidth may change over time. Therefore, the output bit rate of the streaming application needs to be adjusted to fit the available bandwidth in real time in order to avoid congestion. One way to enable real-time bit rate adjustment is to use a real-time encoder, but this requires allocating a coding system to each client, which may be unacceptable when there are many clients for a VOD service. Another way to enable real-time bit rate adjustment is to use scalable video coding. In scalable coding, the video source is encoded into several layers. During transmission, in order to adjust the output bit rate, the server selects the layer to be sent (mode "push") or the decoder requests the layer to be sent (mode "pull back"). This method is suitable for streaming over heterogeneous channels, but compared to single-layer video coding, scalable video coding reduces the overall compression efficiency and increases the computational complexity of both the encoder and decoder. A simple way to achieve bit rate adjustment is to encode multiple versions of the same video sequence. These versions have different resolutions and/or quality levels and therefore have different bit rates. During streaming, when the output bit rate needs to be adjusted, the stream to be transmitted can be dynamically switched from one version to the other to suit bandwidth requirements or user capabilities, as shown in Figure 1. This scheme is called "stream switching". However, direct switching between streams at inter-frame coded pictures (P or B pictures) may cause mismatches in the reconstructed reference pictures and lead to incorrect picture reconstruction. The quality of the reconstructed video may be significantly reduced. One way to solve this problem is to use random access points (RAPs) in the bitstream (typically, I pictures, IDR pictures, or CRA pictures). IDR is the acronym for "Instantaneous Decoder Refresh" and CRA is the acronym for "Clean Random Access". Because switching can only be performed at these RAPs, RAPs need to be frequently allocated in the bitstream to achieve prompt stream switching. However, encoding such I/IDR pictures introduces a large bitrate overhead. In addition, pictures after the RAP that use the reconstructed reference picture located before the RAP are either skipped or not decoded correctly because they use a different reconstructed reference picture than that used in the encoding as shown in Figure 2. In Figure 2, Ic is reconstructed from the reconstructed reference pictures I1 and I2, and is encoded based on the reconstructed reference pictures i1 and i2.

In AVC, special picture types (SI/SP) are designed to allow identical reconstruction of pictures from other streams, thus facilitating stream switching. Consequently, video pictures are encoded as SP pictures at switching points rather than intra-coded pictures, as shown in Figure 3. SP pictures are more efficiently coded than intra-coded pictures, but they are still less efficient than normal P pictures. Therefore, even with many switching points, overall coding efficiency is still reduced.

In a paper titled "Efficient bit stream switching of H.264 coded video" by Zhou et al. and presented at the SPIE conference vol. 5909 (2005), a scheme is disclosed that enables switching at any time without requiring a large bitrate overhead. This scheme is only used for the IPPP GOP structure. In addition to multiple versions of the same video sequence at different bitrates, a DIFF picture is encoded, which is used as a reconstructed reference picture for the current picture when switching occurs, as shown in Figure 4. The DIFF picture is the difference between the reconstructed reference picture of the current picture and the temporally corresponding picture in the other stream. The difference picture is transmitted to the decoder to compensate for the mismatch. Because, as mentioned on page 5 of the paper, the DIFF picture is only transmitted when switching occurs, the bitrate overhead introduced by the above scheme is very small. On the other hand, this scheme only works for P pictures predicted from a single reconstructed reference picture. In addition, this scheme requires that the encoding order and the display order are the same.

Summary of the Invention

A method for decoding a picture block is disclosed. The method comprises:

- decoding at least one stream S_diff into decoded data and an information item for identifying a reconstructed reference picture in the decoder;

- reconstructing a special reference picture from at least the identified reconstructed reference picture and from the decoded data;

- reconstructing a picture block from at least a special reference picture, wherein the special reference picture is not displayed when being reconstructed.

Advantageously, the identified reconstructed reference picture is decoded from a first layer, and wherein decoded data and information identifying the reconstructed reference picture in a decoder picture buffer are decoded from a second layer dependent on the first layer.

According to certain characteristics, the first layer is the base layer.

According to a particular embodiment, the decoding method further comprises decoding a flag indicating that a subsequently decoded picture of the second layer does not use any inter-layer prediction.

A method for encoding a picture block is also disclosed. The encoding method further comprises:

- encoding a picture block based on at least one reconstructed reference picture; and

- encoding at least one reconstructed reference picture as a special reference picture based on the further reconstructed reference picture and for indicating the further reconstructed reference picture in a decoder picture buffer, wherein the special reference picture is not displayed when being reconstructed.

Advantageously, the identified reconstructed reference picture is encoded in a first layer, and at least one reconstructed reference picture and identification of a further reconstructed reference picture in a decoder buffer are encoded in a second layer dependent on the first layer.

According to certain characteristics, the first layer is the base layer.

According to a specific embodiment, encoding a flag indicating that a subsequently decoded picture of the second layer does not use any inter-layer prediction is further included.

A decoding device for decoding a picture block is disclosed. The decoding device comprises:

- means for decoding at least one stream S_diff into decoded data and one piece of information for identifying a reconstructed reference picture in the decoder;

- means for reconstructing a special reference picture from at least the identified reconstructed reference picture and from the decoded data;

- means for reconstructing a picture block from at least a special reference picture, wherein the special reference picture is not displayed when being reconstructed.

The decoding device is adapted to perform the steps of the decoding method.

A coding method for coding a picture block is disclosed. The coding device comprises:

- encoding a picture block based on at least one reconstructed reference picture; and

- encoding at least one reconstructed reference picture as a special reference picture based on the further reconstructed reference picture and for indicating the further reconstructed reference picture in a decoder picture buffer, wherein the special reference picture is not displayed when being reconstructed.

The encoding device is adapted to the steps of the encoding method.

A data stream is disclosed, which contains information encoded therein for identifying a reconstructed reference picture in a decoder picture buffer and data allowing reconstruction of a special reference picture from the identified reconstructed reference picture, the special reference picture being a reference picture that is not displayed.

A decoding method is disclosed. The decoding method includes:

reconstructing a reference picture from a further reference picture of a decoder picture buffer of a first layer of a multi-layer stream and data decoded from a second layer of the multi-layer stream, and storing the reconstructed reference picture in a decoder picture buffer of the second layer, wherein the reference picture is indicated not to be displayed;

decoding a flag indicating that a subsequently decoded picture of the second layer does not use any inter-layer prediction; and

Picture blocks of the subsequently decoded picture are reconstructed at least from the reconstructed reference image.

Advantageously, the decoding method further comprises decoding information identifying said further reference picture from said second layer.

Advantageously, in the decoding method, data decoded from the second layer represent pixel-by-pixel differences between the reference picture and the further reference picture.

Advantageously, in the decoding method, the first layer is a base layer.

Advantageously, in the decoding method, said reference picture and said further reference picture are time aligned.

A coding method is disclosed. The coding method includes:

encoding, in a second layer of a multi-layer stream, a reference picture of a decoder picture buffer of a first layer of the multi-layer stream based on a further reference picture of a decoder picture buffer of the second layer, wherein the reference picture is indicated as not to be displayed;

encoding a flag indicating that a subsequently coded picture of the second layer does not use any inter-layer prediction; and

Picture blocks of the subsequently coded picture are encoded at least based on the reference picture.

Advantageously, the encoding method further comprises encoding in said second layer information for identifying said further reference picture.

Advantageously, in the encoding method, encoding the reference picture in the second layer comprises determining pixel-by-pixel differences between the reference picture and the further reference picture, and encoding the pixel-by-pixel differences.

Advantageously, in the encoding method, the first layer is a base layer.

Advantageously, in the encoding method, said reference picture and said further reference picture are time aligned.

A decoder is disclosed. The decoder includes:

means for reconstructing a reference picture from a further reference picture of a decoder picture buffer of a first layer of a multi-layer stream and data decoded from a second layer of the multi-layer stream, and storing the reconstructed reference picture in a decoder picture buffer of the second layer, wherein the reference picture is indicated not to be displayed;

means for decoding a flag indicating that a subsequently decoded picture of the second layer does not use any inter-layer prediction; and

Means for reconstructing picture blocks of said subsequently decoded picture at least from the reconstructed reference picture.

Advantageously, the decoder further comprises means for decoding from said second layer information identifying said further reference picture.

Advantageously, in the decoder, data decoded from the second layer represents pixel-by-pixel differences between the reference picture and the further reference picture.

Advantageously, in the decoder, the first layer is a base layer.

Advantageously, in the decoder, said reference picture and said further reference picture are time aligned.

An encoder is disclosed. The encoder includes:

means for encoding, in a second layer of a multi-layer stream, a reference picture of a decoder picture buffer of a first layer of the multi-layer stream based on a further reference picture of a decoder picture buffer of the second layer, wherein the reference picture is indicated as not to be displayed;

means for encoding a flag indicating that a subsequently coded picture of the second layer does not use any inter-layer prediction; and

Means for encoding picture blocks of said subsequently coded picture at least based on said reference picture.

Advantageously, the encoder further comprises means for encoding, in said second layer, information identifying said further reference picture.

Advantageously, in the encoder, encoding the reference picture in the second layer comprises determining pixel-by-pixel differences between the reference picture and the further reference picture, and encoding the pixel-by-pixel differences.

Advantageously, in the encoder, the first layer is a base layer.

Advantageously, in the encoder, said reference picture and said further reference picture are time aligned.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features and advantages of the present invention will emerge from the description of some embodiments thereof, which is made in conjunction with the following drawings:

Figures 1 and 2 illustrate the general principles of stream switching;

FIG3 illustrates the principle of stream switching using SI/SP pictures according to the state of the art;

FIG4 illustrates the principle of stream switching using DIFF pictures according to the state of the art;

FIG5 illustrates a flowchart of a decoding method according to the present invention;

FIG6 illustrates a flow chart of an encoding method according to the present invention;

FIG7 illustrates the principle of stream switching using SRP pictures according to the present invention;

FIG8 illustrates another embodiment of a decoding method according to the present invention;

FIG9 illustrates a multi-layer video decoder according to the present invention;

FIG10 illustrates a multi-layer video encoder according to the present invention; and

FIG. 11 shows a multi-layer flow according to the present invention.

DETAILED DESCRIPTION

The present invention relates to a method for decoding a picture block of pixels and a method for encoding such a picture block. A picture block belongs to a picture of a picture sequence. Each picture contains pixels or picture points, each of which is associated with at least one item of picture data. An item of picture data is, for example, a luminance data item or a chrominance data item. The encoding and decoding methods are described below with reference to picture blocks. Obviously, these methods can be applied to several picture blocks of a picture and several pictures of a sequence, with a view to encoding and decoding one or more pictures respectively. A picture block is a collection of pixels in any form. It can be square or rectangular. However, the present invention is not limited to such a form. In the following, the word block is used for a picture block. In HEVC, a block refers to a coding unit (CU).

The "predictor" item specifies data used to predict other data. A predictor is used to predict a picture block. The predictor, or prediction block, is obtained from one or more reconstructed reference samples of the same picture as the block being predicted (spatial prediction or intra-picture prediction) or one (unidirectional prediction) or more (bidirectional prediction or bi-prediction) reference blocks of a reconstructed reference picture (temporal prediction or inter-picture prediction). The reference block is identified in the reconstructed reference picture by a motion vector. The prediction can also be weighted to account for illumination variation models (also known as weighted prediction).

The term "residual" refers to the data obtained after subtracting the predictor from the source data.

The term "reconstruction" refers to the data (e.g., pixels, blocks) obtained after fusing the residual with the predictor. Fusion is typically the summation of the predictor and the residual. However, fusion more generally and more particularly includes an additional post-filtering stage of the reconstructed samples and/or an additional step of adding offsets to the reconstructed samples. When a reference picture is reconstructed, it is stored in the DPB (an acronym for "encoder picture buffer") as a newly reconstructed reference picture.

When referring to decoding a picture, the terms "reconstruction" and "decoding" are often used as synonyms. Therefore, a "reconstructed block" is also referred to as a "decoded block."

The term encoding is used in the broadest sense. Encoding may include applying a transform and/or quantizing the data. It may also simply refer to entropy coding. The DCT ("Discrete Cosine Transform") is an example of such a transform. In the same way, the term decoding may include entropy decoding as well as applying a transform and/or inverse quantization. The transform applied at the decoder is the inverse of the transform applied at the encoder.

A stream is a sequence of bits forming a representation of a coded picture and associated data forming one or more coded video sequences. Stream is a general term used to refer to a stream of NAL units or a stream of bytes.

A NAL (an acronym for "Network Abstraction Layer") unit is a syntactic structure containing an indication of the type of data to be followed and the bytes containing the data. The NAL is specified to format the data and provide header information in a manner suitable for transport over various communication channels or storage media. All data is contained in NAL units, each of which contains an integer number of bytes. The NAL unit specifies a common format for use in both packet-oriented systems and streaming systems. The format of the NAL unit for packet-oriented transport and byte streams is the same, except that in the byte stream format each NAL unit may be preceded by a start code prefix and additional padding bytes.

An AU (the acronym for “Access Unit”) is a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain exactly one coded picture. Decoding an access unit always results in a decoded picture.

In Figures 5 and 6, the represented boxes are only functional entities, which do not necessarily correspond to physically separated entities. It will be appreciated by those skilled in the art that the aspects of the principles can be implemented as systems, methods or computer-readable media. Therefore, the aspects of the principles can take the form of complete hardware implementation, the form of complete software implementation (including firmware, resident software, microcode, etc.) or the implementation of the combined software and hardware aspects generally all referred to as "circuit", "module" or "system" in this article. In addition, the aspects of the principles can take the form of computer-readable storage media. Any combination of one or more computer-readable storage media can be utilized.

The flow chart and/or block diagram in the figure illustrate the configuration, operation and function of the possible implementation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, fragment or code, which contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions recorded in the box may not be performed in the order recorded in the figure. For example, the two boxes shown in a sequential manner can actually be performed substantially in parallel, or the boxes can sometimes be performed in the opposite order, or the boxes can be performed in an alternating order, depending on the functions involved. It should also be noted that each box of the block diagram and/or flow chart example and the combination of the boxes in the block diagram and/or flow chart example can be implemented by executing a specified or capable or available system based on dedicated hardware or a combination of dedicated hardware and computer instructions. Although not explicitly described, the embodiments can be utilized in any combination or sub-combination.

5 shows a flow chart of a decoding method according to a specific and non-limiting embodiment. The method is used to decode a current picture block Bc encoded in a stream S. The picture block Bc belongs to a slice Sc of the current picture Ic. A slice is a part of a picture, such as a collection of picture blocks.

In step 10 , at least one stream S_diff is decoded into decoded data (eg, residual and coding mode) and information INFO for identifying the reconstructed reference picture R2 stored in the DPB.

In step S12, a special reference picture (SRP) R1′ is reconstructed from the identified reconstructed reference picture R2 and the decoded picture. The special reference picture is then placed in the DPB. This reference picture R1′ is special in that it is never displayed but is only used to reconstruct blocks in other pictures. Reconstructing the SRP R1′ involves determining a predictor for each picture block of R1′ and adding a residual. The predictor can be determined based on the identified reconstructed reference picture R2 (as a block in R2 co-located with Bc or a motion-compensated block in R2 identified by a motion vector) or based on neighboring reconstructed samples of R1′ as in conventional intra prediction. A block in R2 is co-located with Bc if its spatial location in R2 is the same as that of Bc in Ic. According to a variant, if the size of the reconstructed reference picture R2 is different from that of the current picture Ic, R2 is rescaled to reconstruct the special reference picture so that the rescaled R2 picture (possibly with appropriate padding) has the same size as Ic. In this case, R1' is reconstructed from F(R2), where F is the rescaling filter. The stream S_diff may be part of the stream S or may be independent of it.

As an example, the stream S_diff encodes the pixel-by-pixel difference between a reconstructed reference picture R1 that is different from R2 and the reconstructed reference picture R2. R1 is, for example, the reconstructed reference picture based on which the current picture block Bc is encoded. In this case, decoding the stream S_diff involves decoding the difference picture, which is typically performed by entropy decoding, inverse quantization, and a transform. The transform is, for example, an inverse DCT. The difference picture is typically an approximation of the difference between the reconstructed reference picture R1 and the reconstructed reference picture R2. The approximation is due to losses during encoding (e.g., due to quantization). If the difference picture DIFF is losslessly coded, the decoded difference picture DIFF is equal to the difference between the reconstructed reference picture R1 and the reconstructed reference picture R2. According to a variant, if R1 and R2 are of different sizes, the difference picture is the difference between the reconstructed reference picture R1 and the rescaled reconstructed reference picture R2. As an example, if R2 is larger than R1, R2 is downscaled, and if R2 is smaller than R1, R2 is upscaled. In this case, the special reference picture R1' is equal to F(R2)+DIFF, where F is the identity matrix if R2 and Ic are of the same size, or a rescaling function in other cases.

According to a variant, the decoding method also comprises the optional decoding of a sign associated with the difference picture DIFF. If such a sign is decoded, the special reference picture R1' is equal to F(R2)+DIFF when the sign is positive and equal to F(R2)-DIFF when the sign is negative.

According to another variant, the stream S_diff encodes, for some blocks of R1, the differences between these blocks and the co-located blocks in R2. Other blocks of R1 are encoded in S_diff using conventional intra prediction (ie from neighbouring reconstructed samples).

According to another variant, the stream S_diff encodes the differences between some blocks of R1 and the corresponding blocks in R2. The corresponding blocks in R2 are either co-located blocks or motion-compensated blocks. The other blocks in R1 are encoded in S_diff using conventional intra prediction (i.e., based on neighboring reconstructed samples).

Decoding the information INFO enables handling of different use cases. As an example, if the current picture block Bc is encoded based on two reconstructed reference pictures R1 and r1, two special reference pictures R1' and r1' and two pieces of information INFO and info are decoded in step 10. The special reference pictures R1' and r1' correspond to R2 and r2, respectively, where R2 and r2 are the two reconstructed reference pictures stored in the DPB from which Bc is reconstructed. Therefore, INFO indicates to the decoder that R1' will be reconstructed from R2, while info indicates that r1' will be reconstructed from r2.

Each special picture is identified, for example, in the stream S_diff, by a dedicated flag indicating a picture/slice type that is different from the conventional I, P, or B picture/slice types. This picture/slice type indicates that the current AU contains a special reference picture that is not displayed. According to a variant, each special picture is identified by a dedicated flag in the slice header.

According to a variant, the picture slice type is I, P or B, but a special flag in the slice header indicates that the reconstructed picture is not displayed but stored as a reference in the DPB.

The information INFO for identifying the reconstructed reference picture R2 in the DPB is, for example, POC (Picture Order Count) defined in ISO/IEC 14496-10 (Section 3.104). According to a variant, the information for identifying the reconstructed reference picture is a reconstructed reference picture index.

In step 16, the current picture block Bc is reconstructed from the special reference picture R1′. Typically, drift is reduced because the special reference picture is closer in content to R1 than to R2. Typically, reconstructing the picture block involves decoding the residual from stream S and adding the residual to the predictor. In skip mode, the residual can be zero. Decoding the residual involves entropy decoding, inverse quantization, and applying the inverse of the transform applied at the encoder. These steps are well known to those skilled in the art of video compression/coding and are not further disclosed. A reference block in the special reference picture R1′ is identified by a motion vector decoded from stream S. The reference block is used as the predictor. In the case of bidirectional prediction, the two reference blocks are identified by two reconstructed reference pictures (possibly the same reconstructed reference picture). The predictor is the weighted sum of these two reference blocks. If Bc is bidirectionally predicted from two reference blocks belonging to two reconstructed reference pictures R2 and r2, which may be different from the reconstructed reference pictures R1 and r1 used in encoding, then two SRPs R1′ and r1′ may be reconstructed. Special reference pictures R1' and r1' are thus used as reference pictures for Bc. If r1 is available in the DPB when reconstructing Bc, Bc can also be reconstructed from a special reference picture R1' and r1. INFO and symbols can be decoded for each special reference picture (in the slice header or in the slice segment header) or can be grouped in a single header for several special reference pictures. INFO and symbols can be decoded, for example, from SEI messages, VPS (Video Parameter Set HEVC), or from the slice header of Sc.

Figure 6 shows a flow chart of a coding method according to a specific and non-limiting embodiment.The method is used to code a current picture block Bc in a stream S.

In step S20, the current picture block Bc is encoded from at least one first reconstructed reference picture R1 in the stream S1. Typically, encoding the current picture block comprises determining a residual; transforming the residual; and quantizing the transformed residual into quantized data. The quantized data is further entropy encoded in the stream S. The residual is obtained by subtracting a predictor from the current picture block Bc. The predictor is determined from the first reconstructed reference picture R1. More precisely, the predictor is determined in the reconstructed reference picture R1 via a motion vector. If the current block is bidirectionally predicted from two reference blocks, the predictor is obtained by averaging the two reference blocks. The two reference blocks may belong to two different reconstructed reference pictures R1 and r1, or to the same reconstructed reference picture. The motion vector is also encoded in the stream S. These steps are well known to those skilled in the art of video compression and are not disclosed further.

In step S24, the reconstructed reference picture R1 and the information INFO are encoded into the stream S_diff. The decoding of S_diff is SRP. The stream S_diff can be part of the stream S or can be independent of the stream S. The reconstructed reference picture R1 is encoded in S_diff based on a second reconstructed reference picture R2, which is different from R1, identified by INFO. According to a variant, if the size of the reconstructed reference picture R2 is different from the size of the current picture Ic, and therefore different from the size of R1, R2 is rescaled so that the reconstructed reference picture R1 is encoded so that the rescaled R2 picture (possibly with appropriate padding) has the same size as Ic. In this case, R1 is encoded according to F(R2), where F is the rescaling filter.

As an example, the stream S_diff encodes the pixel-by-pixel difference DIFF between R1 and R2. The DIFF picture is encoded by transformation (e.g., using DCT), quantization, and entropy coding. According to a variant, if R1 and R2 are of different sizes, the difference picture is the difference between the reconstructed reference picture R1 and the rescaled second reconstructed reference picture R2. As an example, if R2 is greater than R1, R2 is scaled down, while if R2 is less than R1, R2 is scaled up. In this case, DIFF = R1 - F(R2), where F is the identity matrix function when R2 and Ic are of the same size and a rescaling function otherwise.

According to a variant, the decoding method also comprises the optional decoding of a sign associated with the difference picture. If such a sign is decoded, the special reference picture R1' is equal to F(R2)+DIFF when the sign is positive and to F(R2)-DIFF when the sign is negative.

According to another variant, the stream S_diff encodes the differences between some blocks of R1 and blocks in R2 (i.e. blocks co-located with Bc or motion compensated blocks). Other blocks of R1 are encoded in S_diff using traditional intra prediction (i.e. from neighboring reconstructed samples).

Encoding the information INFO enables handling of different use cases. For example, if the current picture block Bc is encoded from two reconstructed reference pictures R1 and r1, the two reconstructed reference pictures are encoded based on two other reconstructed reference pictures R2 and r2. INFO indicates to the decoder that a special reference picture R1' will be reconstructed from R2, while info indicates that another special reference picture r1' will be reconstructed from r2. Each special reference picture is identified, for example, in the stream S_diff with a dedicated flag indicating a picture/slice type that is different from the traditional I, P, and B picture/slice types. This picture/slice type indicates that the current AU is a special reference picture that will be used to replace a picture in the DPB. According to a variant, each special picture is identified with a dedicated flag in the slice header.

According to a variant, the picture slice type is I, P or B, but a special flag in the slice header indicates that the reconstructed picture is not displayed but stored as a reference in the DPB.

In a particular embodiment, a special reference picture and information INFO are encoded for several or each possible pair of reconstructed reference pictures of the DPB. Thus, at any time, block Bc can be reconstructed from any picture of the DPB, even if it is not from the one from which it was coded while limiting drift. When reconstructing Bc, if R1 is not available in the DPB, Bc can be reconstructed from special reference picture R1' instead of R2. Drift is thus limited because, in terms of content, R1' is closer to R1 than to R2.

The information identifying the second reconstructed reference picture is, for example, a POC.According to a variant, the information identifying the second reconstructed picture is a reconstructed reference picture index.

All variants and options of the disclosed decoding method are applicable to the encoding method. In particular, the encoding method comprises the optional encoding of symbols associated with the difference picture.

INFO and SYMBOL are decoded from, for example, SEI messages, VPS (Video Parameter Set HEVC) or from the slice header of Sc.

According to a variant, the encoding and decoding method is used in the context of stream switching, as shown in FIG7 . In this case, a first sequence of pictures is encoded in stream S0 . A second sequence of pictures is encoded in stream S1 . Typically, the second sequence of pictures is identical to the first sequence, but is encoded at a different bit rate, i.e. by using a different quantization step size. According to a variant, the second sequence of pictures is a rescaled version of the first sequence, i.e. an upscaled or downscaled version. According to a particular embodiment, S0 and S1 have the same GOP structure (i.e. the same decoding order and the same reference picture list, as defined in sections 8.3.1 and 8.3.2 of the HEVC standard).

In addition to streams S0 and S1, at each time instant tn, the reconstructed reference picture of S1 is further encoded in the stream S_diff as the SRP of the temporally corresponding, i.e., time-aligned (e.g., same picture order count) reconstructed reference picture from S0 as shown in FIG7 . The reconstructed reference picture is encoded in S_diff together with information info_tn for identifying the corresponding reconstructed reference picture. Note that the source picture corresponding to is encoded in S1, and the source picture corresponding to is encoded in S0.

The decoding method disclosed with reference to FIG5 is used to decode a picture block Bc after switching from a first stream S0 to a second stream S1. Referring to FIG7, a picture is decoded from stream S0 and displayed until time t2. A switch occurs between t2 and t3. After the switch, a picture is decoded and displayed from stream S1. At the time of the switch, DBP0 contains several reconstructed reference pictures decoded from S0. DPB0 is associated with S0. Referring to FIG7, PB0 contains three reconstructed reference pictures at the switching time and

In step 10 , S_diff1 , S_diff2 , and S_diff3 are decoded into decoded data (eg, residual or coding mode) and information info_t0 , info_t1 , and info_t2 identifying the reconstructed reference pictures stored in DPB0 .

In step 12, three special reference pictures SRP_t0, SRP_t1, and SRP_t2 are reconstructed from the corresponding decoded data and the corresponding reconstructed reference pictures. The reconstructed SRPs are then stored in DPB1, which may be different from DPB0. DPB1 is related to S1. According to a first specific embodiment, S_diff encodes the pixel-by-pixel differences between temporally corresponding pictures, which may be rescaled. In this case, the reconstructed SRPs are where diff_t0, diff_t1, and diff_t2 are decoded from S_diff. If necessary, they are rescaled using F to have the same size as the current picture Ic. If no rescaling is performed, F is the identity matrix function. According to a second specific embodiment, S_diff is encoded using the pair, which may be rescaled by F. In this case, the predictor for the block in is either a spatially co-located block in the picture, or a motion-compensated block in or derived from a spatially neighboring block in (spatial intra prediction). In the case of the first specific embodiment, when rescaling is not required, that is, when the picture sizes of the first and second streams are the same, the same difference pictures diff_t0, diff_t1, and diff_t2 can be used to switch from S0 to S1 or vice versa. In the previous example, if diff_t0 encodes the difference between temporally corresponding pictures in stream S1, rather than the reverse, diff_t0 is subtracted from it to reconstruct SRP_t0. Thus, a symbol is decoded to indicate whether the reconstructed reference picture is modified by adding or subtracting the difference picture.

In step S16, Bc is reconstructed from the reconstructed reference pictures in DPB 1. Just after switching, DPB 1 contains three SRPs.

The present invention is obviously not limited to the case of three reconstructed reference pictures. According to a specific embodiment of the present invention, for all reconstructed reference pictures in DPB0, a special reference picture is reconstructed in step 12 and stored in DPB1. According to a variant, an SRP is reconstructed only for each reconstructed reference picture in DPB0 that will be used as a reference picture after switching.

According to a variant, the flag f13 is encoded (respectively, decoded), for example in a VPS or SEI, indicating that subsequently encoded (respectively, decoded) pictures with a given layer_id do not use any inter-layer prediction. More precisely, pictures encoded (respectively, decoded) after the flag do not use any inter-layer prediction.

Figure 8 illustrates another embodiment of a decoding method according to a specific and non-limiting embodiment. The decoder receives different access units. First, access unit AU1 is received and decoded. A first picture I1 is reconstructed from the decoded AU1. Then, a second access unit AU2 is received and decoded. A second picture I2 is reconstructed from the decoded AU2. Pictures I1 and I2 belong to the same stream S0 and are stored in DPB0 when they are signaled to be used as reference pictures. Then, a switch occurs. This switch can be requested by the decoder sending a request to the encoder to receive the S_diff stream. According to a variant, the switch is initiated by the encoder. After this switch, the decoder receives two AU units S_diff1 and S_diff2. S_diff1 and S_diff2 (step 10) are encoded so that SRP1 and SRP2 are reconstructed (step 12) using pictures I1 and I2, respectively. SRP1 and SRP2 are two special reference pictures. SRP1 and SRP2 are then placed in DPB1 associated with S1. The decoder then receives AU3 and decodes it. Picture I3 is reconstructed from the decoded AU3 and possibly from at least one picture of DPB1 (temporal prediction), i.e., SRP1 or SRP2. I3 belongs to the second stream S1 and may be stored in DPB1 for future use as a reference picture for reconstruction. The decoder then receives AU4 and decodes it. Picture I4 is reconstructed from the decoded AU4 and possibly from at least one picture of DPB1 (temporal prediction). Pictures I1, I2, I3, and I4 are displayed, but SRP1 and SRP2 are not. In fact, only one of the two time-aligned pictures is displayed. SRP1 is time-aligned with I1, while SRP2 is time-aligned with I2.

According to a specific embodiment of the present invention, the pictures of the first and second sequences and the special reference pictures are encoded into a multi-layer stream. As a specific example, the pictures identified as special reference pictures are encoded as enhancement layers of a scalable stream that depends on another layer (stream S0), such as the base layer in which the pictures of the first sequence are encoded. If the first layer needs information from the second layer for decoding, the first layer depends on the second layer. The enhancement layer allows the reconstruction of special reference pictures from the reconstructed reference pictures of S0 to be used as reference pictures for reconstructing pictures of S1 after switching from S0 to S1. The enhancement layer is compatible with the SVC or SHVC coding standards, for example. According to a specific embodiment of the present invention, the special reference pictures are encoded using a subset of the coding tools/modes provided by SVC or SHVC for encoding the enhancement layer. According to another embodiment of the present invention, intra-layer motion vector prediction (temporal prediction) is disabled in the SVC or SHVC coding standards. Instead, intra-frame prediction from the S0 layer is activated. Intra-picture prediction can also be activated. According to another embodiment, temporal MV prediction is disabled for encoding S0 and S1, for example by setting the HEVC flag slice_temporal_mvp_enable_flag to false. This means that motion vector prediction (MV prediction) is built using MVs from reconstructed neighboring coding units instead of using MVs of previously reconstructed reference pictures.

In the following Figures 9 and 10, the encoding and decoding modules are referred to as encoder and decoder.

FIG9 illustrates a multi-layer encoder according to a specific and non-limiting embodiment. A first sequence of pictures is encoded in S0 using a first encoder ENC0, such as an MPEG2, H.264, or HEVC-compatible encoder, as a single-layer encoder. The present invention is not limited to the single-layer encoder used. Reference pictures encoded with ENC0 are reconstructed as R2 and provided as input to a third encoder ENC2. A second encoder ENC1 is used to encode a second sequence of pictures in S1. The present invention is not limited to the encoder used. Reference pictures encoded with ENC1 that temporally correspond to the reconstructed reference pictures R2 are reconstructed as R1 and provided as input to the third encoder ENC2. Thus, for each reconstructed reference picture R2 in the DPB of ENC0, a temporally corresponding reference picture R1 is reconstructed. Consequently, encoder ENC2 encodes the reconstructed reference picture R1 based on the temporally corresponding reconstructed reference picture R2, which may be rescaled to the stream S_diff. According to a particular embodiment, the encoder ENC2 comprises a subtractor for subtracting R2 (possibly rescaled) from R1, and further comprises an entropy encoder for encoding the difference picture thus obtained, possibly transformed and quantized. According to a variant, a predictor is subtracted from each block of R1, where the predictor is a spatially co-located block in picture R2 (possibly rescaled) or a motion-compensated block in R2 (possibly rescaled) or is derived from a spatially adjacent block in R1 (spatial intra prediction). A residual is thus obtained, which is further entropy-coded after possible transformation and quantization. In this case, what is encoded in S_diff is not a simple pixel-by-pixel difference between R1 and R2. Information INFO identifying the reconstructed reference picture R2 used to encode the reconstructed reference picture R1 is also encoded in S_diff. The encoder ENC2 is compatible with scalable video coders such as SVC or SHVC, for example. The invention is not limited to the scalable coder used. The Scalable Video Codec standard defines a layer_id indicator that separates/distinguishes AUs belonging to one layer (BL) from AUs belonging to another enhancement layer. According to a specific embodiment, AUs from ENC0 are encoded using a given layer_id that is different from the layer_id used to encode AUs from ENC2. AUs from ENC1 and AUs from ENC2 have the same layer_id. According to an advantageous embodiment, ENC1 and ENC2 can be the same coding module.

Figure 10 illustrates a multi-layer decoder according to a specific and non-limiting embodiment. The first stream S0 is decoded using a first decoder DEC0 that is a single-layer decoder, for example, a decoder compatible with MPEG2, H.264 or HEVC. The present invention is not limited to the single-layer decoder used. Decoder DEC0 reconstructs the picture from the first stream S0, specifically from the reference picture R2 stored in DPB0. The second decoder DEC1 is used to reconstruct the picture from the second stream S1. The present invention is not limited to the decoder used. Decoder DEC2 decodes (step 10) information INFO for identifying the reconstructed reference picture R2 in DPB0 from the stream S_diff. Decoder DEC2 is compatible with scalable video decoders such as SVC or SHVC, for example. The present invention is not limited to the scalable decoder used. Decoder DEC2 further reconstructs (step S12) the special reference picture R1' from the reconstructed reference picture R2, which may have been rescaled in time alignment, and from the data decoded from S_diff (e.g., residual, coding mode). According to a particular embodiment, the decoder DEC2 comprises an entropy decoder for decoding the residual from S_diff and an adder for adding the residual to a predictor derived from a possibly rescaled co-located or motion compensated block in R2 or from reconstructed samples in R1 ' (intra-picture prediction). The special reference picture R1 ' is then placed in DPB1.

According to an advantageous embodiment, DEC1 and DEC2 may be the same decoding module.

Figure 11 illustrates a multi-layer stream according to a specific and non-limiting embodiment. In the figure, dotted lines represent picture dependencies. AU1 and AU2 with layer_id = Layer_A are received and decoded. Reference pictures b1 and b2 are reconstructed based on the decoded AUs and stored in DPB_A of Layer_A. Upon switching, AUs S_diff1 and S_diff2 with layer_id = Layer_B are received and decoded. Decoder DEC2 then reconstructs special reference pictures e'1 and e'2 based on the data decoded from S_diff1 and S_diff2 and also based on b1 and b2 identified by information info_1 and info_2 decoded from S_diff1 and S_diff2, respectively. Special reference pictures e'1 and e'2, which are time-aligned with b1 and b2, respectively, are stored in DPB_B of Layer_B. AU3 is then received and decoded. Picture e3 is reconstructed from this decoded AU3 and also from the special reference pictures e'1 and e'2. The reconstructed picture e3 is stored in DPB_B because e3 is used as a reference picture for the reconstruction of e4. AU4 is received and decoded. Picture e4 is reconstructed from the decoded AU4 and also from the special reference picture e'2 and the reconstructed reference picture e3. The subsequent AU5 and AU6 are received and decoded. Corresponding pictures e5 and e6 are reconstructed from the decoded AU5 and AU6. If the reconstructed pictures are used as reference pictures, DPB_B may be updated by adding e5 and e6. e'1 is preferably an approximation of e1, one of the reconstructed reference pictures used when encoding e3. e'2 is preferably an approximation of e2, one of the reconstructed reference pictures used when encoding e3 and e4. Advantageously, a flag f13 is encoded (respectively, decoded) in, for example, a VPS or SEI, indicating that subsequently encoded (respectively, decoded) decoded pictures with a given layer_id do not use any inter-layer prediction. More precisely, pictures encoded (respectively, decoded) after the flag do not use inter-layer prediction.

Encoding the pictures of the first and second sequences and the special reference pictures into a multi-layer stream makes it possible to reconstruct two reference pictures (b1 and e'1, or b2 and e'2) that are time-aligned, for example, with the same POC. In practice, different DPBs are used in the multi-layer approach. In particular, one DPB is used for each layer. Therefore, the time-aligned reconstructed reference pictures are stored in different DPBs. Due to the dependency of the layers, decoding a multi-layer stream usually requires decoding the layers of level N before decoding the layers of level N+1, where N is an integer. This dependency between layers is not compatible with stream switching applications. Advantageously, encoding the flag f13 introduces independence between the layers and thus presents scalable coding/decoding suitable for stream switching applications.

The encoding and decoding methods according to the present invention enable flexible stream switching with a small bitrate overhead only when switching occurs. These methods are suitable for any GOP structure, any number of reconstructed reference pictures, and even when the decoding order is different from the display order.

An example of the syntax within the framework of the SHVC coding standard for an S_diff stream is provided below.

slice_type The name of the slice_type 0 B (B slice) 1 P (P slice) 2 I(I slice) 3 SRP (SRP slice)

Add slice_type to identify the slice of a special reference picture.

sign_diff_pic equal to 1 indicates that the residual should be added to the prediction, otherwise it indicates that the residual should be subtracted from the prediction.

pic_order_cnt_diffpic_lsb specifies the picture order count modulo MaxPicOrderCntLsb for this special reference picture. Intra BL prediction will then use reference pictures in the DPB with the same pic_order_cnt. The length of the pic_order_cnt_lsb syntax element is log2_max_pic_order_cnt_lsb_minus4 + 4 bits. The value of pic_order_cnt_diffpic_lsb shall be in the range of 0 to MaxPicOrderCntLsb–1, inclusive. When pic_order_cnt_diffpic_lsb is not present, pic_order_cnt_diffpic_lsb is inferred to be equal to 0.

delta_poc_msb_diffpic_cycle_lt is used to determine the value of the most significant bit of the picture sequence count value of the long-term reconstructed reference picture in the DPB used to reconstruct this special reference picture. When delta_poc_msb_cycle_lt is not present, it is inferred to be equal to 0.

num_layer_id_diffpic_apply indicates the num_layer_id of the reconstructed reference pictures used for decoding this special reference picture.

Example of syntax (vps_extension):

diff_pic_flag_enabled equal to 1 indicates that inter_layer_pred_for_non_diff_picture_flag is coded.

inter_layer_pred_for_non_diff_picture_flag equal to 1 indicates that any subsequent pictures of type I, P, or B do not use inter-layer prediction, but pictures of type SRP may use inter-layer prediction, but not temporal intra-layer prediction.

According to the present invention and the video encoder and decoder shown in Figures 9 and 10, for example, are implemented as various forms of hardware, software, firmware, special processors or combinations thereof. Preferably, the principle can be implemented as a combination of hardware and software. Moreover, the software is preferably implemented as an application program tangibly implemented on a program storage device. The application program can be uploaded to a machine comprising any suitable architecture and executed by the machine. Preferably, the machine is implemented on a computer platform with hardware such as one or more central processing units (CPUs), random access memories (RAMs) and input/output (I/O) interfaces. The computer platform also includes an operating system and microinstruction codes. The various processes and functions described herein can be a part of the microinstruction code or a part (or a combination thereof) of the application program via the operating system. In addition, various other peripheral devices can be connected to the computer platform, such as other data storage devices and printing equipment.

According to a variant, the encoding and decoding device according to the invention is implemented according to a purely hardware implementation, for example in the form of a dedicated component such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) or a VLSI (Very Large Scale Integrated Circuit), or in the form of several electronic components integrated in the device, or even in the form of a mixture of hardware and software elements.

Claims (15)

1. A decoding method, comprising: A reference frame is reconstructed based on an additional reference frame from the decoder frame buffer of the first layer of the scalable multi-layer stream and data decoded from the second layer of the scalable multi-layer stream, and the reconstructed reference frame is stored in the decoder frame buffer of the second layer, wherein the reconstructed reference frame is indicated not to be displayed. The subsequent decoding of the image indicating the second layer was performed without using any inter-layer prediction flags; and At least the frame blocks of the subsequently decoded frame are reconstructed based on the reconstructed reference frame. 2. The decoding method according to claim 1 further comprises: Information used to identify the additional reference image is decoded from the second layer. 3. The decoding method according to any one of claims 1 to 2, wherein the reconstructed reference frame and the other reference frame are time-aligned. 4. An encoding method, comprising: Based on an additional reference frame of the decoder frame buffer of the first layer of the scalable multi-layer stream, the reference frame of the decoder frame buffer of the second layer of the scalable multi-layer stream is encoded in the second layer, wherein the reference frame is indicated to not be displayed. The subsequent encoding of the second layer was not coded using any inter-layer prediction flags; and At least the frame blocks of the subsequently encoded frame are encoded based on the reference frame. 5. The encoding method according to claim 4, further comprising: In the second layer, information used to identify the additional reference screen is encoded. 6. The encoding method according to any one of claims 4 to 5, wherein the reference frame and the other reference frame are time-aligned. 7. A decoder, comprising: Components for reconstructing a reference frame based on an additional reference frame from a decoder frame buffer of the first layer of a scalable multi-layer stream and data decoded from the second layer of the scalable multi-layer stream, and storing the reconstructed reference frame in a decoder frame buffer of the second layer, wherein the reconstructed reference frame is indicated not to be displayed. A component for decoding images indicating subsequent decoding of the second layer that do not use any inter-layer prediction flags; and A component for reconstructing, at least based on, the frame blocks of the subsequently decoded frame, based on at least the reconstructed reference frame. 8. The decoder according to claim 7, further comprising: A component used to decode information from the second layer for identifying the additional reference image. 9. The decoder according to any one of claims 7 to 8, wherein the reconstructed reference frame and the other reference frame are time-aligned. 10. An encoder comprising: A component for encoding a reference screen of the decoder screen buffer of the second layer in the scalable multi-layer stream, based on an additional reference screen of the decoder screen buffer of the first layer, wherein the reference screen is indicated to not be displayed. A component for encoding a frame indicating subsequent encoding of the second layer that does not use any inter-layer prediction flags; and A component for encoding at least a block of the subsequently encoded frame based on the reference frame. 11. The encoder of claim 10, further comprising: A component for encoding information used to identify the additional reference screen in the second layer. 12. The encoder according to any one of claims 10 to 11, wherein the reference frame and the additional reference frame are time-aligned. 13. A method for transmitting a stream, comprising: Encoded data of a reference frame of the decoder frame buffer of the second layer of the scalable multi-layer stream is transmitted in the second layer. The reference frame is encoded based on an additional reference frame of the decoder frame buffer of the first layer of the scalable multi-layer stream, wherein the reference frame is indicated to not be displayed. The transmission indicates that the subsequent encoded image of the second layer does not use any inter-layer prediction flags; and Encoded data of frame blocks of the subsequently encoded frame are transmitted, the frame blocks being encoded at least according to the reference frame. 14. The method of transporting a stream according to claim 13, further comprising transmitting information for identifying the additional reference frame in the second layer. 15. The method of transporting a stream according to any one of claims 13 to 14, wherein the reference frame and the additional reference frame are time-aligned.