HK1163404B

HK1163404B - Method and device for displaying a sequence of pictures

Info

Publication number: HK1163404B
Application number: HK12103286.3A
Authority: HK
Inventors: Vincent Bottreau; Christel Chamaret; Jean-Claude Chevet
Original assignee: 汤姆森特许公司
Priority date: 2009-03-02
Filing date: 2010-02-26
Publication date: 2015-07-24

Description

Method and device for displaying picture sequence

Technical Field

The invention relates to a method and an apparatus for displaying a sequence of pictures. More particularly, the invention relates to a method of displaying a sequence of pictures in the form of a multi-layer stream and to a display device for displaying such a sequence of pictures.

Background

In order to enable a user to change the program displayed on his terminal, i.e. to change the display of the first picture sequence to the display of the second picture sequence, it is known to add a Random Access Point (RAP) to the stream representing the second picture sequence. In order to increase the access speed from a first stream representing a first picture sequence to a second stream, it is necessary to allocate random access points to the second stream in a dense manner. Such a solution has the disadvantage of increasing the bit rate of the second stream inserted in these random access points.

Furthermore, it is known to represent a sequence of pictures in the form of a multilayer stream comprising a base layer representing pictures of the sequence called base layer, having a first resolution and/or quality, and at least one enhancement layer representing pictures of said sequence called enhancement layer, having a second resolution and/or quality. In the particular case where the user wants the second picture sequence displayed on his terminal to come in the form of a multi-layer stream, it is known to add a random access point to both the base layer and the enhancement layer. Generally, random access points are allocated in the base layer in a more frequent manner, or in a similar manner as the enhancement layer, in order to limit the increase in bit rate of the multi-layer stream and to speed up the display of the second stream. For example, it is suggested to insert random access points into the base layer once every 2 seconds, or even once every 500 milliseconds in case fast access is required, while random access points can be inserted into the enhancement layer on average at least once every 5 seconds.

When the user indicates to the terminal that the second picture sequence is desired to be displayed, the terminal waits for the arrival of the random access point of the second picture sequence. Terminals typically contain a decoding device linked to a display device (e.g., a set-top box (STB) linked to an SDTV or HDTV (high definition television) screen). In case the base layer contains more frequent random access points than the enhancement layer, the terminal therefore decodes the data of the base layer until the arrival and decoding of the random access point of the enhancement layer. The decoding apparatus reconstructs a corresponding picture of the base layer from the thus decoded data. These pictures are then displayed on a display device. After decoding the random access point of the enhanced layer, the terminal decodes the enhanced layer to reconstruct the picture of the enhanced layer. These pictures are then displayed on a display device.

However, such a display is unsatisfactory from a visual point of view, since the terminal changes the display of the picture of the base layer to the display of the picture of the enhancement layer at the moment of decoding the random access point of the enhancement layer, which may correspond to a sudden jump in quality and/or content.

Disclosure of Invention

The object of the present invention is to remedy at least one of the disadvantages of the prior art.

To this end, the invention relates to a method for displaying a sequence of pictures arriving in the form of a multilayer stream comprising a base layer representing pictures of a sequence with a first resolution and/or a first quality, called pictures of the base layer, and at least one enhancement layer representing pictures of a sequence with a second resolution and/or a second quality, called pictures of the enhancement layer. The base layer and the enhancement layer contain random access points. The method comprises the following steps:

-decoding the base layer from the random access point of the base layer and at least up to the decoding of the random access point of the enhancement layer in order to reconstruct the picture of the base layer;

-displaying a reconstructed picture of the base layer;

-decoding the enhancement layer from the random access point of the enhancement layer in order to reconstruct the pictures of the enhancement layer;

-displaying the reconstructed picture of the enhancement layer.

Advantageously, the method further comprises a processing step of processing the pictures of the enhancement layer before displaying them in such a way that the variation of the picture content and/or of its quality is gradual between the pictures of the base layer and the pictures of the enhancement layer corresponding to the time of the random access point of the enhancement layer.

According to a particular embodiment, said processing step is a filtering step of filtering each picture of the enhancement layer at an intermediate resolution and/or quality which increases progressively from the first resolution and/or first quality to the second resolution and/or second quality over a predetermined time interval.

According to a particular feature, said filtering step is a sub-sampling step.

According to another particular embodiment, the processing step is a reframing step of reframing each picture of the enhancement layer through a cropping window, the size of said cropping window increasing gradually over a predetermined time interval between the size of the cropping window associated with the picture of the base layer and the size of the picture of the enhancement layer.

According to a particular feature, the processing step further comprises a sub-sampling step.

The invention also relates to a device for displaying a sequence of pictures arriving in the form of a multilayer stream comprising a base layer representing pictures of the sequence at a first resolution and/or a first quality, called pictures of the base layer, and at least one enhancement layer representing pictures of the sequence at a second resolution and/or a second quality, called pictures of the enhancement layer, said base layer and enhancement layer comprising random access points. The apparatus includes:

-a decoding module for decoding the base layer from its random access point and at least up to the decoding of the random access point of the enhancement layer in order to reconstruct the pictures of the base layer, and for decoding the enhancement layer from its random access point in order to reconstruct the pictures of the enhancement layer; and

-a display module for displaying reconstructed pictures of the base layer and the enhancement layer.

Advantageously, the device further comprises a processing module for processing the pictures of the enhancement layer before their display by said display module in such a way that the variation of the picture content and/or of the quality thereof is gradual between the pictures of the base layer and the pictures of the enhancement layer corresponding to the time of the random access point of the enhancement layer.

Drawings

The invention will be better understood and exemplified by means of non-limiting examples and advantageous implementations with reference to the attached drawings, in which:

figure 1 shows pictures of the enhancement layer and pictures of the base layer of a multi-layer stream and a cropping window;

fig. 2 shows a block diagram of a display method according to a first embodiment of the invention;

fig. 3 illustrates a display method according to a first embodiment of the invention;

fig. 4 shows a block diagram of a display method according to a second embodiment of the invention;

fig. 5 illustrates a display method according to a second embodiment of the invention;

fig. 6 shows a block diagram of a display method according to a third embodiment of the invention;

fig. 7 shows a block diagram of a display method according to a fourth embodiment of the invention;

fig. 8 illustrates a method of changing a picture sequence using the display method according to the invention;

FIG. 9 shows a block diagram of a method for changing a picture sequence using the display method according to the invention; and

fig. 10 shows a display device according to the invention.

Detailed Description

The invention relates to a method of displaying a multi-layer stream comprising a sequence of representative pictures of at least two layers. According to the invention, the layers of a multi-layer stream can be coded completely independently of each other. The multi-layer stream may be a scalable stream comprising a base layer representing pictures of a sequence with a first resolution and/or quality, called pictures of the base layer, and at least one enhancement layer representing pictures of a sequence with a second resolution and/or quality, called pictures of the enhancement layer. However, the invention is by no means limited to this configuration, and the term "multi-laminar flow" should be understood in its broadest sense. It also includes simulcast streams (simulcast streams) and multi-view streams.

A scalable stream is, for example, a stream compliant with the SVC video coding standard described in the JVT-AC205 file published in 10 months 2008 and entitled "Joint Draft ITU-T Rec.H.264| ISO/IEC14496-10/Amd.3Scalable video coding". However, the invention is in no way limited to this standard and can be used with any multi-layer stream that improves the visual comfort of the user during the display of a sequence of pictures, in particular when changing the display of pictures of the base layer to the display of pictures of the enhancement layer. Pictures of the base layer have a size of (W, H), where W is the width and H is the height, and pictures of the enhancement layer have a size of (W, H), where W is the width and H is the height. Even if they have a lower resolution, the picture content of the pictures of the base layer may be the same as the picture content of the pictures of the enhancement layer. This is the case when the picture of the base layer is generated by sub-sampling the picture of the enhancement layer. However, as shown in fig. 1, the screen contents may be different. This is particularly the case when the pictures of the base layer are generated by reframing (reframe) and then possibly sub-sampling the pictures of the enhancement layer. In this case, the picture content of the picture of the base layer corresponds to the picture content of only a portion of the corresponding picture of the enhancement layer. This portion is bounded by a cropping window of size (w ', h'), where w 'is the width of the cropping window and h' is its height.

According to a first embodiment described with reference to fig. 2 and 3

In step 100, the base layer is decoded from one of its random access points (corresponding to time T1 in fig. 2) and at least until the decoding of the random access point of the enhancement layer (corresponding to time T2 in fig. 2) in order to reconstruct the pictures of the base layer. If the multi-layer stream conforms to the SVC standard, the pictures of the base layer are reconstructed according to the H.264/MPEG-4AVC decoding method described in the JVT-AC205 file entitled "ITU-T Rec.H.264| ISO/IEC14496-10/Amd.3Scalable video coding".

In step 110, a reconstructed picture of the base layer is displayed. This step may require spatial filtering of the pictures of the base layer in order to make them of the resolution of the display device. This is particularly the case if the pictures of the base layer have a 720p format but have to be displayed on the screen in a 1080p High Definition (HD) format. In this particular case, the picture of the base layer is upsampled before display.

In step 120, if the random access point of the enhancement layer is decoded, the method is continued in step 130, otherwise it is restarted in step 100.

In step 130, the enhancement layer is decoded from the random access point (corresponding to time T2 in fig. 2) to reconstruct the picture of the enhancement layer. If the multi-layer stream is a scalable stream compliant with the SVC standard, the pictures of the enhancement layer are reconstructed according to the SVC decoding method described in appendix G of the JVT-AC205 file entitled "Joint Draft ITU-T Rec.H.264| ISO/IEC14496-10/Amd.3Scalable video coding".

In step 140, the reconstructed picture of the enhancement layer is processed. They are sub-sampled gradually over a predetermined time interval, e.g., 500ms (milliseconds). In fig. 2, the progression processing is performed between time T2 and time T3. Time T2 corresponds to the decoding of the first random access point of the enhancement layer. Time T3 may or may not correspond to the decoding of another random access point of the enhancement layer. Time T3 defines the processing time of the pictures of the enhancement layer relative to time T2, thus defining the duration of the progressive transition time between the display of the pictures of the base layer before time T2 or possibly at time T2 and the display of the pictures of the enhancement layer after time T3 or possibly at time T3. This step 140 causes display to occur gradually between the pictures of the base layer at or before time T2 of the enhancement layer random access point and the pictures of the enhancement layer at or after time T3. Thus, the first enhancement layer picture reconstructed at time T2 is subsampled by the size (w, h) of the picture of the base layer. The following picture (pullingpicture) of the enhancement layer is sub-sampled by the size (w + dw, h + dh), i.e. by a size slightly larger than the size of the picture of the base layer. For this purpose, a Lanczos (Lanczos) polyphase filter can be used in, for example, 16 stages. The method is repeated for each reconstructed picture of the enhancement layer by increasing the picture size each time after sub-sampling until the size (H, W) of the picture of the enhancement layer is reached. On each new picture of the enhancement layer, the sub-sampled picture size may be increased by a height dh pixels and a width dw pixels, e.g., dh-4 and dw-4. Therefore, the quality of the picture displayed between the picture of the base layer and the picture of the enhancement layer is gradually increased over the time interval between T2 and T3, rather than being abruptly changed as if the display of the picture of the base layer was directly changed to the display of the picture of the enhancement layer over time T2.

In step 150, the processed picture of the enhancement layer is displayed. This step may require spatial filtering of the processed pictures of the enhancement layer in order to bring them to the resolution of the display device. This is especially the case if the processed pictures of the enhancement layer have a lower resolution than the resolution of the screen on which they have to be displayed. In this particular case, the processed pictures of the enhancement layer (i.e., after sub-sampling) are upsampled before they are displayed.

This embodiment is preferably used when the picture content of the pictures of the base layer and the enhancement layer are the same, i.e. when no re-framing tool is used to generate the base layer picture from the pictures of the enhancement layer as shown in fig. 1.

According to a second embodiment described with reference to fig. 4 and 5.

In step 100, the base layer is decoded from one of its random access points (corresponding to time T1 in fig. 2) and at least until the decoding of the random access point of the enhancement layer (corresponding to time T2 in fig. 2) in order to reconstruct the pictures of the base layer.

In step 120, if the random access point of the enhancement layer is decoded, the method continues in step 130, otherwise it restarts in step 100.

In step 130, the enhancement layer is decoded from the random access point (corresponding to time T2 in fig. 2) to reconstruct the picture of the enhancement layer.

In step 140, the reconstructed picture of the enhancement layer is processed. They are gradually reframed over a predetermined time interval, e.g., 500ms (milliseconds). In fig. 5, the progression processing is performed between time T2 and time T3. Time T2 corresponds to the decoding of the first random access point of the enhancement layer. Time T3 may or may not correspond to the decoding of another random access point of the enhancement layer. Time T3 defines the processing time of the pictures of the enhancement layer relative to time T2, thus defining the duration of the progressive transition time between the display of the pictures of the base layer before time T2 or possibly at time T2 and the display of the pictures of the enhancement layer after time T3 or possibly at time T3. This step 140 causes display to occur gradually between the pictures of the base layer at or before time T2 of the enhancement layer random access point and the pictures of the enhancement layer at or after time T3. Thus, the first enhancement picture reconstructed at time T2 is reframed at the size (w, h) of the picture of the base layer. In this case, the cropping window of size (w, h) is positioned in the picture of the enhancement layer in such a way that its content is identical to, or at least very close to, the picture content of the picture of the corresponding base layer. According to a particular embodiment, the position and size of the cropping window associated with the picture of the enhancement layer at time T2 is decoded from the multilayer stream in which the pictures of the enhancement layer are transmitted. This is especially true if the multi-layer stream is a scalable stream compliant with the SVC video coding standard.

The following pictures of the enhancement layer are reframed at the (w + dw, h + dh) size, i.e. at a size slightly larger than the size of the pictures of the base layer. This method is repeated for each reconstructed picture of the enhancement layer by increasing the size of the cropping window each time until the size (H, W) of the picture of the enhancement layer is reached. On each new picture of the enhancement layer, the clipping window size may be increased by a height dh pixels and a width dw pixels, e.g., dh-4 and dw-4. Thus, the additional picture content between the pictures of the base layer and the pictures of the enhancement layer is gradually added over the time interval between T2 and T3, rather than being suddenly added as if the display of the pictures of the base layer were directly changed to the display of the pictures of the enhancement layer over time T2.

In step 150, the re-framed picture of the enhancement layer is displayed. This step may require spatial filtering of the re-framed pictures of the enhancement layer in order to bring them to the resolution of the display device. This is especially the case if the re-framed pictures of the enhancement layer have a lower resolution than the resolution of the screen on which they have to be displayed. In this special case, the re-framed picture of the enhancement layer is upsampled before display.

The third embodiment described with reference to fig. 6 combines the first two embodiments

In step 140, the pictures of the enhancement layer are processed. They are gradually reframed and sub-sampled over a predetermined time interval, e.g., 500ms (milliseconds). In fig. 2, the progression processing is performed between time T2 and time T3. Time T2 corresponds to the decoding of the first random access point of the enhancement layer. Time T3 may or may not correspond to the decoding of another random access point of the enhancement layer. Time T3 defines the processing time of pictures of the enhancement layer relative to T2, thus defining the duration of the progressive transition time between the display of pictures of the base layer before time T2 or possibly at time T2 and the display of pictures of the enhancement layer after time T3 or possibly at time T3. This step 140 causes display to occur gradually between the pictures of the base layer at or before time T2 of the enhancement layer random access point and the pictures of the enhancement layer at or after time T3. Thus, the first enhancement layer picture reconstructed at time T2 is re-boxed at the size (w ', h') of the cropping window used to generate the picture of the base layer according to the method illustrated in fig. 1. In this case, the cropping window of size (w ', h') is positioned in the picture of the enhancement layer in such a way that its content is identical or at least very close to the picture content of the corresponding base layer. According to a particular embodiment, the position and size of the cropping window associated with the picture of the enhancement layer at time T2 is decoded from the multilayer stream in which the pictures of the enhancement layer are transmitted. This is especially true if the multi-layer stream is a scalable stream compliant with the SVC video coding standard.

The pictures of the enhancement layer framed in this way are then resampled by the size (w, h) of the pictures of the base layer.

The frame is refetched by the (w '+ dw, h' + dh) size and then the following picture of the enhancement layer is sub-sampled by the (w + dw, h + dh) size. The method is repeated for each reconstructed picture of the enhancement layer by increasing the size of the cropping window and the sub-sampled picture each time until the size (H, W) of the picture of the enhancement layer is reached. Thus, the additional picture content between the pictures of the base layer and the pictures of the enhancement layer is gradually added over the time interval between T2 and T3, rather than being suddenly added as if the display of the pictures of the base layer were directly changed to the display of the pictures of the enhancement layer over time T2. Also, the quality of the picture displayed between the picture of the base layer and the picture of the enhancement layer is gradually increased over the time interval between T2 and T3, rather than being abruptly changed as if the display of the picture of the base layer was directly changed to the display of the picture of the enhancement layer over time T2.

In step 150, the processed picture of the enhancement layer is displayed. This step may require spatial filtering of the processed pictures of the enhancement layer in order to bring them to the resolution of the display device. This is especially the case if the processed pictures of the enhancement layer have a lower resolution than the resolution of the screen on which they have to be displayed. In this special case, the re-framed picture of the enhancement layer is upsampled before display.

These last two embodiments are preferably used when the picture content of the pictures of the base layer and the enhancement layer are different, in particular when the pictures of the base layer are generated only from a part of the pictures of the enhancement layer, said part being delimited in the pictures of the enhancement layer by a cropping window, as shown in fig. 2.

A fourth embodiment is described with reference to fig. 7.

In step 140, the pictures of the enhancement layer are processed. They are gradually filtered over a predetermined time interval, e.g., 500ms (milliseconds). In fig. 2, the progression processing is performed between time T2 and time T3. Time T2 corresponds to the decoding of the first random access point of the enhancement layer. Time T3 may or may not correspond to the decoding of another random access point of the enhancement layer. Time T3 defines the processing time of pictures of the enhancement layer relative to T2, thus defining the duration of the progressive transition time between the display of pictures of the base layer before time T2 or possibly at time T2 and the display of pictures of the enhancement layer after time T3 or possibly at time T3. This step 140 causes display to occur gradually between the pictures of the base layer at or before time T2 of the enhancement layer random access point and the pictures of the enhancement layer at or after time T3. Thus, the first enhancement picture reconstructed at time T2 is filtered to obtain a quality close to that of the picture of the base layer. This filtering may be performed by sub-sampling at any lower resolution (W ", H") followed by up-sampling at resolution (W, H).

The following picture of the enhancement layer is filtered to obtain an intermediate quality between the quality of the picture of the base layer and the quality of the picture of the enhancement layer. This filtering may be performed by sub-sampling at a lower resolution (W "+ dw", H "+ dh") followed by up-sampling at resolution (W, H). The method is repeated for each picture of the reconstructed enhancement layer by increasing the quality of the filtered picture each time until the quality of the picture of the enhancement layer is reached. Therefore, the picture quality between the picture of the base layer and the picture of the enhancement layer is gradually improved over the time interval between T2 and T3, rather than being suddenly improved as if the display of the picture of the base layer was directly changed to the display of the picture of the enhancement layer over time T2.

This embodiment is preferably used when the pictures of the base layer and the enhancement layer have the same resolution, i.e. when (W, H) is (W, H), but when the quality/fidelity of the pictures of the base layer compared to the pictures of the original sequence (also referred to as source sequence) is lower than the quality/fidelity of the pictures of the enhancement layer. The progressive processing of the pictures of the enhancement layer during the transition interval between T2 and T3 proceeds in such a way that the quality of the processed pictures gradually increases between the quality of the pictures of the base layer at a time T2 or at a time T2 and the quality of the pictures of the enhancement layer at a time T3 or after.

These four embodiments advantageously provide improved visual comfort on the display. Indeed, the transition between the display of pictures of the base layer and the enhancement layer is gradual in terms of content and/or quality, i.e. fidelity of the pictures of the source sequence. Note that for these four embodiments, the picture of the corresponding base layer or the picture of the processed enhancement layer may be displayed at time T2. Also, the picture of the enhanced layer after processing or the corresponding picture of the enhanced layer may be displayed at time T3.

The method according to the invention is advantageously used to improve the display from a visual point of view in case the user wants to change the display of the first sequence to the display of the second sequence. Thus, referring to fig. 8 and 9, the user indicates in step 70 that he wants to change the display of the sequence of picture a to the display of the sequence of picture B represented in a multi-layer form.

In step 80, if a random access point is decoded for the base layer, the method continues in step 100, otherwise, it continues in step 90.

In step 90, a predetermined picture is displayed on the screen as long as the random access point of the base layer is not decoded. Steps 100 to 150 are the same as steps 100 to 150 described above with reference to one of these embodiments and will not be described further here.

The predetermined picture is, for example, a black image or still the last picture of the sequence a displayed before receiving a signal indicating that the user wants to change the sequence.

The invention also relates to a display device 20 as shown in fig. 10. The display device 20 comprises an input 200 capable of receiving the picture sequences seq a, seq B and seq C. On this input, the display device 20 is also able to receive a signal sig indicating which picture sequence the user wants to see displayed. The display device 20 further comprises a decoding module 210 connected to the input 200 for decoding the multi-layer stream to reconstruct pictures of the base layer and pictures of the enhancement layer. In particular, the decoding module 210 is adapted to implement the steps 100, 120 and 130 of the method according to one of the embodiments of the invention. Furthermore, the display device 20 comprises a processing module 220 connected to the decoding module 210, the processing module 220 being capable of processing the pictures of the enhancement layer before displaying them according to the steps of the method according to one of the embodiments of the present invention. The processing module 220 is connected to a display module 230, and the display module 230 is adapted to display the picture of the base layer and the picture of the enhancement layer on a screen of the display module 230. The display module 230 is also adapted to spatially filter the pictures received from the processing module 220, if necessary, in order to adapt their size to the size of the screen. In general, the display module 230 upsamples the pictures received from the processing module 220.

Claims

1. Method for displaying a sequence of pictures coded into a multilayer stream, said multilayer stream comprising a base layer representing pictures of said sequence at a first resolution and/or a first quality, called pictures of the base layer, and at least one enhancement layer representing pictures of said sequence at a second resolution and/or a second quality, called pictures of the enhancement layer, said base layer and enhancement layer comprising random access points, said method comprising the steps of:

decoding (100) the base layer from the random access point of the base layer and at least up to the random access point of the enhancement layer in order to reconstruct the picture of the base layer;

-displaying (110) the reconstructed picture of the base layer;

-decoding (130) the enhancement layer from said random access point of the enhancement layer in order to reconstruct a picture of the enhancement layer;

-displaying (150) the reconstructed picture of an enhancement layer,

said method is characterized in that it further comprises a step (140) of reframing said pictures of the enhancement layer by means of a cropping window whose size increases progressively between the size of the cropping window associated with the picture of the base layer and the size of the picture of the enhancement layer over a time interval whose starting point is a random access point of the enhancement layer, before displaying said pictures of the enhancement layer.

2. The method of claim 1, wherein the reframing step (140) further comprises a sub-sampling step.

3. A display device coded into a sequence of pictures of a multilayer stream, said multilayer stream comprising a base layer representing pictures of said sequence at a first resolution and/or a first quality, called pictures of the base layer, and at least one enhancement layer representing pictures of said sequence at a second resolution and/or a second quality, called pictures of the enhancement layer, said base layer and enhancement layer comprising random access points, said device comprising:

-a decoding module (210) for decoding the base layer from a random access point of the base layer and at least up to a random access point of the enhancement layer in order to reconstruct pictures of the base layer, and decoding the enhancement layer from said random access point of the enhancement layer in order to reconstruct pictures of the enhancement layer;

a display module (230) for displaying the reconstructed pictures of a base layer and an enhancement layer,

the device is characterized in that it further comprises a processing module (220) for reframing said pictures of the enhancement layer through a cropping window before their display by said display module, the size of said cropping window increasing progressively between the size of the cropping window associated with the picture of the base layer and the size of the picture of the enhancement layer over a time interval whose starting point is the random access point of the enhancement layer.