GB2480819A

GB2480819A - Error resilience for multimedia transmission

Info

Publication number: GB2480819A
Application number: GB1009128A
Authority: GB
Inventors: James Chung-How; Andrew Nix; David Bull
Original assignee: GLOBAL PROVISION Ltd; PROVISION COMM TECHNOLOGIES Ltd; GI Provision Ltd
Current assignee: GLOBAL PROVISION LIMITED
Priority date: 2010-06-01
Filing date: 2010-06-01
Publication date: 2011-12-07
Anticipated expiration: 2030-06-01
Also published as: GB2480819B; GB201009128D0

Abstract

A method of transmitting a multimedia datastream over a transmission channel, the method comprising: receiving a multimedia datastream; slicing 66 the received datastream into a plurality of multimedia slices having a predetermined slice size; encoding 68 (and 40, Figure 2) the multimedia slices into first data packets of a first predetermined size; dividing 72 (and 42, Figure 2) each of the first data packets into a respective integral number of second data packets of a second predetermined size; aggregating 74 the second data packets into a stream of third data packets of a third predetermined size, each third data packet containing all of the second data packets relating to a single one of the first data packets; and transmitting 78 the series of third data packets over a transmission channel. The method may further comprise the step of encoding 70 the first data packets into respective encoded first data packets of a predetermined size before dividing each of the first data packets, each such encoded first data packet including all of the first data packets relating to a single one of the multimedia slices. The method may further comprise a step of encapsulating the third data packets 76 into respective encapsulated third data packets of a predetermined size before transmission of the third data packets over the transmission channel. The method may be embodied, for example, in a server of a video transmission system. The predetermined slice size may be chosen such that the predetermined size of a transmitted third data packet is not greater that a permitted maximum size for the transmission channel (MTU, maximum transmission unit). Aggregation of the second data packets into third data packets may include applying a forward error correction scheme to the second data packets and including forward error correction data in the third data packets.

Description

ERROR RESILIENCE MECHANISM FOR A VIDEO TRANSMISSION

SYSTEM

The present invention relates to an error resilience mechanism for a video transmission system. In particular the present invention relates to an error resilience mechanism which implements intelligent video encoding and packetisation to minimise the effect of packet data loss in a video transmission system.

When digital media data, such as video media data, are being transmitted from a server to an end user using a streaming application, the maximisation of quality of the output media presented to the end user is a high priority. However, video media data can be sizeable and require compression to enable more effective and efficient data delivery.

A typical video transmission system 10 is shown in Figure 1. The system 10 consists of a transmitting server 12 and a receiving client 20. The server 12 comprises a video encoder 12, in this case H.264 which is a standard for video compression; a formatting multiplexer 14, and a transmitter 18. The client 20 comprises a receiver 22, a formatting demultiplexer 24 and a video decoder 26.

The encoder 12 receives input video media data and generates a compressed video bit stream consisting of variable size chunks at the application layer of the server 12. The variable size chunks of compressed video are then packaged by formatting multiplexer 14 which aggregates and/or fragments them into a suitable container format, in this case an MPEG-2 Transport Stream as specified in ISO-IEC 138 18-1.

The Transport Stream is then encapsulated by subsequent protocol layers such as the transport and network layers before being provided to transmitter 18 for transmission over the network, which may be unreliable.

I

The receiving client 20 receives the transmitted data at receiver 22 which is then formatted and demultiplexed by formatting demultiplexer 24 into a bit stream of variable size slices which is provided to the video decoder 26 to be returned to video media data for provision to a display device (not shown). The transport stream data is generally transmitted over a network by the physical layer in the form of data packets known as Physical layer Protocol Data Units (PPDUs). If the network is unreliable, PPDUs can be lost or received with errors. Therefore, the video bit stream obtained by the receiver 20 may be incomplete or incorrect, It is desirable to limit the effect of a missing or corrupted PPDU on the reconstructed video media data at the client 20.

Video encoders, such as video encoder 12 which is in this case H.264, use video compression algorithms. The video compression algorithms exploit the spatial and temporal redundancy between the individual pixel values within a raw video signal and produce a video bit stream that is a more compact representation of the original raw video signal. Such a video bit stream is very sensitive to loss or errors in the bit stream and distortion due to loss or errors will generally propagate spatially and temporally.

State-of-the-art video coding standards, such as the H,264 standard, generally partition the compressed video bit stream into self-contained chunks. In the H.264 standard, a slice is a portion of the bit stream that is self-contained in the sense that if the active sequence parameter set (SPS) and picture parameter set (PPS) are known, the syntax elements within a slice can be parsed from the bit stream and the values of the samples in the area of the picture that the slice represents can be decoded without the use of data from other slices, provided that the previously decoded pictures referenced by the slice are available at the decoder. Slices are typically used to limit the extent of error propagation, and thus increase robustness to loss of data. However, the robustness to loss of data also depends on how slices are fragmented and/or aggregated by the subsequent protocol layers prior to transmission.

A good system solution must aim to minimise the bandwidth utilisation of the network while at the same time providing good video quality and robustness to loss of compressed video media data.

The present invention seeks to address the problems of the prior art, According to one aspect of the present invention, there is provided a method of transmitting a multimedia datastream over a transmission channel, the method comprising receiving a multimedia datastream, slicing the received datastream into a plurality of multimedia slices having a predetermined slice size, encoding the multimedia slices into first data packets of a first predetermined size dividing each of the first data packets into a respective integral number of second data packets of a second predetermined size, aggregating the second data packets into a stream of third data packets of a third predetermined size, each third data packet containing all of the second data packets relating to a single one of the first data packets, and transmitting the series of third data packets over a transmission channel.

In one embodiment, the method further comprises the step of encoding the first data packets into respective encoded first data packets of a predetermined size before dividing each of the first data packets into a respective integral number of second data packets, each such encoded first data packet including all of the first data packets relating to a single one of the multimedia slices.

The method may further comprise the step of encapsulating the third data packets into respective encapsulated third data packets of a predetermined size, before transmitting the series of third data packets over a transmission channel.

According to a second aspect of the present invention, there is provided apparatus for transmitting a multimedia datastream over a transmission channel, the apparatus comprising an input unit operable to receive a multimedia datastream, a slicing unit operable to slice a received datastream into a plurality of multimedia slices having a predetermined slice size, a first encoder operable to encode such multimedia slices into first data packets of a first predetermined size, a divider operable to divide or fragment each such first data packet into a respective integral number of second data packets of a second predetermined size, an aggregation unit operable to aggregate such second data packets into a stream of third data packets of a third predetermined size, each third data packet containing all of the second data packets relating to a single one of the first data packets, and a transmitter operable to transmit such a series of third data packets over a transmission channel.

The apparatus may further comprise a second decoder operable to encode such first data packets into respective encoded first data packets of a predetermined size, each encoded first data packet including all of the first data packets relating to a single one of the multimedia slices.

The apparatus may further comprise an encapsulation unit operable to encapsulate the third data packets into respective encapsulated third data packets of a predetermined size.

The predetermined slice size may be chosen such that the predetermined size of a transmitted third data packet is not greater than a permitted maximum size for the transmission channel. In such a case, the predetermined size of a transmitted third data packet may be substantially equal to the permitted maximum size for the transmission channel.

Each encapsulated third data packet may include a single third data packet.

Aggregation of second data packets into third data packets may include applying a forward error correction scheme to the second data packets, and may include forward error correction data in the third data packets.

The second data packets may be grouped into blocks, and the forward error correction scheme may be applied to all of the second data packets in a block.

The forward error correction data may include forward error correction repair symbols.

It will be readily appreciated that the techniques embodying the present invention are applicable to compressed and uncompressed data streams, and are applicable to a wide range of compression algorithms and container formats.

An embodiment of the invention will now be described, by way of example only, and with reference to the accompanying drawings, in which: Figure 1 illustrates a schematic diagram of a known video transmission system: Figure 2 illustrates a schematic diagram of a video transmission system in which an error resilience mechanism according an embodiment of the present invention is implemented; and Figure 3 illustrates a block diagram of an embodiment of the server error resilience mechanism implemented in the system of Figure 2.

With reference to Figure 2 there is shown a video transmission system 30, provided with a transmitting server 32 and a receiving client 34. The server 32 is provided with an encoder 40, which in this case implements a H.264 video coding standard, multiplexer 42, which in this place implements a MPEG-2 Transport Stream (TS) container format, a server delivery protocol mechanism 44 and a transmitter 46.

The client 34 is provided with a receiver 50, a client delivery protocol mechanism 52. a demultiplexer 54 which in this case implements the demultiplexing of the MPEG-2 Transport Stream (TS) container format and a decoder 56 which in this case implements the decoding of the H.264 video coding standard.

With reference to Figure 3 there is shown a block diagram of the error resilience mechanism 60 which is implemented in the server 32 of video transmission system of Figure 2. A raw video data signal 62 is input into the H.264 standard video encoder 40 which compresses the video data signal into a compressed video bit stream 64 before slicing the video bit stream 66 into self-contained chunks. In the H.264 standard, a slice is a portion of the bit stream that is self-contained in the sense that if the active sequence parameter set (SPS) and picture parameter set (PPS) are known, the syntax elements within a slice can be parsed from the bit stream and the values of the samples in the area of the picture that the slice represents can be decoded without the use of data from other slices, provided that the previously decoded pictures referenced by the slice are available at the decoder.

Within the H.264 encoder 40, the slices are encapsulated into Network Adaptation Layer Units (NALU5) 68. H.264 NALUs include, in this case, a 1 byte NALU header and form a ff264 elementary stream (ES). The NALUs produced by the H.264 encoder 40 are provided to multiplexer 42 In the multiplexer 42, the H.264 ES is packetized into a Packetized Elementary Stream (PES) 70 with, every PES packet containing a single slice, A data_alignment_indicator field in the PES header of every PES packet is, in this case, set to indicate that each PES packet contains one slice. In addition. NALUs that do not contain a slice are inserted in the same PES packet as the slice preceding the non-slice NALUs and NALUs containing SPS or PPS information are inserted into the PES packet containing the first slice following the SPS or PPS NALUs. Furthermore, each PES packet contains an integral number of H.264 NALUs. A presentation time stamp (PTS) or decoding time stamp (DTS) is provided in PES packet headers which contain the first byte of an advanced video coding (AVC) access unit. The PTS or DTS refer to the first access unit that commences in a given PES packet. Therefore, when an access unit is split into multiple PES packets, only the first PES packet contains the PTS and DTS information.

The PES is in turn packetized into a MPEG-2 Transport Stream (TS) 72. TS packets are, in this case, always 188 bytes, with 4 bytes of header and 184 bytes of payload. In this case a payload_unit_start_indicator field is used to indicate that the payload of the TS packet commences with the first byte of a PES packet. Each PES packet is fragmented into one or more TS packets with padding included where necessary to produce an integral number of TS packets. Therefore any one TS packet only contains data from one PES packet.

The TS packets are provided to delivery protocol mechanism 44 where they are aggregated using the Delivery Protocol (DP) 74 into DP packets. In this case, all TS packets belonging to the same PES packet, as indicated by the payload_unit_start_indicator, are packetised into a single DP packet and each DP packet contains all the TS packets belonging to only one PES packet. Furthermore, every DP packet contains an integral number of TS packets.

The DP packets are then encapsulated into Network physical layer protocol data units PPDUs via the network protocol mechanism 76 The DP packet size is determined by the delivery protocol mechanism 44 such that every network PPDU contains a single Delivery Protocol packet which means no packet aggregation or fragmentation occurs at the network or subsequent protocol layers. After taking into account any header introduced by the network and subsequent protocol layers, the resulting network PPDU is as close as possible but less or equal to the maximum transmission unit (MTU) size of the underlying network The PPDU's are then provided to transmitter 46, from where they are transmitted over the network 78 The error resilience mechanism 60 implements the co-ordinated configuration of the transmission stream data PDDU's within the server 32 to minimising the impact of packet loss on the video quality received by the client 34. Quantitatively, error resilience mechanism 60 ensures that a single network PPDU loss from the transmitted data stream will never result in more than one H.264 slice being lost or corrupted at the display device to which the client 34 provides the received video data. The robustness of the error resilience mechanism 60 is optimized by the determination of appropriate initial slice size for each encoded media data within system 30. Within the encoder 40, every picture included in the raw video data must be encoded into one or more slices. The number of bytes in a slice is variable. However, in this case, the H.264 encoder 40 is configured to encode a variable number of macroblocks (MBs) per slice such that each slice is close to a specified size in bytes. By generating more slices per picture, the video encoder 40 increases the robustness to loss of the data stream and in turn any errors arising will account for a smaller component of the picture data and therefore a smaller region of the picture will be affected. However, in one embodiment of the system 10 small slices are aggregated in the transport and network layers (not shown). In this case, the loss of a PPDU will result in multiple lost slices. In an alternative embodiment of the system 10, where no aggregation of the small slices is performed, each small slice is carried in a separate PPDU; the header overhead incurred in the network will increase, resulting in a reduction in throughput of transmitted video data. The use of small slices also reduces the compression efficiency of the codec mechanism implemented across video encoder 40 and decoder 56 as more re-synchronization information is needed in the bit stream in order to make each slice independently decodable. Therefore. within encoder 40 the determination of slice size affects the relative optimization of the compression efficiency, packetization overhead, network throughput and robustness to loss of data of the system 30.

The error resilience mechanism 60 determines maximum slice size optimal for the system headers and limitations of the parameters of the encoder 40, multiplexer 42 and delivery protocol mechanism 4, along with the mechanism implemented with reference to Figure 3. The maximum network PPDU size is determined by the maximum transmission unit (MTU) of the underlying physical network, e.g. the size of the largest data packet that the underlying physical network protocol can transmit. In the case of the Ethernet, the MTU size is 1500 bytes.

As an example, Table 1 below lists the maximum slice size for a number of DP packet sizes. The calculation in Table 1 assumes that the NALU packet header is 1 byte. the PES packet header contains PTS/DTS fields only for the first NALU of a picture and therefore in this case the PES header for the PES packet containing the first slice of a picture is 19 bytes, whereas the PES header for all other PES packets is 9 bytes. The TS packet header is 4 bytes. The Delivery Protocol header is 8 bytes and the DP packet must contain an integral number of 188 byte TS packets.

In addition, the maximum Internet Protocol (IP) packet size is also shown to illustrate the example with reference to an underlying network which is IP-based, Max slice Max. slice size Delivery size (first PES Number Max. IP (not first slice Protocol slice in packet of TS Packet in picture) Packet picture) size(bytes) packets size(bytes) bytes) size(bytes) bytes) 164 174 184 1 196 224 348 358 368 2 384 412 716 726 736 4 760 788 1452 1462 1472 8 1512 1540 Table 1: Slice Size and Delivery Packet size The data shown in Table 1 illustrates an example where certain factors are not taken into account. An example of a factor which has not been taken into account is the situation when SPS or PPS NALUs are present in a PES packet, the maximum slice size for that PES packet must be reduced by the corresponding size of the SPS or PPS NALUs. In the example of Table 1, the maximum slice size has not been adjusted. Similarly, the last slice in a picture is commonly followed by NALUs not containing slices, e.g. SEI messages or access unit delimiters (AUD).

When these NALUs are present in a PES packet, the maximum slice size for that PES packet must be reduced by the corresponding size of these NALUs, in the example of Table 1, this factor has not been taken into account.

In a second embodiment of the implementation of error resilience mechanism 60 in a video transmission system, the mechanism 60 is enhanced with the addition of error protection to the Delivery Protocol packets in the form of a forward error correction (FEC) scheme applied across the TS packets and integrated with the Delivery Protocol signaling.

A typical FEC scheme generates a number of repair symbols from a number of source symbols. A number of symbols could be lost during transmission. FEC decoding succeeds if a sufficient number of symbols are received correctly, and in this case, all the source symbols can be recovered. FEC decoding fails if an insufficient number of symbols are received, and, in that case, none of the missing source symbols can be recovered.

An FEC scheme can be included in an embodiment of the present invention as follows: * All the TS packets belonging to a group of picture (GOP) are grouped into an FEC Block. A random_access_indicator field of the TS Header Adaptation Field can then be used to indicate the start of a GOP I FEC is applied over all the TS packets in an FEC Block * The FEC Repair symbols are encapsulated into Delivery Protocol packets I. An example rateless FEC scheme known as Raptor FEC is described in RFC5053.

Additional optimizations that can be applied when using such a scheme are: To maximise the efficiency of the Raptor code, 1 Raptor symbol = 1 TS packet I For each FEC block, a number of 188-byte Raptor repair symbols are generated I A FEC symbol can be split into sub-symbols if a larger K is needed, or if the optimum value of K cannot be achieved because of delay constraints on the length of a GOP The above scheme enables the video bit-rate and the amount of FEC to be changed for each FEC block.

When FEC symbols are not present, a delivery protocol packet contains a block ID and a sequence number, and the payload data consists of an integer number of TS packets. In this case, the sequence number is always monotonically increasing and is not reset when the block ID is incremented. Lost delivery protocol packets can be detected by gaps in the sequence number.

With a first FEC scheme, each delivery protocol packet contains one and only one FEC symbol. Symbols are delineated by monotonic sequence numbers (symbol number sequence number).

I -

With a second FEC scheme, each packet can contain one or more FEC symbols.

The number of symbols per packet is not fixed. Symbols are still delineated by monotonic sequence numbers. In this case, the sequence number in the delivery protocol packet indicates the symbol number of the first FEC symbol present in that delivery protocol packet. Missing symbols can be identified by gaps in symbol number inferred from the symbol number of the first FEC symbol present in this packet Embodiments of the present invention provide methods and/or apparatus for controlling an output of a video encoder in conjunction with aggregation and fragmentation mechanisms occurring at the subsequent protocol layers such that the effect of a lost network PPDU on the reconstructed video at the receiver is minimised.

Such techniques can be combined with a rateless Forward Error Correction (FEC) scheme with minimal signalling such that the FEC coding rate and the video bit-rate can be changed on-the-fly in a seamless manner.

This document details a novel method by which knowledge of this fragmentation and aggregation can be used to dictate a slicing strategy with the objective of minimizing the effect of a lost packet on the reconstructed video at the receiver.

Although aspects of the invention have been described with reference to the embodiments shown in the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments shown and that various changes and modifications may be effected without further inventive skill and effort, for example, the error resilience mechanism 60 can be implemented in a system which include other multimedia streams such as audio streams in addition to the video stream. In such a system, the additional multimedia streams are encapsulated inside the MPEG-2 TS as specified by the MPEG-2 TS standard. When the bit-rate of the additional stream is low compared to the video stream, the TS packets be'onging to the additiona' stream are inserted in the same FEC block as the video TS packets. When the bit-rate of the additional stream is comparable to the video stream, it is coded separately within its own FEC block.

Claims

CLAIMS1. A method of transmitting a multimedia datastream over a transmission channel, the method comprising: a. receiving a multimedia datastream; b. slicing the received datastream into a plurality of multimedia slices having a predetermined slice size; c. encoding the multimedia slices into first data packets of a first predetermined size; d. dividing each of the first data packets into a respective integral number of second data packets of a second predetermined size; e. aggregating the second data packets into a stream of third data packets of a third predetermined size, each third data packet containing all of the second data packets relating to a single one of the first data packets; and f. transmitting the series of third data packets over a transmission channel.
2. A method as claimed in claim 1, further comprising the step of encoding the first data packets into respective encoded first data packets of a predetermined size before dividing each of the first data packets into a respective integral number of second data packets, each such encoded first data packet including all of the first data packets relating to a single one of the multimedia slices.
3. A method as claimed in claim 1 or claim 2, further comprising the step of encapsulating the third data packets into respective encapsulated third data packets of a predetermined size, before transmitting the series of third data packets over a transmission channel.
4. A method as claimed in any one of the preceding claims, wherein the predetermined slice size is chosen such that the predetermined size of a transmitted third data packet is not greater than a permitted maximum size for the transmission channel.
5. A method as claimed in claim 4, wherein the predetermined size of a transmitted third data packet is substantially equal to the permitted maximum size for the transmission channel.
6. A method as claimed in any one of claims 3 to 5, wherein each encapsulated third data packet includes a single third data packet.
7. A method as claimed in any one of the preceding claims, wherein aggregation of second data packets into third data packets includes applying a forward error correction scheme to the second data packets, and including forward error correction data in the third data packets.
8. A method as claimed in claim 7, further comprising grouping the second data packets into blocks, and applying the forward error correction scheme to all of the second data packets in a block.
9. A method as claimed in claim 7 or 8, wherein the forward error correction data include forward error correction repair symbols.
10. Apparatus for transmitting a multimedia datastream over a transmission channel, the apparatus comprising: a. an input unit operable to receive a multimedia datastream; b. a slicing unit operable to slice a received datastream into a plurality of multimedia slices having a predetermined slice size; c. a first encoder operable to encode such multimedia slices into first data packets of a first predetermined size; d. a divider operable to divide each such first data packet into a respective integral number of second data packets of a second predetermined size; e. an aggregation unit operable to aggregate such second data packets into a stream of third data packets of a third predetermined size, each third data packet containing all of the second data packets relating to a single one of the first data packets; and f. a transmitter operable to transmit such a series of third data packets over a transmission channel.
ii. Apparatus as claimed in claim 10, further comprising a second encoder operable to encode such first data packets into respective encoded first data packets of a predetermined size, each encoded first data packet including all of the first data packets relating to a single one of the multimedia slices.
12. Apparatus as claimed in claim 10 or claim ii, further comprising an encapsulation unit operable to encapsulate the third data packets into respective encapsulated third data packets of a predetermined size.
13. Apparatus as claimed in any one of claims 10 to 12, wherein the predetermined slice size is chosen such that the predetermined size of a transmitted third data packet is not greater than a permitted maximum size for the transmission channel.
14. Apparatus as claimed in claim 13, wherein the predetermined size of a transmitted third data packet is substantially equal to the permitted maximum size for the transmission channel.
15. Apparatus as claimed in claim 12, 13, or 14, wherein each encapsulated third data packet includes a single third data packet.
16. Apparatus as claimed in any one of claims 10 to 15, wherein the aggregation unit is operable to apply a forward error correction scheme to the second data packets, and to include forward error correction data in the third data packets.
17. Apparatus as claimed in claim 16, wherein the aggregation unit is operable to group the second data packets into blocks, and to apply the forward error correction scheme to all of the second data packets in a block.
18. Apparatus as claimed in claim 16 or 17, wherein the forward error correction data include forward error correction repair symbols