CN106100803A - The method and apparatus determined is retransmitted for making - Google Patents

Abstract

Methods and apparatus are described herein that include analyzing (410) abstraction layer headers and assigning priorities to packets to be sent. A sender's network monitor (270, 405) is connected between the network interface (250) and the retransmission decider (275) for use in accordance with, for example, assigned priorities and collected network statistics or conditions, such as , the packet loss rate determines which packets to resend. A network monitor (271) may also be provided on the receiver for collecting current network statistics like cumulative number of lost packets and providing such statistics to the sender (200). The method further includes buffering the data to be sent along with the assigned priority, sending the data retrieved from the buffer (235) via the datagram protocol (240, 245), receiving (405) a request to resend the data, determining (415 ) whether the requested data is in the buffer, and resend the requested data via a protocol that provides end-to-end acknowledgment and error correction of the data.

Description

Method and apparatus for making a retransmission decision

This application is a divisional application of an invention patent application with a filing date of January 28, 2010, an application number of 201080065862.2, and an invention title of "Method and Device for Making Reissue Decision".

Cross Reference Related Applications

This application is related to the following co-pending, commonly-owned U.S. patent applications: (1) filed on October 7, 2009 as International Patent Application (Application No. PCT/US09/005499, Thomson Corporation Docket No. PU090136) titled No. XXX of "ANEFFICIENT APPLICATION-LAYER AUTOMATIC REPEAT REQUEST (ARQ) RETRANSMISSIONSCHEME FOR RELIABLE REAL-TIME STREAMING IN WIRELESS NETWORKS"; and (2) an invention filed as an International Patent Application (Application No. XXX, Thomson Corporation Docket No. PU090157) No. XXX entitled "AMETHOD AND APPARATUS FOR PARSING A NETWORK ABSTRACTON-LAYER FOR RELIABLE DATA COMMUNICA TION".

technical field

This application relates generally to digital data networks, and more particularly to network monitors and data retransmission deciders for reliable digital data transmission.

Background technique

In multicast or broadcast applications, data is sent from a server to multiple receivers, typically over a wired and/or wireless network. A multicast system as used herein is a system in which a server simultaneously sends the same data to multiple receivers, where the receivers form a subset of all servers up to and including all servers. A broadcast system is one in which the server sends the same data to all receivers simultaneously. That is, a multicast system may by definition include a broadcast system.

Data is usually formatted into packets and/or frames for transmission. That is, packets and/or frames are data formatting schemes. As used herein, data may be formatted in any convenient format for transmission including packets and/or frames. Accordingly, "packet" is used herein to define any data formatting scheme known to those of ordinary skill in the art.

As an example of an application of the digital data network herein, video transmission or distribution in a wireless network is used. Video transmission in wireless data networks typically involves packet loss caused by channel error conditions like interference, channel fading, collisions, and the like. When such a channel error condition occurs, the radio link layer of the protocol stack may attempt to retransmit the packet a fixed number of times within a fixed time interval. If these retransmissions are unsuccessful, the radio link layer discards the packet. Video distribution over Internet Protocol (IP) networks typically transports video packets to their destinations (receivers; sometimes referred to herein as clients) using the Real-time Transport Protocol (RTP) protocol, which in turn uses A reliable Transmission Control Protocol (TCP) transport protocol or a less reliable User Datagram Protocol (UDP) transport protocol. When the less reliable UDP protocol is used, for example, the protocol does not provide a means of detecting garbled packets or recovering lost packets, leaving the onus to the application to correct packet delivery errors. In contrast, when using the TCP protocol, end-to-end acknowledgments are provided so that the protocol tries to send and/or receive media (audio, video, multimedia...) packets (data) strictly in the order of the application management packets. When a packet error is detected, TCP provides a sliding window mechanism for data flow control and reduces the packet transmission rate. TCP continues to resend lost packets until they are recovered.

Video transmission is an example of an application that occurs in real time and has a user viewing experience tied to the reception and reproduction of data. Delays or time constraints within which packets must be delivered or recovered should not affect the end user's viewing experience. Therefore, grouping errors should be corrected within a limited time, otherwise, the data may not be visible. TCP currently cannot provide guarantees for control of packet recovery based on time constraints. Therefore, using TCP as a transport protocol for wireless networks will result in poor viewing experience for users. Also, TCP requires positive acknowledgments for all sent data packets. TCP uplink acknowledgments (from data receiver to data sender (sender)) compete for wireless bandwidth with downlink data traffic (from sender (sender) to receiver). If collisions occur between downlink and uplink transmissions, these collisions will result in further throughput reduction.

PCT application US/09/005,499 filed on October 7, 2009 discloses an efficient application layer automatic repeat request retransmission method in which data to be transmitted is buffered or cached on a module implementing a reliable media protocol for recovery Lost data packets are also useful for real-time streaming (eg video) data applications, for example. Referring briefly to FIG. 1 (derived from FIG. 5 of the aforementioned PCT application US/09/005,499), a real-time protocol packetization module 120 is provided, eg, on real-time server 100 , to receive video digital data input 105 . A real-time protocol (RTP) packetization module 120 that provides streaming applications interfaces with a reliable media protocol module 130 via, for example, one or more socket-like application programming interfaces (APIs) 115 . The reliable media protocol module 130 accepts configuration data parameters 125 like cache or buffer size, maximum time to wait for packet recovery, and the like. These parameters are for example determined by digital applications such as telephony, audio, video or multimedia and other known applications. A local buffer or cache memory 135 temporarily stores initially transmitted packets until it can be determined whether to retransmit or discard them. Server 100 transmits digital data through one of UDP/IP sender interface 140 or TCP/IP sender/receiver interface 145 and a digital data network interface, eg, Ethernet/802.11 interface 150 . Initial real-time transmission may occur via initial transmission 160 to network 110 as real-time packets are temporarily stored in cache/buffer 135 , eg, while awaiting receipt of retransmission ACK/NACK control 155 or a predetermined time limit.

Contents of the invention

It would be advantageous to have efficient methods and apparatus for further increasing the reliability of real-time data transmission systems based on reliable media protocols like the one disclosed in the aforementioned PCT application US/09/005,499. The present invention addresses these and/or other problems.

According to one aspect of the invention, a method is disclosed. According to an exemplary embodiment, the method includes monitoring the digital data network to collect network transmission statistics; and deciding whether to retransmit the digital data packet to the receiver based on the collected network transmission statistics.

According to another aspect of the present invention, an apparatus is disclosed. According to an exemplary embodiment, a device such as a digital data transmitter includes a digital data network monitor for collecting network transmission statistics; Transmission statistics determine whether to retransmit a digital data packet.

According to another aspect of the present invention, an apparatus is disclosed. According to an exemplary embodiment, the device as a digital data transmitter includes means, like a network monitor, for collecting network transmission statistics; Transmission Statistics A device that determines whether to retransmit a digital data packet.

According to another aspect of the present invention, an apparatus is disclosed. According to an exemplary embodiment, a device such as a digital data receiver includes a network monitor for collecting network statistics from the perspective of the receiving device; and a network interface for outputting a control channel message including said collected network statistics .

According to another aspect of the present invention, an apparatus is disclosed. According to an exemplary embodiment, a device such as a digital data receiver includes means, such as a network monitor, that collects network statistics from the perspective of the receiving device; and, such as a network interface, outputs controls that include said collected network statistics A device for channel messages.

Description of drawings

The above and other features and advantages of the present invention and the manner of realizing them will become more apparent and the present invention can be better understood by referring to the following description of the embodiments of the present invention made in conjunction with the accompanying drawings, in which:

Figure 1 is a schematic block diagram of a server containing reliable media protocol modules and a local cache or buffer storing eg real-time packets while awaiting a retransmission decision from Figure 5 of the aforementioned related PCT application US/09/005,499.

Figure 2A is a schematic block diagram of a server in accordance with the principles of the present invention showing a Network Abstraction Layer (NAL) monitoring network interfaces for current network statistics and providing input to a Reliable Media Protocol (RMP) module in accordance with the present invention Header Analyzer, Network Monitor, and Retransmission Decider; FIG. 2B is a schematic block diagram of an associated client including a network monitor that collects receiver network-specific statistics for sending to the server of FIG. 2A in accordance with the principles of the present invention picture.

3A is a simplified block diagram of a streaming server including a NAL header analyzer receiving input from a scalable video coding (SVC) encoder, a network monitor, and a retransmission decider in accordance with the principles of the present invention. Figure 3A further depicts a streaming receiver receiving digital data from a streaming server. 3B is a block schematic diagram of the streaming server of FIG. 3A showing the NAL header analyzer receiving input from a local MP4 file reader. Figure 3C shows the NAL header analyzer of Figure 3A receiving input from the network via a depacketizer.

FIG. 4 shows a flow chart of the operation of abstract layer header analysis and network monitoring at the sender side according to the principle of the present invention, where the sender can be the sender, server or streaming server in FIG. 2A or FIG. 3 . FIG. 4 further illustrates a data receiver and related network monitoring operations, where the receiver may be the client of FIG. 2B or the streaming receiver of FIG. 3 .

FIG. 5 shows an exemplary data content representation format of a Scalable Video Coder (SVC) Network Abstraction Layer (NAL) extension header analyzed according to the principles of the present invention.

Figure 6 illustrates an exemplary format of a Real Time Protocol (RTP) header from which priority data may be transmitted and from which network statistics may be calculated in accordance with the principles of the present invention.

7 illustrates an exemplary format of an MPEG Transport Stream (TS) header, eg, MPEG-2, including a Continuation Count (CC) field from which network monitoring statistics may be derived, in accordance with aspects of the present invention.

8 illustrates an exemplary format of a message including a send request and an end-to-end packet loss rate from which network monitoring statistics may be derived in accordance with aspects of the invention.

9 illustrates a first exemplary format of an ARQ request (NACK) packet from which network monitoring statistics may be derived in accordance with aspects of the present invention.

10 illustrates a second exemplary format of an RMP ARQ request (NACK) packet from which network monitoring statistics may be derived and transmitted in accordance with aspects of the present invention.

detailed description

The present invention is directed at a sender or method for analyzing an abstraction layer header as input to a real-time (e.g., real-time protocol or RTP) packetization module for which the improved reliable media protocol module of FIGS. 2A and 2B is a module. and devices, and the network monitor and retransmission decider of the sender/server 200 (FIG. 2A) that can selectively retransmit data using the priority output of the analyzer according to the monitored network conditions, such as packet loss rate. According to one embodiment, the network abstraction layer of a real-time video encoder such as MPEG-4 Scalable Video Coder (SVC), for example, H.264/AVC network abstraction layer includes providing The header of the field. For example, the NAL is network friendly and can represent video telephony, eg, video conversational applications and non-conversational applications like storing downloaded movies in memory, broadcast or multicast applications or streaming applications, or other non-conversational applications. A NAL can be defined, for example, as a number of abstraction layer units in packets containing one or more bytes. The first byte of each NAL unit may be a header byte indicating the data type of the unit, while the remaining bytes contain payload data of the type indicated by the NAL header.

According to one embodiment, the abstraction layer header parser includes parsing the scalable video coder abstraction layer header to obtain data from one of the plurality of fields. According to some aspects of this embodiment, the abstraction layer header parser may, for example, operate on output digital video data from an MPEG 4 file reader, or accept digital data from a network as received at a depacketizer. For example, the abstraction layer header analyzer may analyze the DID field representing the degree of inter-layer coding dependency of layer representations. The QID field represents the quality level represented by the Medium Granular Scalability (MGS) layer. The TID field represents the time level represented by the layer. One result of analyzing an abstraction layer header containing such a field is to identify the layer to which the subsequent payload data is associated. Each identification layer may be assigned a different priority in response to the identification of the payload data type. For example, the base layer may receive high priority for retransmission from a buffer or cache since the base layer is absolutely necessary for full decoding at the receiver. In response to identification of payload data types, the analyzer may assign different priorities to each identified layer of data as follows. For example, the base layer may receive high priority for retransmission from a buffer or cache because base layer data is absolutely necessary for full decoding at the receiver. Medium priority may be assigned to enhancement layers, since enhancement layer data is necessary for decoding of higher layers. On the other hand, low priority can be assigned to high enhancement layers of payload data. Once a priority is specified, e.g., low, medium, and high, the priority information may be indicated in the "Payload Type (PT)" or similar field of the Real-Time Protocol (RTP) header if retransmissions are required at the transport layer, Or priority information may be indicated in a Type of Service (TOS) field of an IP header. The service type field may also be called, for example, a differentiated service (DS) field in the prior art. The first two bits are called Explicit Congestion Notification (ECN) bits, and the next six bits of the DS field byte are called Differentiated Services Code Point (DSCP) bits. Service types are used herein generally to refer to these and other formats that provide service type data. Thus, real-time packet retransmission decisions made by the reliable media control protocol module may be layer aware when modified to be incorporated into the abstraction layer header analyzer and method according to one embodiment.

In a further embodiment, a network is provided that monitors data network quality via a data network interface and, for example, collects current network data statistics input into a retransmission decider, such as packet loss rate, available bandwidth, and round-trip delay monitor. The retransmission decider can in turn provide an input to the Reliable Media Protocol (RMP) module so that the retransmission decision can be based on the current network condition according to the data network statistics collected in the network monitor, and according to its priority and monitored Network conditions selectively determine the retransmission of packets.

Accordingly, the digital sending method includes analyzing an abstraction layer header of a digital data packet to obtain layer representation data, and assigning a priority to the digital data packet of the presentation layer responsive to the analysis. Obtaining layer representation data may comprise one or all of the following steps: determining the degree of inter-layer encoding dependencies; determining the quality level of granular scalability; and determining the temporal scale of the layer representation. In addition, the method may further include indicating one of the priority levels in the payload type field of the real-time transmission packet header or in the service type field of the Internet protocol packet header. As mentioned above, the input to the abstraction layer header parser may be received from a depacketizer receiving network data or from local server memory such as that associated with an MPEG file reader.

Additionally, in a further embodiment, the digital data transmitter may comprise an abstraction layer header analyzer for obtaining and prioritizing layer representation data from a digital data packet; a digital data monitor for collecting a network transmission statistic; and a retransmission decider, configured to decide whether to retransmit the digital data packet according to the assigned priority of the digital data packet and the collected network transmission statistic. As mentioned above, in a further embodiment such an abstraction layer header parser may be a network abstraction layer header parser for a digital video packet application.

More specifically, with reference to Fig. 2-Fig. 10, the abstraction layer header analyzer of server, data monitor and retransmission decider and the network monitor of receiver are further described for RMP module 235 in making whether to retransmit High reliability operation is guaranteed when the decision to report a lost packet on the transmit control channel 255 is made. Video transmission or distribution in wireless networks typically uses the Real-time Transport Protocol (RTP) Moving Picture Experts Group 2 Transport Stream (MPEG2TS) over UDP. Live video can be distributed from a single source to a single destination (unicast mode) or from a single source to multiple destinations (multicast mode). Since channel conditions are changing in wireless networks, when channel conditions are bad, packet transmission results in packet loss if link layer error correction is not successful. Under these conditions, there are gaps in the sequence of packets that result in poor viewing quality for the end user. The present invention provides an abstract layer header analysis function here called Reliable Media Protocol (RMP) based application layer efficient retransmission scheme to recover lost packets to facilitate reliable real-time streaming applications. The present invention also provides network monitoring capabilities on the sender side and on the receiver side to improve packet retransmission decision making at the server/sender 200 of FIG. 2A. Therefore, a network monitor 271 is provided on the receiver/client 201 of FIG. 2B.

2A and FIG. 2B, in the reliable media protocol (RMP) method of the present invention, help RTP packetization module 220 and reliable media protocol (RMP) module 230 by incorporating NAL header analyzer 210 according to the principle of the present invention . In addition, the network monitor 270 interfacing with the network interface 250 collects current network statistics and improves the reliability of the RMP module 230 via the retransmission decider 275 . NAL header analyzer 210, network monitor 270, and retransmission decider 275 are emphasized in FIG. 2A to show that FIG. 2A differs from FIG. 1 as further described below.

Initially, RMP module 230 sends conventional unicast and multicast data or packets using UDP 240 via network interface 250 to send packets to network 110 using initial send channel 260 . The data is initially stored in the local cache 235 via the RTP packetization module 220 with the priority specified by the NAL header analyzer 210 . In addition to this, an additional reliable TCP-based control channel 245 is established between the source (sender, sender) 200 of FIG. 2A and the destination (receiver, sink, client) 201 of FIG. 2B. TCP/IP 245 requests retransmissions and receives lost packets through network interface 250 via retransmission ACK/NACK control channel 255 to network 110 . In order for this mechanism to work properly, the sender (sender, server) 200 keeps in cache 235 the most recent packets sent to its receiver/client. One or more receivers/clients 210 receive data packets from the sender/server 200, and may use the sequence number field present in the RTP (FIG. 6) or MPEG Transport Stream (TS) header (FIG. 7) Detect sequence gaps in the data packets. If the receiver 201 detects a sequence gap, the receiver 201 sends a request on the TCP-based control channel 255 for selective retransmission of the missing data packets. When a sender/server 200 receives a retransmission request from one or more of its receiver/clients 210, it looks in its local cache 235 for the most recent packet. If the requested packet is found in the local cache 235 , the sender/server 200 retransmits a copy of the packet to the receiver 201 via the network 110 in unicast form on the TCP-based control channel 255 . If the requested packet is not found in its local cache/buffer 235, the sender 200 continues to service the remaining retransmission requests. The receiver/client 201 maintains a transmit queue (buffer/cache) 236 to hold all received data packets from both the data channel and the control channel. The receiver/client 201 also reorders the retransmitted packets into their normal sequence (position) within this queue and delivers the packets to an application, eg, a video player application 221 for display 206, in the proper order at the correct time.

The receiver/client 201 (FIG. 2B) maintains a configurable time window to wait for any retransmission instead of waiting forever so that packet delay and delay jitter can be kept within application bounds. Furthermore, according to one embodiment, the receiver/client 201 is equipped with a network monitor 271 that collects network statistics from the perspective of the receiver/client 201 to send to the sender/server 200 via the control channel 255 . Network monitor 271 is highlighted in FIG. 2B to show that network interface 251 and network statistics are collected and provided to sender/server 200 in accordance with the present invention to improve retransmission decision making, as further described herein. Collaboration between monitors 271 .

If retransmission acknowledgments for some lost packets are not received in time, the receiver/client 201 of FIG. If some retransmitted packets are received outside the acceptable recovery time window, the receiver discards them. It should be noted that video applications can tolerate some data packet loss using error concealment techniques during video decoding.

Referring again to FIG. 2A , in order to make packetization and improve retransmission decisions by means of network monitors 270 and 271, the Reliable Media Protocol (RMP) scheme in the present invention operates in real-time applications via RMP modules 230, 231 with the help of analyzer 210 Between /RTP/MPEG TS and UDP/TCP/IP. On the server/transmitter 200, an abstraction layer header parser, eg, a Network Abstraction Layer (NAL) header parser 210 operates on locally provided or networked digital video data 205 . NAL header analyzer 210 outputs priority and data for real-time protocol (RTP) packetization module 220 . In addition, the network monitor 270 retrieves current network 114 statistics and outputs control data to the retransmission decider 275 to assist the RMP module 230 . For example, taking the example where the enhancement layer is assigned a medium priority and the base layer is assigned a high priority, the RMP module 230 retransmits the high priority base layer when the available bandwidth detected by the network monitor 270 is lower than a given threshold data without sending medium priority enhancement layer data. For example, instead of sending all packets stored in cache/buffer memory 235, retransmission decider 275 may collect statistics based on network monitors such as packet loss rate or other network conditions, and NAL header analyzer 210 specified Priority is used to select a portion of packets stored in memory 235 for retransmission. Thus, as further described in connection with the discussion of FIG. 4 , the combination of prioritizing data at analyzer 210 and making retransmission decisions per retransmission decider 275 provides guarantees for highly reliable media protocol (RMP) module 230 .

Referring again to FIG. 2A , a typical network interface 250 is shown between the network 110 and the RMP module 230 on the server/sender 200 . Examples of network interface modules are Ethernet cards, IEEE 802.11/WiFi cards connected to the computer network 110 .

FIG. 2B shows a schematic diagram of an exemplary implementation of a client device 201 . The client computer may include a video player/streaming application module 221, a display 206, a buffer (cache memory) 236, an optional media protocol (RMP) module 231, a UDP/IP module 241, a TCP/IP module 246 and a network Interface 251. Network interface 251 may be, for example, an Ethernet interface or an IEEE 802.11 interface or other known network interface. The network interface 251 receives all incoming messages. These messages arrive on different sockets/addresses. The network monitor 271 collects current network statistics from the network interface 251 from the perspective of the receiver/client 201 and outputs appropriate control messages on the channel 255 reporting the collected statistics. Thus, the network interface 251 can determine where to forward received messages from the network 110 and output statistics from the network monitor 271 and messages from the TCP/IP module 246 . New incoming data packets are forwarded by the network interface module 251 to the UDP/IP interface. The request to resend the data packet and the resent data packet are forwarded by the RMP module 231 to the TCP/IP module 246 . The RMP module 231 determines whether a received data packet is corrupted and orchestrates packet recovery using both UDP/IP and TCP/IP. RMP module 231 generates retransmission requests for any corrupted data packets. Then, the RMP module 231 forwards the retransmission request to the TCP/IP module 246 for sending on the network 110 . At network interface 251 , any network statistics collected by network monitor 271 are incorporated into a retransmission request for transmission over network 110 . RMP module 231 also stores received packets in local buffer 236 for reordering. Once the retransmitted packets are received from the TCP control channel via the TCP/IP module 246, the RAMP module 231 arranges the packets in the correct order. RMP 231 maintains a queue sorted by sequence number and reorders and inserts packets into this buffer area/queue. When the recovery window expires, the RMP module 231 delivers the packet to, for example, the player/streaming application 221 for display 206 . The RMP module 231 provides a socket-like application protocol interface (API) 216 for data transfer and integration with applications. Note that some packets may not be recovered when the recovery window expires.

Data packets arriving after the recovery window expires are discarded as shown in FIG. 4 . Depending on the application, eg, for video, the streamer/player application 221 depacketizes and/or decodes the data and passes the data to the display/speaker 206 . Incoming packets are stored in the RMP "buffer area" and forwarded to, eg, the application for rendering (display) 206 whenever an application requests a packet. The box labeled "Configuration" 226 is the "Configuration Interface" with the RMP module 231 . The RMP module can be configured at initialization time to set parameters like cache size, maximum time to wait for packet recovery, etc.

On receiver/client 201 according to FIG. The client 201's network experience is output to a server connected to it via the digital data network 110. Once received at sender/server 200 via network 110, the network statistics thus collected are output to network monitor 270 via interface 250 in order to supplement what network monitor 270 collects from the sender/server perspective of network 110 as discussed above statistics. Digital data network 110 may be any digital data network including, but not limited to, satellite networks, terrestrial wireless networks, fiber optic networks, coaxial cable networks, twisted pair networks, local area networks, wide area networks, and other known digital data networks. A typical network interface 251 is shown between the network 110 and the RAM module 231 on the client/server 201 . Examples of network interface modules include Ethernet cards, IEEE 802.11/WiFi cards, etc. connected to the computer network 110 .

Before discussing Figure 3, it is important to note that like numerals refer to like elements in the Figures. Furthermore, the first digit of a reference number like the first digit 1 of reference numeral 110 of network 110 indicates the figure number in which the element first appears. Thus, for example, RAM module 230 appears for the first time in FIG. 2 and is similar to, but different from, Reliable Media Protocol (RAM) module 130 of FIG. NAL header analyzer 210, network monitor 270 and retransmission decider 275 appear in FIG. Additionally, network monitor 271 of client/receiver 201 of FIG. 2B improves the operation of RMP module 230 and retransmission decider 275 of FIG. 2A in accordance with the principles of the present invention. Some of the advantages of analyzer 210, network monitors 270 and 271, and retransmission decider 275 have been described above, but will be further described below in connection with the discussion of FIGS. 3 and 4 .

FIG. 3 provides a direct representation of NAL abstraction layer header analyzer 210 , network monitor 270 and retransmission decider 275 involved in sending to one or more streaming receivers connected via network 110 . Referring to FIG. 3A, there is shown a scalable video codec (SVC) encoder 300 which may be located in the streaming server or remotely from it. As described further below, a scalable video codec (SVC) provides temporal or frame rate scalability, spatial or frame size scalability (e.g., encoding video at multiple resolutions and aspect ratios), and signal-to-noise ratio (SNR) or quality/fidelity scalability. As further described in connection with FIG. 4, NAL header analyzer 210 may be an H.264 compliant AVC Network Abstraction Layer (NAL) header analyzer or other abstraction layer header analyzer with similar functionality. For example, referring to FIG. 3B , NAL header analyzer 210 may receive input from a local memory bank, eg, MP-4 file reader 302 . An exemplary streaming server embodiment may be a streaming server providing IP television channel or movie services. In an alternative embodiment FIG. 3C , a depacketizer 304 that receives, for example, streaming video from network 110 and outputs the received packets to NAL header analyzer 210 is shown. An exemplary video application for the streaming server of FIG. 3C may be video telephony in which the streaming server is depicted as a server in a network path from one video communication terminal to another video communication terminal.

In each of FIGS. 3A , 3B, and 3C , NAL header analyzer 210 may provide packet/frame priority to local cache 235 via packetizer 310 . For an initial transmission, analyzer 210 outputs the analyzer data and priorities to packetizer 310 . Via path 312 a copy is placed in the local cache 235 and via path 314 the packetizer sends the packetized data to the depacketizer 320 of the streaming server. Network monitor 270 provides network traffic status data to retransmission decider 275 to selectively decide whether to retransmit packets stored in local cache 235 for retransmission on path 316 via network 110 to depacketizer 320 /frame. Path 316 is used to retransmit selected packets based on traffic conditions and analyzer-specified priorities.

Referring now to FIG. 4 , the abstraction layer header analyzer, network monitor and Functional operation of the retransmission decider. In block 400, the server/transmitter of FIG. 3 retrieves, for example, data from a local file, according to FIG. 3B, file reader 302 or receives data from a network interface, e.g., from network 110 and depacketizer 304 of FIG. 3C . The received data is provided to block 410 representing abstraction layer header analyzer 210 of FIGS. 2A and 3 .

Referring briefly to FIG. 5, there is shown a typical abstraction layer header providing fields to be analyzed, eg, an SVC NAL extension header. For example, abstraction layer header analyzer 410 may analyze positions 1-3 of the second byte (byte 1) that are shown to occur at a level representing inter-layer coding dependencies or spatial/resolution scalability of the layer representation DID field on the . The QID field, for example at positions 4-7 of the second byte (byte 1), may follow, representing the quality level indicated by the Medium Granular Scalability (MGS) layer. The TID field at, for example, position 0-2 of the third byte (byte 2) represents the temporal level of the layer representation. One result of analyzing an abstraction layer header containing such a field is to identify the layer to which the subsequent payload data is associated. In response to identification of payload data types, the analyzer at analysis block 410 of FIG. 4 may assign different priorities to each identified layer of data: eg, high, medium, and low priority. More than three levels of priority may be applied in alternative embodiments. For example, the base layer may receive high priority for retransmission from a buffer or cache since the base layer is absolutely necessary for full decoding at the receiver. Medium priority may be assigned to enhancement layers, since enhancement layer data is necessary for decoding of higher layers. An SVC encoded stream of digital data may contain a base layer and an enhancement layer. On the other hand, in order to enjoy a higher picture quality, a second or higher enhancement layer may be provided, so a lower priority may be assigned to the higher enhancement layer in the cache 235 for the relevant payload data.

To provide a simplified example, if SVC encoded video data has a base layer resolution of 416×240 and a bit rate of 600 kbps and a higher resolution of 832×480 and a random bit rate of 1.2 megabits per second (Mbps) for the video stream A base layer of the bit rate, then, the analyzer 410 can identify the base layer NAL unit according to its DID field and assign a higher priority to the base layer. On the other hand, enhancement layers can be assigned lower priority (in order to provide higher resolution). In this example the base layer is assigned a higher priority than the enhancement layer. The output priority level may be indicated in the Payload Type (PT) field of the RTP header shown in FIG. 6 at positions 9-15. Once a priority is specified, e.g., high, medium, and low, the priority information can also be indicated in a similar field of the Real-Time Protocol (RTP) header if retransmission is required at the transport layer, or in the Type of Service header of the IP header The priority information is indicated in the (TOS) field. Initially, as the packet is sent to the receiver via "Send Packet" 450 via the network 110 , a copy is cached in a local cache/buffer at 430 in block 420 . Packets may be lost in the network or received. Packets in local cache 430 await a determination regarding network conditions and whether the packet was received.

If the packet is received and network conditions appear to be good, the packet is received at 460 and the receiver determines at 465 that the expected packet was lost (Yes) or not (No). If lost (Yes), a retransmission request is sent at 485 and the receiver's recovery timing is set back into the sender/transmitter/server 200 via the control channel 255 of FIG. 2B. If not, then in 490 the received packets are placed in the receive buffer in the proper order, eg, for the display function 206 on the receiver/client 201 .

If a retransmission request is sent via 485, the network monitor function 405 is activated and then the received retransmission request is processed. As further described herein, the end-to-end packet loss rate at a given time can be determined on the network monitor 271 as current network 110 conditions from the receiver side. The current network conditions, eg, end-to-end packet loss ratio are provided to the retransmission decider 415 as an output of 405 together with the retransmission request.

An exemplary application of the network monitor 405 could be that the sender/server is on the wired network 110, e.g. a set-top box (cable or satellite) or a home gateway and the receiver is a mobile device or a personal computer associated with a wireless access point AP Case. Intermediate nodes/wireless access points (AP/routers) can report network and wireless channel conditions to the sender. Thus, real-time packet retransmission decisions made by the Reliable Media Protocol (RMP) module 230 according to the principles of the present invention may be layer-aware when modified to be incorporated into the abstraction layer header analyzer and method according to one embodiment. of.

The retransmission decider 415 asks, for example, whether the current end-to-end transmission packet loss rate is high, meaning whether it is above a threshold level set in the memory of the RMP module 230 . If the answer is "yes", then the lower priority packets are dropped at 445 and only the higher priority packets are retrieved from cache/buffer 430 and retransmitted. Other measures of transmission conditions besides packet loss rate may be available bandwidth (e.g., lowest available bandwidth in an end-to-end path) and round-trip delay (longer round-trip delays may require storage in cache/buffer for retransmission) Pause of packets in device 235). Each of these transmission conditions, including available bandwidth, may be compared to an associated threshold level set in memory to decide to retransmit the packet. One or more transmission conditions may be applied in making the retransmission decision and in the packet prioritization set by analysis 410 . If a negative answer is returned, then at 435 all request packets are resent from cache/buffer 430 via "Send Packets" 440 and network 110 .

Once received, the retransmission packet is received at 460 and identified as a retransmission packet at 470 . Given the live viewing experience, check the reception time on the 475. Ask: Too late to repost? If the retransmission packet is received too late, that is, the program viewing experience has shifted to the next frame, then the retransmission packet is discarded at 480 . On the other hand, if the retransmitted packet is not too late (ie, the answer is no), then at 490 the retransmitted packet is placed in the receive buffer in order for display.

The RMP method of the present invention can be implemented in flexible software library, hardware, firmware, any computer or processor, application specific integrated circuit (ASIC), reduced instruction set computer (RISC), field programmable gate array (FPGA) or their combination accomplish. The RMP method of the present invention uses a socket-like user-space API and underlying transport for easy integration with streaming server and player applications. The RMP method of the present invention is transparent to the streaming applications it supports. Internally maintain UDP data channel and TCP control channel. The RMP method of the present invention is extensible to support other error correction schemes like FEC and hybrid ARQ.

The network monitor function 405 of the present invention is now further described with reference to FIGS. 6 , 7 and 8 . One possible approach is shown in Figure 6, where the network monitor 271 can detect sequence gaps eg from the sequence number, ie the 6-1 position of the first row. Another approach is to use a Continuation Count (CC) field of a one-byte MPEG-2 transport stream header as shown in FIG. 7 .

An exemplary format of a message to send statistics collected by network monitor 271 of receiver/client 201 of FIG. 2B to server/sender 200 of FIG. 2A is shown in FIG. 8 . The message format may be similar to an RTP Control Protocol (RTCP) Receiver Report (RR) message. In particular, for example, a "loss ratio" field may be provided on the 4th row at positions 0-7. Other relevant data may also be provided such as the cumulative number of lost packets which may be compared with previously received cumulative numbers of lost packets received on the same channel. The difference provides the number of packets lost in the interval defined by the time between successive messages being successfully received.

FIG. 9 provides a format for a sender/server/transmitter to obtain network packet loss through a retransmission request received from a receiver/client 201. The network monitor function 405 of Figure 4 calculates the number of lost packets from the "Basic Sequence Number of Starting Packet" and "Offset of Ending Packet" fields. Similarly, the RMP ARQ request (NACK) packet of Figure 10 provides a base sequence number that can be stored and compared with a subsequently received base sequence number. The depicted base sequence numbers and bitmap indicate the sequence of receiver requested packets to be retransmitted. Yet another means of determining network traffic conditions is to monitor lower layer eg network/MAC/PHY layer transmission statistics to estimate channel conditions and available bandwidth.

In the RMP scheme of the present invention described above, no changes are made to the packets sent on data channel 255 . Therefore, backward compatibility is maintained. Furthermore, the RMP scheme of the present invention utilizes bandwidth efficiently, since lost media packets are only requested and retransmitted on low-overhead control channels. The lost packet request serves as a NACK (Negative Acknowledgment) and also provides feedback to the sender. Because lost packets can be retransmitted multiple times within the recovery time window, high reliability can be provided under widely varying channel conditions. Furthermore, the RMP scheme of the present invention enforces application delay constraints by having retransmissions with a maximum latency (ie, recovery window), thus acting on a best-effort delivery model given time constraints.

Note that the above embodiments are described using video transmission. But the invention can also be applied to the transmission of audio, for example, telephony, and other real-time multimedia streaming applications.

Although the above scheme of the invention has been described for a wireless network, the scheme can also be used in any type of network involving packet loss.

It should be understood that the present invention can be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof. Preferably, the invention is implemented as a combination of hardware and software. Furthermore, the software is preferably implemented as an application program tangibly embodied on a program storage device. Applications can be uploaded to and executed by a machine containing any suitable architecture. Preferably, the implementation is implemented on a computer platform containing hardware such as one or more central processing units ("CPUs"), random access memory ("RAM"), and input/output ("I/O") interfaces. machine. The computer platform also includes an operating system and microinstruction code. The various processes and functions described herein may be part microinstruction code or part application program (or a combination thereof) executable via the operating system. In addition, various other peripheral devices, such as additional data storage devices and printing devices, may be connected to the computer platform.

It should also be understood that because some of the subsystem components and method steps depicted in the figures are preferably implemented in software, the actual connections between system components (or process steps) may vary depending on the manner in which the invention is programmed. Given the teachings herein, one of ordinary skill in the relevant art can contemplate similar implementations or configurations of the present invention.

Claims (6)

1. a method, including:

Receive the video data of coding from digital video data source, the video data of described coding is scalable coding and takes Band video content, the video data of described coding includes network abstract layer NAL extension header, for the first of described video content Layer, and the second layer for described video content, described ground floor includes being used for from the video data of described coding obtaining First video data of the video content of gross, the described second layer includes the use being used together with described first video data The second video data of the video content strengthening quality is obtained in the video data from described coding；

Resolve NAL extension header to identify described first video data and described second video data；

Described first video data will be distributed to relative to the described second higher priority of video data be used for retransmitting；

In buffer storage, caching has the first video data and second video data of the priority of distribution；

Via the first data channel of digital data network to the video data of coding initially sent receptor, described numeral Data network also include the second data channel, described first data channel utilize the first host-host protocol and provide from conveyer to The one way data communication of described receptor, described second data channel utilize the second host-host protocol and provide described conveyer and Bidirectional data communication between described receptor, the reliability of described first host-host protocol can less than described second host-host protocol By property；

Monitor digital data network with from the angular collection of described conveyer about described digital data network statistics first Network data, described first network data instruction first network status transmission, described first network status transmission includes the first end Opposite end packet loss rate, the first available bandwidth of described digital data network, and at least one in the first round-trip delay；

The repeat requests transmitted via receptor described in described second data channel reception, described repeat requests includes connecing from described Receiving the second network data of the statistics of the relevant described digital data network of the angle of device, described second network data indicates in institute Stating the second network status transmission of the described digital data network determined at receptor, described second network status transmission includes Two port end packet loss rate, the second available bandwidth of described digital data network, and at least in the second round-trip delay ?；

Utilize described second network data that described first network data are supplemented；And

Determine whether to recover at least described first video data and described second video data from described buffer storage Individual and whether in response to described repeat requests in view of the first network data after supplementing and according to the priority warp distributed By the second data channel, the video data recovered is sent to described receptor.

Method the most according to claim 1, wherein, described digital video data source includes MP4 document reader and packet One of de-packetizer.

Method the most according to claim 1, wherein, described first host-host protocol includes UDP UDP, described Second host-host protocol includes transmission control protocol TCP.

Method the most according to claim 1, wherein, described digital data network include satellite network, roadbed wireless network, At least one in fiber optic network, coax network.

Method the most according to claim 3, wherein, described digital data network include satellite network, roadbed wireless network, At least one in fiber optic network, coax network.

Method the most according to claim 1, wherein said network abstract layer NAL extension header includes H.264/AVC network The NAL extension header of level of abstraction.