CN100466725C - Multimedia communication method and terminal thereof - Google Patents

Abstract

The invention relates to multimedia communication technology, discloses a multimedia communication method and the design and realization of the terminal, which realizes unequal protection and facilitates the realization of QoS guarantee. The present invention adopts ERRTP on the basis of the existing RTP protocol to provide a transport layer encapsulation format that can carry information related to the error elastic coding scheme, so that the multimedia data is transmitted on the ERRTP while marking its corresponding error elastic coding scheme information, thereby making the error elastic The mechanism is integrated into the transport layer; for the H.264 NALU structure, a special ERRTP encapsulation method and a modification scheme for the protocol header information are given, which can combine the header information bytes of all NALUs in the same ERRTP packet into its header information, so that The important information of NALU is reflected in the ERRTP header information, and the transmission efficiency is improved.

Description

Multimedia Communication Method and Terminal

technical field

The invention relates to multimedia communication technology, in particular to a multimedia communication technology supporting error resilience.

Background technique

With the rapid development of computer Internet (Internet) and mobile communication network, the application of streaming media technology is becoming more and more extensive, from online broadcasting, movie playing to distance teaching and online news websites, etc., all use streaming media technology. Currently, there are mainly two ways to transmit video and audio on the Internet: Download and Streaming. Streaming transmission is the continuous transmission of video/audio signals, and when the streaming media is played on the client computer, the rest continues to download in the background. There are two ways of streaming: Progressive Streaming and Realtime Streaming. Live streaming is real-time delivery, especially suitable for live events, live streaming must match the connection bandwidth, which means that the image quality will be degraded by lower network speeds to reduce the need for delivery bandwidth. The concept of "real-time" means that the delivery of data in an application must maintain a precise time relationship with the generation of data.

Especially with the emergence of the third generation mobile communication system (3rd Generation, referred to as "3G") and the rapid development of networks based on Internet Protocol (Internet Protocol, referred to as "IP"), video communication is gradually becoming one of the main services of communication. one. And two-party or multi-party video communication services, such as videophone, video conferencing, mobile terminal multimedia services, etc., put forward strict requirements on the transmission of multimedia data streams and service quality. Not only requires better real-time performance of network transmission, but also requires higher efficiency of video data compression and encoding.

In view of the current needs of media communications, the International Telecommunication Union Telecommunications Standardization Sector ("ITU-T") formulated video compression standards such as H.261, H.263, and H.263+ in 2003. Officially released the H.264 standard. This is a high-efficiency compression coding that is jointly formulated by ITU-T and the Moving Picture Experts Group (MPEG) of the International Standardization Organization (International Standardization Organization, "ISO") to meet the needs of network media transmission and communication in the new stage. standard. It is also the main content of Part 10 of the MPEG-4 standard.

The purpose of formulating the H.264 standard is to more effectively improve video coding efficiency and its adaptability to the network. In fact, due to its superiority, the H.264 video compression coding standard has gradually become the mainstream standard in current multimedia communication. A large number of H.264 multimedia real-time communication products (such as conference TV, videophone, 3G mobile communication terminals) and network streaming media products have come out successively. Whether or not to support H.264 has become a key factor in determining product competitiveness in this market field . It can be predicted that with the formal promulgation and widespread use of H.264, multimedia communications based on IP networks and 3G and post-3G wireless networks will inevitably enter a new stage of rapid development.

The following is a brief introduction to the message composition and sending mechanism of the H.264 standard: The H.264 standard adopts a layered mode, which defines the Video Coding Layer (Video Coding Layer, referred to as "VCL") and the Network Abstraction Layer (Network Abstraction Layer, referred to as "NAL") ”), the latter is specially designed for network transmission, and can adapt to video transmission in different networks, further improving the "affinity" of the network. H.264 introduces an IP packet-oriented coding mechanism, which is beneficial to packet transmission in the network and supports streaming video transmission in the network; it has strong anti-error characteristics, and is especially suitable for wireless video with high packet loss rate and serious interference send request. All data to be transmitted in H.264, including image data and other messages, are encapsulated into a uniform format for packet transmission, that is, a network abstraction layer unit (NAL Unit, "NALU" for short). Each NALU is a variable-length byte string of certain syntax elements, including header information containing one byte, which can be used to indicate the data type, and several integer bytes of payload data. A NAL unit can carry a coded slice, data partition of the respective type or a sequence or picture parameter set. In order to enhance data reliability, each frame of image is divided into several slices (Slice), each Slice is carried by a NALU, and Slice is composed of several smaller macroblocks, which is the smallest processing unit. Generally speaking, the Slices at the corresponding positions of the previous and subsequent frames are related to each other, and the Slices at different positions are independent of each other, so as to avoid the mutual diffusion of bit errors between the Slices.

H.264 data includes non-reference frame texture data, sequence parameters, image parameters, Supplemental Enhancement Information (Supplemental Enhancement Information, referred to as "SEI"), reference frame texture data, etc. Wherein, the SEI message is a general term for messages that play an auxiliary role in decoding, displaying and other aspects of the H.264 video. Various SEI messages are defined in the prior art, and SEI reserved messages are reserved at the same time, leaving room for expansion for various possible applications in the future. According to H.264, SEI messages are not necessary to reconstruct luma and chrominance images during the decoding process. A decoder that conforms to the H.264 standard does not need to do any processing on the SEI. In other words, not all terminals that meet the basic requirements of H.264 can process SEI messages, but for terminals that cannot process SEI messages, sending SEI has no effect on it, and it will simply ignore the SEI that it cannot process information. According to SEI syntax rules, users can use reserved messages to transmit custom messages to realize function extension.

The following first introduces the structure and grammar of the SEI message and other related materials. H.264 provides a variety of mechanisms for message extension, including SEI. Supplemental Enhancement Information (SEI) is defined in H.264. Its data representation area is independent of video coding data. Its usage method is given in the description of NAL in the H.264 protocol. The basic unit of the H.264 stream is the NALU, which can carry various H.264 data types, such as video sequence parameters (Sequence parameters), image parameters (Picture parameters), Slice data (that is, specific image data), and SEI messages data. SEI is used to deliver various messages and supports message extensions. Therefore, the SEI domain is used to transmit messages customized for specific purposes without affecting the compatibility of the H.264-based video communication system. The NALU carrying the SEI message is called SEI NALU. An SEI NALU contains one or more SEI messages. Each SEI message contains some variables, mainly payload type (payloadType) and payload size (payloadSize), these variables indicate the type and size of the message payload. The syntax and semantics of some commonly used H.264SEI messages are defined in H.264Annex D.8 and D.9. The payload contained in the NALU is called Raw-Byte Sequence Payload ("RBSP" for short), and SEI is a type of RBSP.

The data representation area of the SEI is referred to as the SEI domain for short. Each SEI field contains one or more SEI messages, and SEI messages are composed of SEI header information and SEI payload. The SEI header information includes two fields: one gives the type of payload in the SEI message, and the other gives the size of the payload. When the load type is between 0 and 255, it is represented by one byte 0x00 to 0xFE; when the type is between 256 and 511, it is represented by two bytes 0xFF00 to 0xFFFE; when the type is greater than 511, the representation method is deduced by analogy. Users can customize any number of load types. Among them, type 0 to type 18 have been defined in the standard as specific information such as cache period, image timing, etc. It can be seen that the SEI field defined in H.264 can store enough user-defined information according to requirements. For H.264 decoders that do not support parsing these user-defined information, the data in the SEI field will be discarded automatically. Therefore, recording useful custom information in the SEI field will not affect the compatibility of the H.264-based video communication system.

As mentioned earlier, multimedia communication not only requires high media compression and coding efficiency, but also requires real-time network transmission. At present, multimedia streaming basically adopts Real-time Transport Protocol (Real-time Transport Protocol, referred to as "RTP") and its control protocol (Real-time Transport Control Protocol, referred to as "RTCP"). RTP is a transmission protocol for multimedia data streams on the Internet, released by the Internet Engineering Task Force (Internet Engineering Task Force, referred to as "IETF"). RTP is defined to work in the case of one-to-one or one-to-many transmission, and its purpose is to provide time information and realize stream synchronization. The typical application of RTP is based on the User Datagram Protocol ("UDP" for short), but it can also be used in the Transmission Control Protocol (Transport Control Protocol, "TCP") or Asynchronous Transfer Mode (Asynchronous Transfer Mode, "ATM" for short). ) and other protocols to work on.

RTP itself only guarantees the delivery of real-time data, and does not provide a reliable delivery mechanism for in-order delivery of data packets, nor does it provide flow control or congestion control. It relies on RTCP to provide these services. RTCP is responsible for managing the quality of transmission and exchanging control information between current application processes. During the RTP session, each participant periodically transmits RTCP packets, which contain statistical data such as the number of data packets sent, the number of lost data packets, etc. Therefore, the server can use this information to dynamically change the transmission rate, or even Change payload type. The combination of RTP and RTCP can optimize the transmission efficiency with effective feedback and minimum overhead, so it is suitable for transmitting real-time data on the Internet.

The transmission of H.264 multimedia data on the IP network is also based on UDP and its upper layer RTP protocol without exception. RTP itself can be applied to different media data types in structure, but in multimedia communication, different high-level protocols or media compression coding standards (such as H.261, H.263, MPEG-1/-2/-4, MP3 etc.), the IETF will formulate a specification file for the RTP payload (Payload) packaging method of the protocol, specifying the RTP packaging method in detail, which is optimized for this specific protocol. Similarly, the corresponding IETF standard for H.264 is RFC 3984: RTPPayload Format for H.264 Video. This standard is currently the main standard for the transmission of H.264 video streams on IP networks, and is widely used. In the field of video communication, the products of all mainstream manufacturers are based on RFC 3984, which is currently the only H.264/RTP transmission method.

In fact, the key difference between H.264 and other previous video compression coding protocols is that H.264 defines a new layer called Network Abstract Layer (NAL). In order to increase the separation and independence of its video coding layer (Video Coding Layer, referred to as "VCL") and the following specific network transmission protocol layer, H.264 brings greater application flexibility and defines a new level of NAL. This layer does not exist in ITU-T's early video compression coding protocols such as H.261, H.263/H.263+/H.263++. However, how to design a more efficient and better solution based on the advantages of H.264 in the cooperative work of NAL and RTP protocol bearer, so that RTP can better bearer performance for H.264, is a problem worthy of research and has great applications. question of value.

The method of RTP carrying H.264 NAL layer data proposed by the RFC 3984 specification is the current mainstream transmission method. On the basis of the RTP protocol (RFC 3550), the NAL layer data is encapsulated in the RTP payload for carrying. The NAL layer is located between VCL and RTP, and it is stipulated that the video stream should be divided into a series of NAL data units (NAL Units, referred to as "NALU") according to the defined rules and structures. The encapsulation format of RTP payload for NALU is defined in RFC 3984. The frame format of the RTP and the encapsulation method of the NALU in the prior art are briefly introduced below in sequence.

The main purpose of RTP design is real-time multimedia conference and continuous data storage, interactive distributed simulation, control and measurement applications, etc. RTP is usually carried on top of UDP protocol to take advantage of its multiplexing and verification functions. RTP supports multi-address delivery if the underlying layer provides multi-point distribution. The functions provided by RTP include: load type identification, sequence number, time stamp, and transmission monitoring.

The packet format of RTP is as follows: the basic options of RTP header information occupy 12 bytes (minimum case), while the header information of IP protocol and UDP protocol occupy 20 bytes and 8 bytes respectively, so RTP packets are encapsulated in UDP packets and then encapsulated in IP packets. In the package, the total number of bytes occupied by the header information is 12+8+20=40 bytes. The detailed structure of the header information of the RTP packet is shown in FIG. 1 .

The RTP header information shown in Fig. 1 from front to back is as follows: the first byte (byte 0) is some fields about the header information structure itself, the second byte (byte 1) is to define the payload type, and the third byte , 4 bytes (bytes 2 and 3) are the Sequence Number, the 5th to 8th bytes are the timestamp, and the 9th to 12th bytes are the Synchronous Source Identifier (Synchronous Source Identifier, referred to as "SSRCID"), and finally a list of Contributing Source Identifiers ("CSRCIDs"), the number of which is indeterminate. Note that the first byte in this description is marked byte 0, and so on.

Among them, the first 12 bytes appear in all different types of RTP packets, while other data in the header information, such as the contribution source identifier, is only available when the mixer is inserted. Therefore, CSRC is generally used when there is media mixing. For example, in a multi-party conference, audio needs to be mixed, and video can also use this method to provide multi-screen functions. The synchronization source identifier SSRC is actually the identifier of the carried media stream.

The specific meanings and full names of the above fields are described as follows:

The V field is version information, which occupies 2 bits. The currently used version is 2, so set V=2, and other values such as V=1 represent an earlier RTP version, and V=0 represents the most original The predecessor of RTP, that is, adopted in the Voice over IP (VOIP) communication system used on the early Mbone network, later evolved into RTP, and V=3 has not yet been defined.

The P field is the padding flag (Padding), which occupies 1 bit. If P is set, it means that the end of the data packet contains one or more padding bytes (Padding), and the padding is not part of the payload;

The X field is the extension identification bit (Extension), which occupies 1 bit. If X is set, the end of the RTP header must be followed by a variable-length header extension (if there is a CSRC list, the header extension must follow it), mainly Reserved for some application environments where the header information field is not enough, the header information extension contains a 16-bit length field to count how many 32-bit long words there are in the extension, the first 16 bits of the header extension are left open, In order to distinguish between identifiers and parameters, the format of these 16 bits is defined by specific layer specifications, and the format definition of this header extension is described in detail in Section 5.3.1 of RFC 3550.

The CC field is the number of contributing sources (CSRC Count), which occupies 4 bits and indicates the number of CSRC identifiers at the end of the header information. The receiver can determine the length of the CSRCIDs list behind the header information according to the CC field;

The M field is the marker bit (Marker), which occupies 1 bit. The interpretation of the marker bit is defined in a specific profile (Profile), which allows to identify important events in the data packet flow. A layer can define additional marker bits or regulations. There is no identification bit. The so-called level here refers to the specific application environment settings, which are specifically agreed by the two parties in communication and are not limited by the agreement;

The PT field is the payload type (Payload Type, referred to as "PT"), a total of 7 bits, which identifies the format of the RTP payload and determines its interpretation in the application; the flag bit and the payload type share one byte to carry layer-specific information, this word Sections may be redefined at the specific level to meet different needs. In specific applications, the so-called profile can be defined. match the media format. Of course, the relationship between the PT value and the media format can also be defined through dynamic negotiation through signaling other than RTP. In an RTP session (Session), the RTP source can change the PT.

The next field is the serial number with a total of 16 bits. Every time an RTP data packet is sent, the serial number value is increased by one, so that the receiver can use it to detect data packet loss and restore the order of data packets. The initial value of the serial number in a communication can be given randomly. , does not affect communication.

The timestamp occupies 32 bits, and it reflects the sampling time of the first byte in the RTP packet. The sampling time here must come from a clock that increases monotonically and linearly, and the receiver adjusts the media playback time or performs synchronization according to it.

The synchronization source SSRC ID occupies 32 bits, and its specific value can be selected randomly, but the uniqueness in the same RTP session must be ensured, that is, a media source can be uniquely identified. If a source changes the source transmission address, a new SSRC must be selected identifier.

The contribution source CSRC list can be 0-15 items according to the needs, and each item occupies 32 bits. The length of the list, that is, the number of CSRC IDs, is just marked by 4 bits of the CC field. In fact, the CSRC identifier used to identify a certain media source is consistent with the SSRC identifier of the corresponding contribution source, except that different receivers have different roles and are set as SSRC or CSRC. In multiparty communication, the CSRC ID is inserted by the mixer.

In the case of carrying H.264 video, RTP encapsulates the NALU of H.264 into an RTP packet stream. The NALU is mainly defined in the RFC 3984 file, and based on this, the packaging format of the H.264 layer NAL data in RTP is given. The RTP encapsulation format of this NALU is shown in FIG. 2 .

Figure 2 shows the encapsulation structure of a NALU in the RTP payload. The first byte in the front is the NALU header information, followed by the data content of the NALU. Multiple NALUs are filled end to end into the payload of the RTP packet. , there is optional RTP padding at the end, which is the content specified in the RTP packet format, in order to make the length of the RTP packet meet certain requirements (such as reaching a fixed length), and the optional RTP padding data is generally filled with zeros.

The NALU header information is the first byte, also known as an octet (Octet), which has three fields in total. The meaning and full name are described as follows:

The F field is defined as a forbidden bit (forbidden_zero_bit), which occupies 1 bit and is used to identify syntax errors, etc. If there is a syntax conflict, it is set to 1. When the network recognizes that there is a bit error in this unit, it can be set to 1 , so that the receiver can discard the unit, it is mainly used to adapt to different types of network environments (such as the combination of wired and wireless environments);

The NRI field is defined as the NAL reference identifier (nal_ref_idc), which occupies 2 bits and is used to indicate the importance of the NALU data. Its value is 00, indicating that the content of the NALU is not used to reconstruct the reference image for inter-frame prediction, and not 00, indicating that the current NALU is Important data such as slices or sequence parameter sets (Sequence Parameter Set, referred to as "SPS") and picture parameter sets (PictureParameter Set, referred to as "PPS") belonging to the reference frame. The larger the value, the more important the current NAL is;

The Type field is defined as the NALU type (Nal_unit_type), with a total of 5 bits, and there may be 32 types of NALUs. The corresponding relationship between its values and specific types is given in Table 1 in detail.

Table 1 Correspondence between Type field value and type in NALU header information

Type value Types of NALU content 0 not specified 1 Encoded slices for non-IDR images 2 Encoding slice data partition A 3 Encoding slice data partition B 4 Encoding slice data division C 5 Encoded slices in IDR images 6 SEI (Supplementary Enhancement Information) 7 SPS (Sequence Parameter Set) 8 PPS (Picture Parameter Set) 9 access unit delimiter 10 end of sequence 11 end of stream 12 Data input 13-23 reserve 24-31 not specified

It can be seen that the information given in one byte of the header information of the NALU mainly includes the validity and importance level of the NALU, and the importance of the data carried by the RTP can be determined according to these information.

Under the current H.264/RTP multimedia communication framework, the quality of service (Quality of Service, referred to as "QoS") monitoring, and congestion control and flow control based on this are mainly completed by using the cooperative control protocol RCTP. RTCP is mainly used for control and reporting of the RTP protocol. The main content of the report is the information related to QoS. The reporting method adopted by RTCP is periodic reporting, that is, the control data packet is periodically transmitted to all participants in the two-party or multi-party session (Session), and the report adopts the same distribution mechanism as the RTP data packet. The underlying protocol provides multiplexing of data and control packets (eg each using a separate UDP port number, etc.). In the RFC 3550 document, it is recommended to increase the session bandwidth for RTCP to be 5% of the media bandwidth.

The types and structures of RTCP packets are introduced below. The following RTCP packet types are defined in RTCP to carry a variety of control information: sender report (Sender Report, referred to as "SR"), statistical information about the transmission and reception of the active sender; receiver report (Receiver Report, "RR" for short) receives statistical information from a participant that is not an active sender; resource description item (SourceDescription, "SDES" for short), which includes CNAME; participant end (exit) identifier (BYE); special function (Application-specific function, referred to as "APP").

The packet structure of the RTCP sending and receiving report is shown in Figure 3. It can be divided into three sections according to the content type. The first is the header information, followed by the sender information, followed by the report content block, and the last is the specific level (Profile) The extension of (the so-called level represents a special case of specific rules formulated for a specific application scenario). The meaning and full name of each specific field shown in Fig. 3 are described in detail as follows:

The V field is version information (Version, referred to as "V"), which occupies 2 bits, and the current RTCP version number is V=2;

The P field is the padding flag bit (Padding), which occupies 1 bit. If P is set, it means that the RTCP packet contains some additional padding bytes that are not control information at the end, but these bytes are calculated in the length field;

The RC field is the reception report count (Reception Report Count, referred to as "RC"), which occupies 5 bits, and the number of reception report blocks contained in the data packet is allowed to be 0;

The PT field is the data packet type (Payload Type, referred to as "PT"), which occupies 8 bits. When the value is 200, it indicates that this is an SR data packet;

The Length (Length) field, which occupies 16 bits, is equal to the length of the RTCP packet in units of 32-bit words (32bitWord) minus 1, including the header and any padding;

The sender's SSRC, which occupies 32 bits, indicates the Synchronous Source Identifier (Synchronous Source Identifier, referred to as "SSRC") of the originator of the SR data packet. The synchronization source here uniquely identifies a media data source, such as the source of a video;

The NTP timestamp field is the Network Time Protocol timestamp (Network Time Protocol, referred to as "NTP"), which occupies 64 bits. When the report is sent, it indicates the wallclock (absolute date and time), which is used in combination with the RTP timestamp;

The RTP timestampe field is the RTP timestamp, which occupies 32 bits, which is the timestamp generated by the RTP protocol;

The data packet count field of the sender, which occupies 32 bits, indicates the total number of RTP data packets transmitted by the sender during the period from the establishment of the transmission to the generation of the SR data packet;

The sender byte count field, a total of 32 bits, indicates the total number of bytes (not including header or padding) transmitted by the sender in the RTP data packet during the period from the establishment of sending to the generation of this SR data packet. can be used to estimate the average velocity of the load;

The following fields contain 0 or more reception report blocks. The specific number of blocks depends on the number of other sources known from the sender of the previous report. Each reception report block conveys the number of RTP packets received from a single synchronization source. Statistics, which include:

Fragment loss (fraction lost) occupies 8 bits, indicating the number of lost fragments of the media from the source after the last report was sent; the cumulative number of lost packets, accounting for 24 bits, indicating the cumulative number of lost packets since the start of reception;

The second is the received extended maximum sequence number and arrival delay jitter, which all reflect the network transmission status;

The last SR (Last SR, referred to as "LSR") occupies 32 bits, which refers to the timestamp mark of the previous SR report on the source, and the value is the middle 32 bits of the NTP of the previous SR;

The delay since the last SR (Delay since Last SR, referred to as "DLSR"), which occupies 32 bits, refers to the length of the time interval from the last SR to this SR. This parameter is a key parameter used to calculate the QoS report .

The difference between the reception report (RR) packet format and the transmission report (SR) is: the value of the packet type field is 201; there is no sender information part.

According to the RTP/RTCP protocol standard, RTCP completes four functions as follows:

The basic function is to provide a feedback reporting mechanism for the quality of real-time multimedia data transmission. This is an integral part of RTP as a transport layer protocol. This feedback function is realized by transmitting the sender report (SR) and receiver report (RR) through RTCP;

RTCP transmits a permanent transport layer identifier for each RTP source, called the canonical name (CanonicalName, referred to as "CNAME"). The SSRC identifier may change when a conflict is found or the program is restarted, so the receiver needs to use CNAME to track each participant. square;

The first two functions require all participants to send RTCP packets, so in order for RTP to increase the number of participants proportionally, the rate of RTCP packets must be controlled;

The fourth item is an optional function, that is, to transmit as little control information as possible.

It can be seen that the RTCP protocol is used to transmit the QoS report, and the QoS information is reported according to the report content stipulated by the RTCP protocol, based on which the QoS monitoring of the bearer media such as H.264 is realized.

However, it should be pointed out that while RTCP can provide a QoS reporting mechanism, because of the periodic reporting method, the overhead of additional network bandwidth can reach up to 5%. If the network is congested and the delivery QoS drops, the extra traffic generated by RTCP will make the problem worse.

After understanding the transmission structure of H.264/RTP and its RTCP-based quality of service monitoring, the following briefly introduces the error resilience and related technical background of video network transmission.

H.264 video is the main protocol of multimedia communication in the future, and the network of future multimedia communication application is mainly packet switching network and wireless network represented by IP. Neither of these two types of networks can provide a good Quality of Service ("QoS") guarantee, so the video transmission on the network will inevitably be affected by various transmission errors and packet loss, thereby reducing the communication quality. One of the most important problems here is that the IP network realizes "best effort" (best effort) transmission, which cannot guarantee the QoS of transmitting video data. Especially for the H.264 code stream that has been compressed and encoded with high efficiency, the problem is more prominent. Best-effort delivery over IP networks cannot guarantee the QoS of real-time video communication, which is manifested in three aspects: packet loss, delay and delay jitter. Among them, data packet loss has the greatest impact on the quality of the restored video. Since the H.264 compression coding algorithm uses motion estimation and motion compensation technology, once there is data packet loss, it will not only affect the current decoded image, but also affect the subsequent decoded image, namely Error propagation. Bit error diffusion has a great impact on the restored video quality. Only by combining the encoding end and the decoding end to jointly resist bit errors can the bit error diffusion be completely avoided.

Error resilience (Error Resilience) means that the transmission mechanism has the ability to prevent errors from occurring or to correct them with a certain ability after an error occurs (error intensity within a certain range can be completely corrected; beyond a certain range, it can only be partially corrected). In the future extensive (it can be said to be ubiquitous) multimedia communication environment, whether a video transmission mechanism has error resilience will be very critical.

There are a variety of error resilience mechanisms, such as forward error correction (Forward Error Correction, referred to as "FEC"), automatic retransmission request (Automatic Retransmission Request, referred to as "ARQ"), error concealment (Error Concealment), source channel joint coding (Joint Source-Channel Coding, referred to as "JSCC"), interleaving (Interleaving), and eliminating error diffusion. For the transmission of H.264 video over the data packet network, FEC is a very practical technology, and the effect is very good. This method mainly uses a variety of error correction codes to encode the data to be protected, and the essence is to form data redundancy, thereby increasing the ability to resist errors.

The main error on the data packet network is the packet loss error, which is called erasure error in error correction coding theory. Error correction codes for erasure errors are a large class called erasure codes (Erasure Codes). The so-called erasure code is to divide the data code stream into units of the same size segment by segment, also called data nodes (Data Nodes). For convenience, it is assumed that there are n data nodes in total. Then calculate these data nodes according to certain mathematical operation rules to generate check nodes (Parity Nodes or Check Nodes). The second layer of check nodes, and so on, can generate the third layer, the fourth layer, and up to the Nth layer of check nodes.

Generally speaking, if multi-layer verification nodes are involved, the number of nodes on each layer is reduced according to a certain law (the most common is the law of proportionality) relative to the previous layer, so that a layer-by-layer shrinking multi-layer node structure. It can be visualized as a pyramid turned 90 degrees to the right. Among them, the leftmost layer is the data node layer, and the rightmost layer is the first layer of check nodes, the second layer of check nodes, ..., the Nth layer of check nodes.

One type of erasure code has a very important property, that is, the time complexity required for processing is linearly related to the number n of data nodes, so it is called linear-time characteristic. And many other erasure codes, such as the famous Reed-Solomon code, require a much higher time complexity, which is on the order of n ^* log2n ^* log(logn). Therefore, erasure codes with linear temporality are much better for real-time communication.

Tornado erasure codes (hereinafter referred to as Tornado codes) are a new type of erasure codes that appeared around 1998. The structure of Tornado code is simple; the operation is efficient because it has linear time; and the protection ability is strong. In practical applications, good results have been obtained. At present, it has been widely used. According to the latest ITU-T trends, SG16 is currently considering the possibility of standardizing Error Control Codes (Error Control Codes) technology, mainly for the protection of video and audio network transmission. Tornado code and its many variants are likely to be one of the important technologies.

In the Tornado code, multiple layers of check nodes are generated layer by layer from the data nodes. Both the check node and the data node are sent from the sender to the receiver through the network. If some nodes are lost during the network transmission process, because the upper layer nodes participated in the generation of the lower layer nodes, the information of the upper layer nodes has been included in the lower layer nodes and lower layer nodes, so the information of the lost nodes can pass through enough lower layer nodes node or lower-level nodes to fully recover. If each node is a packet, lost packets can be fully recovered by other packets received correctly. Let the number of data nodes be n, and the number of generated check nodes be 1. Then define the code rate and redundancy rate of erasure code to be respectively: r=n/(n+1), 1-r=1/(n+1); Under the same situation of other conditions (protection ability, the delay caused etc.), the higher the code rate (necessarily, the lower the redundancy rate), the higher the efficiency of the erasure code.

FIG. 4 shows a typical Tornado code data node and the relationship between check nodes of each layer. The connection between the nodes in the figure is called an edge, which means that the left node of the edge participates in the calculation of the right node. It can be seen that there is a many-to-many logical relationship between the nodes in the front and back layers. Assuming that the number of data nodes is n and the total number of check nodes is m, then define the code rate r=n/(n+m) and redundancy rate 1-r=m/(n+m) of the erasure code , under the same circumstances (protection capability, delay caused, etc.), the higher the code rate and the lower the redundancy rate, the higher the efficiency of the erasure code. The structure and performance of the Tornado code are mainly determined by three factors: (a) the number of data nodes and the law of layer-by-layer shrinkage, generally shrinking in equal proportions; (b) the calculation method for generating the next layer of nodes; (c) The association relationship between two adjacent layers of nodes.

The following relationship can be deduced between the various parameters of the Tornado code. The number of data nodes is set to n, the number of check nodes is set to m, the scaling ratio is set to p, and the number of layers of check nodes is i. The number of test nodes is np, np ² ,..., np ^i-1 respectively, and the number of the last layer, i.e. the i-th layer, is set as np ⁱ /(1-p), so that the total number of nodes n+m= n+n+np ² +..+np ^i-1 +np ⁱ /(1-p)=n/(1-p), then there is m=np/(1-p), which is the scaling ratio and Check the implicit relationship satisfied between the number of nodes. Because it is necessary to ensure that the number of nodes np, np ² , ..., np ^i-1 and np ⁱ /(1-p) of each layer are all integers, the feasible value of n can be calculated according to the given i and p, For example, i=4, p=1/2, then it can be deduced that n must be a multiple of 16.

The most commonly used calculation method in the process of Tornado code generation is XOR operation, because XOR operation has a very convenient recovery function. For two equal-length bit sequences A=[a ₀ , a ₁ , a ₂ ,..., a _L ], B=[b ₀ , b ₁ , b ₂ ,..., b _L ], carry out the exclusive OR operation by bit to obtain the same long bit sequence C, then have the following properties: A and C XOR obtain B, B and C XOR obtain A; similarly for the XOR operation between multiple sequences, There is also a corresponding recovery method. It can be seen that after the XOR operation, the data nodes or check nodes are connected to each other, and if any node is lost, it can be restored by all other nodes. Since the shrinkage ratio of the check nodes in the last layer is different, it is generally calculated using a conventional error correction coding strategy, such as a Reed-Solomon code.

Another important factor of the Tornado code is the relationship between the front and back layers, that is, a certain node in the lower layer is calculated from which nodes in the previous layer. According to graph theory, a bipartite graph is formed between nodes in the front and back layers. The two ends of any edge are in the previous layer and the next layer respectively. The nodes in the previous layer are also called left nodes, and the nodes in the latter layer are called right nodes. Side nodes, the number of edges associated with each node is called the degree. According to the mathematical proof of random graph theory by Luby et al., the parameters that determine the protection ability of Tornado codes are actually the degree vectors of the nodes on both sides of the bipartite graph composed of the front and rear layers, and this degree vector is randomly generated. In practical applications, before Tornado encoding, it is necessary to determine the random distribution of node degree vectors, and then randomly match the distribution to generate bipartite graphs at all levels. According to the correlation between the left and right nodes of the bipartite graph, the correlation between the front and rear nodes relation.

In the current Tornado code strategy, the parameters n, m, i, p, etc. are determined by given protection capabilities and other requirements, such as the rationality of the data node size, the maximum acceptable network delay, etc., and the given node degree vector Randomly distributed and can be Tornado encoded. When decoding at the receiving end, according to the bipartite graph at each level, if a right node is received correctly and only one node is lost among all the left nodes associated with it, then the lost node can pass through the right node. The node and all the left nodes that are not lost are recovered, that is, the effect of error correction is achieved.

In fact, the range of erasure codes is very large, and Tornado codes are just one of the more typical ones. In addition, there are RS (Reed-Solomon) codes, Low Density Parity Codes (Low Density Parity Codes, referred to as "LDPC") and so on.

An important performance index of erasure codes is its error correction capability (or protection capability), which is directly reflected in the maximum number of packet losses allowed to completely correct packet loss errors (under the premise of a certain total number of packets), or when the packet loss error is completely corrected. Percentage of correctly corrected packets above this maximum allowed number. Generally speaking, under other conditions being the same, the higher the protection capability, the higher the redundancy rate.

The protection ability is not only applicable to erasure codes, but in a larger scope, all FEC codes can be measured by the protection ability. In video data, some data are relatively important, such as structural parameters of video sequences, image structural parameters, header information, etc.; other data are relatively low in importance, such as image content data. When FEC is used for protection, a code with stronger protection ability is used for relatively important data; a code with weaker protection ability is used for relatively unimportant data. This allows for a balance between protection capability and efficiency. The protection ability cannot be emphasized blindly, because this will lead to high redundancy and sacrifice efficiency. This method of FEC protection with different protection capabilities according to the relative importance of data is called Unequal Protection ("UEP" for short). Through unequal protection, it is easy to realize the QoS guarantee of the video communication service.

The idea of unequal protection is to adopt protection mechanisms with different protection capabilities/protection strengths for data with different importance (relatively) in multimedia data. Different protection mechanisms can refer to major categories or subcategories. For example, the major categories are different in principle, and the small categories are only different in structure or parameters. Hierarchical protection is to divide the protection mechanism into multiple levels according to the protection ability. These levels can be across large categories, or across subcategories within a large category, that is, only the structure and parameters are different. In network transmission, network conditions change with time, and this change rule is complex and unpredictable. Therefore, hierarchical protection is actually an adaptive strategy, which is selected according to a set of predefined rules according to the instantaneous network conditions optimal protection. Unequal protection and hierarchical protection can be combined to form a more complex and powerful protection strategy.

In actual H.264 video communication, the degradation of image quality due to deletion errors caused by packet loss is very serious, even worse than causing the collapse of the decoding end system. This is because H.264 is more capable, more efficient, and more functional than other video coding standards, and in turn has lower tolerance for deletion errors. Therefore, in the video communication based on the H.264 standard, in addition to the error resilience protection strategy, effective anti-packet loss and other error deletion technologies must be adopted, combined with a variety of video anti-error methods to ensure the quality of the restored image .

Existing anti-packet loss error technologies can be roughly divided into two categories: (a) Active error-proof type: take protective measures in advance, such as introducing redundancy mechanisms, try to ensure that data packets are not lost or ensure that the receiving end can recover a small amount of lost data (b) error compensation type: take certain compensation measures in the event of bit errors, such as when the network condition deteriorates seriously, the packet loss rate is very high, and the active error prevention method loses its effect. Bit errors are compensated.

According to different emphases, the bit error elimination method of error compensation is divided into two types: error concealment and error diffusion elimination. Among them, error concealment focuses on compensating for the current impact of bit errors. For example, if the current video frame or slice at the receiving end is lost, the image cannot be displayed correctly, that is, certain measures are taken to compensate, so as to minimize the impact on users. The error diffusion elimination is to eliminate the subsequent impact of error diffusion in space and time. For example, the frame received by the receiving end is lost or partially lost. Since this frame may be the prediction reference frame of the subsequent frame, its error will be It will spread to subsequent frames in the time domain; or due to the possible intra-frame prediction and loop filtering in H.264, the bit error of the frame may spread to other positions of the frame through spatial prediction. Elimination of bit error diffusion is to take certain measures to limit the influence of bit errors in a limited area in space and a limited time in time to avoid video communication failure, which is even worse than the decoder system working disorder and collapse.

It can be seen that in a bit error environment, bit error diffusion not only degrades the restored image quality of the erroneous frame, but also may cause unrecoverable losses to subsequent frames. Even if the error concealment technology is used at the decoding end, the restored image quality cannot be avoided. In addition, due to the strong real-time requirements of video communication, the ARQ mode is usually not used to retransmit erroneous data.

The problem of bit error compensation can be solved by error concealment, that is, by simply replacing or complex prediction, interpolation, and other methods of covering up the error portion with the correct data of the adjacent portion of the bit error portion in space and time. This is done on the receiving end without the involvement of the sending end. The problem of bit error diffusion is more complicated, and requires the cooperation of the sending end and the receiving end to adopt appropriate strategies to suppress and eliminate it.

It should be noted that error concealment can also lead to error diffusion. In fact, due to error concealment, the contents of the reconstructed image cache at the encoding end and the decoding end do not match, resulting in the diffusion of bit errors in the time domain. For example, when there is a packet loss in decoding frame n-1, the decoding end will use the image data corresponding to the position of frame n-2 to cover up the error, while the sending end does not know that there is packet loss in frame n-1 when encoding, and will use The correct n-1 frame image is used to encode the n-th frame image, but when the n-th frame is decoded at the receiving end, the n-2 frame is used instead of the n-1 frame for decoding, which causes error diffusion.

The existing H.264/RTP transmission architecture and RTCP-based QoS reporting method use RTP to directly encapsulate NALU for transmission, and use RTCP SR/RR reports to monitor QoS information. The relevant technical details have been introduced above.

In addition, the Tornado code used in the prior art is a relatively complicated solution. Adopting Tornado code to realize the data transmission protection method based on H.261/H.263/H.263+/H.263++/H.264 video compression coding comprises the steps:

等，确定任何相邻两层的节点之间的二部图的左边度分布向量和右边度分布向量。In step 1, the structure of the Tornado code is designed. Specifically: according to the given protection capability, other requirements, specific data types such as audio or video, rate and other factors, the size of the data node is determined to be L1 bits, and the number of data nodes is determined according to factors such as the maximum acceptable network delay in specific applications. n, the total number of verification nodes L, the number of layers in the middle verification layer m, and the scaling factor for the number of nodes between layers

etc., determine the left degree distribution vector and right degree distribution vector of the bipartite graph between nodes of any adjacent two layers.

Go to step 2, according to the above-mentioned left degree distribution vector and right degree distribution vector, a random bipartite graph between each adjacent two-layer nodes is generated by means of random graph matching.

Go to step 3 and start the data transmission process. The H.261/H.263/H.263+/H.263++/H.264 video encoder at the sending end generates a data stream, which is required for data transmission protected.

Go to step 4, divide the data code stream to be protected into equal-sized data nodes D ₀ , D ₁ , D ₂ , D ₃ . . . _DT with L1 bits.

Go to step 5, let t=0.

Go to step 6, obtain n data nodes in total D _t , D _t+1 ... D _t+n-1 .

Go to step 7, according to the random bipartite graph between the adjacent two-layer nodes determined in step 2, each intermediate school is generated layer by layer for the above n data nodes D _t , D _t+1 .... D _t+n-1 The check node layers MC ⁽⁰⁾ , MC ⁽¹⁾ ... MC ^(m) and the final check node layer FC.

Go to step 8, pass data nodes D _t , D _t+1 ... D _t+n-1 and check nodes in each check node layer generated through step 7 through UDP/IP or TCP/IP, etc. In the network packaging method, it is packaged and sent to the receiving end.

Go to step 9, the sender judges whether all data nodes have been processed according to the values of t and T, if all data nodes have been processed, go to step 10; otherwise, t=t+n, go to step 6.

In step 10, the communication process of the sending end ends.

Since the sending end usually transmits while encoding and compressing in the process of real-time communication, the above steps 3 to 8 represent not a chronological sequence, but a logical sequence.

The above process describes the sending end. For the receiving end, after receiving a batch of data nodes and check nodes, the receiving end should first determine which data nodes should be received but actually lost and recoverable data The node then performs decoding and recovery on the above-mentioned lost and recoverable data according to the general decoding process of the Tornado code. The receiving end repeats the above receiving and recovery process until the communication process ends.

In addition, the existing bit error elimination methods are all independent error concealment methods or error diffusion elimination methods, and there are many methods with different implementation technical details. Error concealment methods include time domain concealment, space domain concealment, joint spatiotemporal concealment and so on. Error diffusion elimination includes intra-frame coding, identification, adaptive intra-frame block refresh, etc.

The time domain masking method is to use the information of adjacent frames on the time axis to calculate the missing data. The method of inferring can be: simply adopt the data of the same position of the adjacent frame to replace the missing data; consider the motion prediction factor, and perform motion prediction according to the data of the adjacent frame. In addition, there are more complex masking strategies, but the amount of calculation is very large.

The spatial domain masking method is to use the spatial adjacent area of the missing data area to conceal the error. The same is as follows: simple replacement with domain; based on data fusion, there are multiple spatially adjacent areas to calculate the lost data, such as spatial interpolation; algebraic inversion method, the packet loss process is modeled with a linear model, and its input is Data, the output is the data received correctly, using algebraic inversion method, such as the least squares method, to invert the input from the output, and replace the wrong data with the inversion result, this method is computationally intensive.

The spatio-temporal joint concealment method is to jointly use the error concealment of space domain and time domain. For example, according to the characteristics of the missing data and the situation of adjacent time data and spatial data, a certain strategy is adopted to determine whether it is better to use the space domain or the time domain to cover up, and then implement this better cover strategy. Alternatively, fuse spatial and temporal data to jointly perform masking.

The error diffusion elimination method based on intra-frame coding is to use intra-frame coding for the macroblocks affected by the errors, that is, to use the forward dependence of the motion vector for accurate error tracking, and to use frame Inner coding can effectively prevent bit errors from spreading. Firstly, the inter-frame dependence caused by motion compensation is given; then the "energy" of the bit error is calculated according to the correlation between the forward dependence of the motion vector and the weight factor, and intra-frame coding is used for the macroblock with the largest "energy", so that Prevent error propagation.

The identification-based error diffusion elimination method is to identify the macroblock affected by the error, so as to avoid using the identified macroblock as a reference frame during encoding, which directly prevents the error from spreading. This method needs to establish a feedback mechanism from the sending end to the receiving end. The receiving end will feed back the information of the lost data to the sending end, and the encoding end will use a specific value to mark all the pixels after the error macroblock in the same block group according to the error information. The marked area is not referred to in the coding of several subsequent frames, which avoids error diffusion at the receiving end.

The error diffusion elimination method based on the adaptive intra-frame block refresh strategy is based on the "error sensitivity scale" of the encoder to measure the vulnerability of each coded macroblock to channel error, and then performs adaptive intra-frame block refresh . This method does not require a feedback channel. The encoding end first initializes the value of the "bit error sensitivity scale": the farther the macroblock is from the synchronization mark, the higher the sensitivity to bit errors; the more bits of the encoded macroblock, the more vulnerable it is to being damaged by bit errors. During encoding, this scale is updated by calculating the accumulation of the "error sensitivity scale" value for each macroblock, and then macroblocks are selected for intra coding according to the ESM scale.

In addition, because there is no convenient solution for network status monitoring and description of the relative importance of data in the prior art, multi-level protection and unequal protection have not been realized.

In practical application, the above-mentioned scheme has the following problems: the Tornado code scheme in the prior art is too complex and inefficient, and when applied to the protection of video data, the delay is large and cannot meet the performance requirements of real-time communication.

At the same time, there is a lack of a mechanism that can report network conditions. Therefore, the communication parties cannot decide to adopt an appropriate protection mechanism according to the network conditions, so that unequal protection and adaptive hierarchical protection cannot be used effectively, and the reliability of multimedia communication cannot meet the requirements.

Moreover, the two methods of error concealment and bit error diffusion elimination are not well unified, and sometimes contradict each other, and their functions cancel each other out.

In the prior art, the joint working mechanism of H.264 NAL and RTP protocol is lacking, and how to provide a protection mechanism based on H.264 NAL and the corresponding RTP encapsulation method is not defined, which is a blank. Furthermore, good methods, such as high-efficiency Tornado coding and other protection measures, as well as unequal protection and adaptive hierarchical protection, cannot be applied to H.264 video data yet.

In the prior art, the message extension mechanism of H.264 cannot be used to report the network status and QoS information. Without this mechanism, many good technologies do not have the necessary preconditions for application.

The existing technologies are relatively fragmented, lack of integration, and the effects of each other do not enhance each other. At the same time, many technologies are still in the academic discussion stage, and have not entered the definition and development of communication protocols, which affects practical applications. In the integration of these technologies, the constraints of real-time communication performance requirements must be considered, and the selected technology must have good performance and the calculation should not be too complicated.

The main reason for this situation is that the existing technology uses a fixed erasure code strategy to protect the video communication stream, which cannot adapt to network communication changes; the alternative mechanism adopted by the error concealment method will cause error diffusion; the error diffusion elimination method requires Complicated mechanisms or additional feedback channels consume system processing resources and network bandwidth resources.

In the existing technical solution, the header information of the NALU is completely encapsulated in the payload, so that the RTP protocol cannot directly obtain the attributes, levels, and importance of the payload, so that the QoS mechanism based on this cannot be realized. Secondly, this encapsulation format also causes NALU header information to occupy payload resources, because each NALU has header information attached, resulting in many cases, because the header information of multiple NALUs of the same type in one RTP are the same , thus wasting RTP transmission bandwidth resources.

The multimedia communication framework of H.264/RTP adopts a common cooperative control protocol RTCP to transmit QoS reports to realize QoS monitoring. However, RTCP itself is not necessarily the most suitable for specific video communication applications such as H.264, because Its own out-of-band re-opens the logical channel to transmit the QoS report, which affects the network status and leads to conflicts.

The key point is that the existing technology does not implement the error resilience protection strategy of the transport layer, and cannot provide the reliability and communication quality of multimedia transmission.

Contents of the invention

In view of this, the main purpose of the present invention is to provide a multimedia communication method and its terminal, which can realize unequal protection and facilitate the realization of QoS guarantee.

To achieve the above object, the present invention provides a multimedia communication method, the communication process includes the following steps,

A The sending end protects the multimedia data according to the error resilience protection strategy based on the real-time transport protocol, and sends it to the receiving end. middle;

B. The receiving end receives the multimedia data, and in the case of a transmission error, restores or partially restores the multimedia data according to the error resilience protection policy carried in the error resilient real-time transport protocol packet encapsulating the multimedia data.

Wherein, its communication process also includes the following steps,

C The receiving end counts the communication quality, generates a service quality report, and sends it back to the sending end;

In the step A, the sending end adjusts the error resilience protection policy according to the service quality report.

In addition, in the method, the communication process also includes the following steps,

D The receiving end counts and transmits error information, and implements an error concealment strategy;

In the step C, the receiving end also feeds back the transmission error information to the sending end;

E. The sending end implements an error diffusion elimination strategy according to the transmission error information.

In addition in described method, described step A comprises following sub-step:

A1 The sender selects the error elastic coding scheme to perform forward error correction coding on the multimedia data;

A2 The sending end encapsulates the coded multimedia data with the error RRT protocol, and carries the information related to the error elastic coding scheme in the error RRT protocol packet header information, and then sends it to the receiving end;

Described step B comprises following substep:

B1 The receiving end decapsulates the received error RRTTP packet, and extracts the forward error correction coding scheme related information from the error RRTTP packet header information;

B2 If the error elastic real-time transport protocol packet corresponding to the data node is lost during the transmission process, then the receiving end selects the forward error correction decoding scheme to perform forward error correction according to the information about the error elastic coding scheme Decoding, recovering or partially recovering the lost multimedia data; if no loss of data nodes occurs, forward error correction decoding is not required.

In addition, in the method, the forward error correction coded multimedia data is divided into two types: data nodes and check nodes.

In addition, in the method, in the step A1, the sending end selects the forward error correction coding scheme at least according to one of the following factors:

The current network transmission status, and the quality of service level of the multimedia data to be sent, wherein the quality of service level of the multimedia data to be sent depends on the relative importance of different data.

In addition, in the described method, the error RRTTP packet header information includes:

Error RRTTP identification field, used to indicate to distinguish it from RTP;

The forward error correction coding type field is used to indicate the forward error correction code type adopted by the forward error correction coding scheme;

The forward error correction coding subtype field is used to indicate the relevant parameter settings of the forward error correction coding scheme;

The data packet length field is used to indicate the length of the node obtained by the forward error correction coding scheme after performing forward error correction coding on the multimedia data;

The number of data packets field is used to indicate the number of nodes carried by the error RRT protocol packet.

In addition, in the method, in the step A1, the sending end divides at least one H.264 network abstraction layer unit into at least one data node of equal length, and then performs forward error correction coding on it, Obtain at least one check node;

In the step A2, the sending end encapsulates the data node and the check node into at least one error RRT protocol packet for sending;

In the step B1, after receiving the error RRTTP packet, the receiving end decapsulates to obtain the data node and the check node;

In the step B2, if the data node in the transmission process is lost, the receiving end restores or partially restores the lost data node based on forward error correction decoding according to the check node, and The H.264 network abstraction layer unit is obtained by dividing.

In addition, in the method, before starting the transmission, it also includes the step of,

The sending end and the receiving end negotiate and determine: for each type of the forward error correction code, the value of the forward error correction code subtype field and the relevant parameters of the forward error correction code indicated Correspondence of settings.

In addition, in the method, both the sending end and the receiving end establish a correspondence table according to the correspondence indicated by the forward error correction coding subtype field, and are used to establish a correspondence table according to the forward error correction coding type field and The forward error correction coding subtype field queries the corresponding forward error correction coding or forward error correction decoding processing module;

In the step A1, the sending end invokes a corresponding forward error correction encoding processing module to perform forward error correction encoding;

In the step B2, the receiving end invokes a corresponding FEC decoding processing module to perform FEC decoding.

In addition, in the method, in the step A1, the sending end, according to any one of the network abstraction layer reference identification field and the network abstraction layer unit type field in the header information of the H.264 network abstraction layer unit or a combination of both, and any other rules that can be defined in advance to evaluate the relative importance of corresponding data, thereby determining the service quality level, and then selecting the forward error correction coding scheme, determining the forward error correction Encoding type field and forward error correction encoding subtype field.

In addition, in the method, in the step A, the sending end evaluates the network transmission status according to the transmission report fed back by the receiving end, and then selects the forward error correction coding scheme to determine the forward error correction coding scheme. Error correction coding type field and forward error correction coding subtype field.

In addition, in the method, the value of the version information field in the incorrect RRTTP packet header information is binary value "11" or decimal value "3", so as to distinguish it from the real-time transport protocol;

The forward error correction coding type field is located after the contribution source identifier list and occupies 4 bits;

The forward error correction coding subtype field is located after the forward error correction coding type field and occupies 9 bits;

The data packet length field is located after the forward error correction coding subtype field and occupies 11 bits;

The data packet number field is located after the data packet length field and occupies 8 bits.

In addition, in the method, in the step A, the sending end removes the header information of at least one network abstraction layer unit with the same header information, and then divides, encodes, and encapsulates them into the error RRTTP packet , and integrating the same header information of the network abstraction layer unit into the header information of the wrong RRTTP packet;

In the step B, the receiving end obtains the carried header information from the header information of the received error RRT protocol packet, and adds it to the stripped information extracted from the error RRT protocol packet. The head of the network abstraction layer unit of the header information, to obtain the complete network abstraction layer unit; if there is a transmission error, perform forward error correction decoding recovery or partially restore the data node according to the preset strategy, and then extract the network abstraction layer from it layer unit.

In addition, in the method, in the incorrect RRTTP header information, the network abstraction layer reference identification field and the type field in the network abstraction layer unit header information are filled in the incorrect RRTTP header information In the payload type field, the payload type field is located in the last 7 bits of the second byte of the error RRT Protocol packet header information.

In addition, in the method, the error RRT protocol identification field is the version information field of the error RRT protocol header information, and the version information field is located in the first word of the error RRT protocol header information. The first 2 bits of the section.

In addition, in the method, in the error RRT-P encapsulation format, the forbidden bit field in the network abstraction layer unit header information is filled in the flag field of the error RRT-P packet header information, the flag The field is located at the first bit of the second byte of the error RRTTP packet header information;

And in the step B, the receiving end judges whether the network abstraction layer unit carried by the wrong RRT-Packet has an error according to the flag field of the wrong RRT-Packet.

In addition, in the method, in the steps A and B, the error RRTTP identifier is the value of the flag field of the error RRTTP packet header information, and the flag field is located in the error RRTTP The first bit of the second byte of the protocol header information.

In addition in described method, described step A comprises following sub-step:

The sending end first judges whether the prohibited bit field in the header information of at least one network abstraction layer unit is valid, and accordingly divides it into a normal network abstraction layer unit and an error network abstraction layer unit;

Then encapsulate the normal network abstraction layer unit into the error RRT protocol package according to the error RRT protocol encapsulation format, and set the error RRT protocol identifier;

encapsulating the erroneous network abstraction layer unit into the real-time transport protocol packet according to the real-time transport protocol encapsulation format;

Described step B comprises following substep:

The receiving end first judges whether the header information of the received packet is set with the wrong RRTTP identifier, and divides it into the wrong RRTTP packet and the RRTTP packet;

Then process the erroneous RRT protocol packet according to the erroneous RRT protocol encapsulation format, and process the RRT protocol packet according to the RRT protocol packet encapsulation format.

In addition, in the method, the following sub-steps of the step C,

C1 The receiving end statistics generate the service quality report;

C2 The receiving end uses the H.264 extended message to carry the service quality report and sends it to the sending end.

In addition, in the method, the H.264 extended message is supplementary enhanced information;

The encapsulation format of the quality of service report in the supplementary enhanced information is as follows:

The first byte is the payload type field, which is used to indicate that the payload is a corresponding quality of service report;

The 2nd and 3rd bytes are payload length fields, which are used to indicate the length of the corresponding quality of service report;

The 4th byte and the following are payloads, which are used to fill in the corresponding QoS report.

In addition, in the method, the quality of service report is divided into a sender report and a receiver report, which are distinguished by the payload type field indication;

When the quality of service report is filled in the payload of the supplementary enhancement information, the payload of the supplementary enhancement information includes:

The version information field occupies 2 bits;

Padding field, occupying 1 bit, used to indicate whether there is padding content;

The received report number field occupies 5 bits and is used to indicate the number of received report blocks reported in the quality of service report;

The sender synchronization source identifier field, which occupies 32 bits, is used to identify the sender of the QoS report;

When the QoS report is a sender report, it also includes a sender information block, which is used to describe information about the sender of the QoS report;

including at least one of said reception report blocks for describing multimedia statistical information from different sources;

Contains layer-specific extensions for layer-specific reserved function extensions.

In addition, in the method, the supplementary enhanced information for carrying the quality of service report is further carried by an abstract network layer unit;

The communication terminal sets the network abstraction layer reference identifier of the abstract network layer unit according to the reliability requirement transmitted by the quality of service report.

In addition, in the method, the communication terminal dynamically adjusts the cycle of statistical generation and sending of the quality of service report according to the current network status and high-level application requirements.

In addition, in the method, when the communication terminal uses the supplementary enhanced message to mix and carry at least one quality of service report of the media stream,

The quality of service report includes the reception report block corresponding to at least one media stream carried.

In addition, in the method, in the step C, the receiving end counts the number of lost network abstraction layer units according to the received network abstraction layer unit serial number of the video stream data, and generates the service quality report , sent back to the sending end;

In the step A, the sending end calculates the cumulative packet loss rate according to the lost network abstraction layer unit serial number, and adjusts the error resilience protection strategy accordingly.

In addition, in the method, the receiving end analyzes and calculates network status parameters according to the received service quality report; the parameters include end-to-end instantaneous bandwidth, delay and jitter.

In addition, in the method, the sending end sets a series of error resilience protection strategies of different levels, and in the step A, selects and uses the error resilience protection strategy of the corresponding level according to the service quality report.

In addition, in the method, in the step C, the receiving end calculates the location information of the lost video stream data according to the received network abstraction layer unit serial number of the video stream data, and sends it back to the the sender;

In the step A, the sending end resends the lost video stream data to the receiving end according to the location information of the lost video stream data.

In addition, in the method, in the step E, the sending end obtains the location information of the lost slice according to the transmission error information, and performs segment-by-segment intraframe coding on the lost slice to realize The error diffusion elimination strategy.

In addition, in the method, the segmented successive intra-frame coding includes the following steps,

E1 Divide a group of continuous macroblocks from the lost slice to form a new slice, and the remaining macroblocks still belong to the lost slice, and enter step E2;

E2 performs intra-frame coding on the new strip and sends it in the next frame, after which the new strip is routinely coded and enters step E3;

E3 When encoding the next frame, judge whether the lost slice still contains unprocessed macroblocks, if yes, return to step E1, otherwise end intra-frame encoding.

In addition, in the method, in the step E1, the size of the group of continuous macroblocks separated each time satisfies: after the group of continuous macroblocks is intra-frame encoded, the data rate of this frame is within the H.264 data rate. within the rate control range.

In addition, in the method, the step D includes the following sub-steps,

D1 The receiving end detects transmission errors and counts transmission error information;

D2 The receiving end resynchronizes the video information after a transmission error occurs;

D3 The receiving end implements the error concealment strategy according to the transmission error information.

In addition, in the method, in the step D1, the receiving end detects and counts the transmission error information according to the discontinuity of the sequence number of the network abstraction layer unit.

In addition, in the method, in the step D1, the receiving end obtains the location information of the lost slice according to the interruption of the serial number of the network abstraction layer unit, and the location information includes the frame number where the lost slice is located and the position of the missing slice in that frame.

In addition, in the method, the error concealment strategy includes a step: the receiving end replaces the missing segment with a corresponding segment of the frame preceding the frame where the missing segment is located.

Also in the method, the error resilient coding scheme comprises a modified "Tornado" erasure code;

The improved "Tornado" erasure code generates only one layer of check nodes for a set of data nodes.

In addition, in the method, the transmission error in the step B includes data packet loss, or random bit error.

The present invention also provides a multimedia communication terminal, comprising a basic functional module for realizing multimedia communication, including at least a codec module for realizing a multimedia codec function, and further comprising the following modules:

An error-resilient real-time transmission control protocol module, used for performing error-resilience protection on the multimedia data encoded by the codec module before transmitting on the network side, and performing error correction on the multimedia data from the network side before transmitting Decoding the codec module, the error resilience protection related information is carried in the error resilience real-time transport protocol packet encapsulating the multimedia data.

Among them, the following modules are also included:

The protection method and policy negotiation module is used to be responsible for negotiating error resilience protection strategies between the communication parties, and determining a set of protection strategies for selection by the error resilience real-time transmission control protocol module;

A forward error correction module, configured to implement at least one forward error correction protection method, and maintain relevant parameters of the forward error correction protection method, wherein the protection method and policy negotiation module controls the forward error correction module to The functions of unequal protection and self-adaptive hierarchical protection are realized, and the error resilient real-time transmission control protocol module realizes the functions of error resilient protection and error correction by calling the forward error correction module.

Additionally, it contains

An error masking module is used to realize the error masking function;

The encoding and decoding module is used to realize the H.264 encoding and decoding standard, and is also used for the error diffusion elimination function;

It also includes a network status analysis and calculation module, which is used to analyze and calculate network status, and provide information to the error concealment module and the codec module.

Additionally, it contains

The supplementary enhanced message extension processing module is used to implement the functions of service quality report and network status report, and send the reports to the network status analysis and calculation module.

In addition, its transmission layer is based on the error-resilient real-time transport protocol/real-time transport control protocol, and is used to realize the multimedia transmission function supporting error resilience;

Its application protocol layer includes a protection mechanism and a policy negotiation sublayer, which are used to realize hierarchical protection and unequal protection functions;

Its H.264 video coding layer includes a supplementary enhanced message extended reporting layer, which is used to realize the reporting function based on the supplementary enhanced message extension;

Its H.264 network abstraction layer includes a forward error correction coding layer, which is used to realize the forward error correction coding function.

In addition, the basic functional modules for realizing multimedia communication include one of the following or any combination thereof:

The main control module is used to control the entire terminal;

The user interface module is responsible for the interaction of user input and output and the display of information;

The network communication module is responsible for communicating with the network and providing the lower layer transmission channel;

The input and output and underlying driver modules are responsible for driving hardware devices;

Business module, used to realize high-level business;

The communication process control module is used to control the communication process;

The application protocol module is used to implement the application protocol function;

The real-time transmission control protocol module is used to realize the real-time transmission control protocol function;

H.264 network abstraction layer module, used to realize the network abstraction layer function;

The audio codec module is used to realize the audio codec function.

Through comparison, it can be found that the main difference between the technical solution of the present invention and the prior art is that the error resilient real-time transport protocol (ERRTP) is adopted, and a transport layer encapsulation that can carry information related to the error resilient coding scheme is provided on the basis of the existing RTP Format, so that multimedia data is marked with its corresponding error-resilient coding scheme information when it is transmitted on ERRTP, so that the error-resilience mechanism is integrated into the transport layer;

Aiming at the H.264NALU structure, a special ERRTP encapsulation method and a transformation scheme for protocol header information are given. By combining the header information bytes of all NALUs in the same ERRTP packet into its header information, a clever combination method is adopted. It does not affect the operation of the existing ERRTP protocol and equipment, and can directly reflect the attributes of the NALU payload in the ERRTP header information. On the one hand, it greatly improves the carrying efficiency, and on the other hand, it provides the basis for the realization of the QoS mechanism;

Based on the H.264 message extension mechanism, the receiving end counts the communication quality and feeds it back to the sending end, and directly adopts the extended message mechanism of the high-level media protocol H.264 itself to carry the QoS report information, avoiding the use of additional channels, and realizes a " In-band "QoS reporting mechanism;

At the sending end, various alternative error elastic coding schemes can be selected according to factors such as the current network status and the importance level of multimedia data, so as to achieve the purpose of unequal protection and achieve a balance between protection capability and transmission efficiency;

On the basis of the feedback mechanism from the receiving end to the sending end, the alternate mixed use of unequal protection and multiple error resilience schemes is realized. The sending end chooses to use different levels of protection according to the QoS report fed back by the receiving end and related network transmission status messages In addition, based on the data importance level reflected from the ERRTP header information, you can also choose to use an appropriate protection strategy for data of different levels;

For the H.264 NALU data stream, a combined scheme of error concealment and error diffusion elimination is given, which comprehensively reflects the advantages of the two technologies, and the error diffusion elimination is realized through the error information feedback mechanism and segmented sequential intra-frame coding;

An efficient Tornado code scheme is also provided. In the case of ensuring that the protection capability of data transmission is not significantly reduced, by setting an erasure code with only one check node layer, the generation of erasure codes is reduced. The amount of calculation reduces the delay time of data transmission and improves the performance and cost ratio of data transmission protection;

Finally, the above-mentioned various enhancement technologies related to multimedia communication are integrated into the multimedia communication system, and various technologies and protocol frameworks are implemented in a modular manner. Various technologies work in harmony with each other to further enhance the reliability of multimedia communication.

The difference in this technical solution has brought more obvious beneficial effects, that is, the ERRTP protocol transmission architecture implements an error resilience mechanism at the transport layer, which greatly simplifies the error resilience transmission structure and saves network transmission bandwidth; the realization of unequal protection achieves The balance of protection capability and transmission efficiency facilitates the realization of QoS guarantee for multimedia transmission, further improves service quality, reduces redundancy, improves transmission efficiency, and achieves compatibility with existing technologies, all of which improve the robustness of this new method of ERRTP sex;

Based on the QoS report of H.264 message extension mechanism, QoS monitoring can be realized in-band, bandwidth overhead can be reduced, and the complexity of system implementation can be reduced. H.264 video network transmission quality;

The unequal protection and multi-level protection strategies are more flexible, accurate and timely to adapt to network transmission requirements, improve protection capabilities, improve system efficiency and reliability, ensure accurate statistical information and save system resources;

Combining error concealment and error diffusion elimination, avoiding error diffusion caused by error concealment, achieving ideal error elimination effect under the premise of simple complexity, improving video transmission quality, saving overhead, simplifying mechanism, and ensuring system compatibility ;

Use the improved Tornado erasure code scheme to improve the cost performance of data transmission protection, improve data transmission efficiency, and promote the application of new technologies such as H.264;

Integrating multiple enhancement technologies into the multimedia communication system to jointly improve the quality of multimedia communication can greatly improve the performance and user experience of H.264-based multimedia communication products such as conference TV and videophone on IP networks, and improve product competition. Power, bring significant economic benefits, has unlimited practical value.

Description of drawings

Fig. 1 is a schematic diagram of the header information structure of the RTP packet;

Fig. 2 is a schematic diagram of the encapsulation format of RTP packet payload to NALU data;

Fig. 3 is a schematic diagram of the QoS report packet format based on the RTCP protocol;

Figure 4 is a schematic diagram of the principle of Tornado erasure code;

5 is a schematic structural diagram of a multimedia communication terminal module supporting error resilience according to a first embodiment of the present invention;

6 is a schematic structural diagram of a multimedia communication protocol stack according to a first embodiment of the present invention;

Fig. 7 is a schematic diagram of the header information structure of the ERRTP packet according to the second and third embodiments of the present invention;

FIG. 8 is a schematic diagram of an SEI encapsulation format of a bearer QoS report according to a fourth embodiment of the present invention;

Fig. 9 is a schematic diagram of the principle of error diffusion elimination based on segmented successive intra-frame coding according to the sixth embodiment of the present invention.

Fig. 10 is a schematic diagram of the structure of the erasure correction code of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings.

The invention integrates various enhanced technologies into one multimedia communication system, and combines the respective advantages of various enhanced technologies to jointly improve system performance, transmission reliability and communication quality. These enhanced technologies include the Error Resilience Real-time Transport Protocol (ERRTP) that integrates FEC into the RTP protocol, the technology that integrates NALU header information into the RTP header, uses SEI extended messages to carry QoS reports, and Feedback technology of network status, multi-level protection and unequal protection mechanism based on feedback, combined technology of error concealment and diffusion elimination, and improved Tornado coding scheme.

The present invention modularizes various enhanced technologies and combines them in a multimedia communication system to realize error-resilient H.264 video communication. The system includes a general main control module, user interface, network communication module, I/O and underlying drivers Modules, various business modules, communication process control modules, application protocol modules, etc., also include protection methods and policy negotiation modules for implementing various enhanced technologies, FEC modules, ERRTP modules, RTCP modules, H.264NAL modules, H.264 encoding Converter module, H.264 decoder module, audio codec module, error concealment module, SEI message extension processing module, network status analysis and calculation module.

In the first embodiment of the present invention, a variety of enhanced technologies are implemented and modularized into a multimedia communication system, mainly referring to a multimedia communication terminal. The following first describes the implementation of the device from the functional modules of the terminal , a complete internal module structure diagram of the terminal is shown in FIG. 5 . It should be noted that the functional modules mentioned here are all defined in terms of functions, and specific implementation methods may be software, hardware, firmware (firmware) or a combination of software and hardware. A complete multimedia communication terminal must first include the following modules:

Main control module: responsible for the control of the entire terminal system;

User interface (or interface) module: responsible for user input and output interaction, users operate through interface control elements such as menu buttons, and display feedback information at the same time, such as current system status, parameters, network status, etc.;

Network communication module: responsible for communicating with the network, providing TCP, UDP, IP and lower communication protocol stacks such as Ethernet, PPP, ATM, etc.;

I/O and underlying driver module: responsible for driving hardware devices, such as video, audio acquisition devices and display/playback device drivers, and responsible for input and output of video and audio data;

Various business modules: realize various specific services, such as videophone, multi-party conference, video mail, instant message, video chat, etc.;

Communication process control module: control the specific communication process, such as applying for the chairman, releasing the chairman, applying for a speech, controlling the broadcast of a certain venue, browsing the venue, etc. in a multi-party conference;

Application protocol module: it can be specific application protocols such as H.323 system (including H.225.0, RAS, H.245, H.235, H.460, etc.) and SIP. Generally speaking, this agreement is a general term for a series of agreements, called "protocol umbrella" (protocol umbrella);

In addition, corresponding to various enhancement techniques, they are implemented in the following modules:

Protection method and strategy negotiation module: This module is responsible for carrying out protection method negotiation between communication parties, determining the allowed set, and then negotiating a set of strategies for mixed and alternate use of protection methods according to the allowed set. Negotiation is done through "application protocol module" to communicate. This module controls the FEC module, which implements different FEC protection methods, unequal protection and adaptive hierarchical protection functions;

FEC module: This module supports a variety of FEC protection methods, and they can belong to multiple categories as subclasses, assuming that T different methods are supported in total. According to the negotiation result (from the "protection method and policy negotiation module"), the H.264 video data and audio data (not within the scope of this patent) are protected. This module internally saves the generation rules and parameters corresponding to various FEC subclasses, so it contains an internal database for storing these data. This module can realize the mixed and alternate application of different protection methods;

ERRTP module: realize the ERRTP protocol, the protocol encapsulation format of ERRTP and the relevant encapsulation and decapsulation steps corresponding to H.264 will also be described in detail in the following embodiments;

RTCP module: realize normal RTCP function, although the present invention provides the report mechanism based on H.264 SEI message extension, can realize the report of main RTCP information, but do not get rid of the use of RTCP, two kinds of report mechanisms can coexist, do like this The main consideration is compatibility and interoperability, so the other terminal may not support the SEI message extended reporting mechanism;

H.264 NAL module: realize the function of H.264 network abstraction layer;

H.264 encoder module: in addition to realizing the normal H.264 encoder function, the error diffusion and elimination function of the present invention has also been realized, so the information according to the data comes from the "network status analysis and calculation module";

H.264 decoder module: realize normal H.264 decoder function;

Audio codec module: realize audio codec function, supported protocols can be ITU-TG.711, G.722, G.723.1, G.728, G.728, G.722.2 (3GPP AMR), MPEG organization MP3, AAC, etc.;

Error masking module: realize the error masking function provided by the present invention. The information based on it comes from the "network status analysis and calculation module" and "H.264 encoder module";

SEI message extension processing module: realize QoS and network status report function based on SEI message extension of the present invention, at sending end, collect data and form RTCP SR, RR report, send out by SEI extension message encapsulation then; From SEI extension at receiving end Extract RTCP SR and RR reports from the message, and then send these data to the "network status analysis and calculation module" for analysis and calculation;

Network status analysis and calculation module: according to the data from the "SEI message extension processing module", analyze and calculate to obtain network status data, such as packet loss rate, jitter, delay, end-to-end bandwidth (throughput) in a clockwise manner, etc., and then, These data are used to control the "H.264 encoder module" and "error concealment module", and these data are also sent to the "user interface module" to be displayed for the user.

After understanding the module composition of the communication terminal as a whole, we will describe this terminal from the level of the protocol stack. A communication terminal system implements multiple protocols at different levels, and these protocols constitute a protocol stack (Protocol Stack). For the terminal of the present invention, its protocol stack has the same place as that of the common multimedia communication terminal, and also has some differences, and some new layers are added in some places. Fig. 6 shows a schematic structural diagram of a multimedia communication protocol stack according to the first embodiment of the present invention.

The difference between the multimedia transmission architecture of H.264/ERRTP of the present invention and the traditional H.264/RTP architecture mainly lies in:

The RTP/RTCP layer in the general terminal protocol stack is replaced by the ERRTP/RTCP layer. After using ERRTP that supports error resilience, the error resilience protection mechanism is combined with the transport layer for implementation;

A "protection mechanism and policy negotiation layer" is added to the application protocol layer. This layer is mainly used for the communication parties to negotiate various protection levels and related protection schemes when realizing multi-level protection and unequal protection;

The "SEI extended report layer" is added between the H.264 VCL layer and the NAL layer. This layer is convenient for both senders and senders to realize QoS monitoring and network transmission status feedback based on SEI extended messages;

The "FEC layer" is added between the H.264 NAL layer and the ERRTP/RTCP layer. This layer realizes the node division, encoding and encapsulation of the H.264 NALU data stream.

Those skilled in the art can understand that in the first embodiment of the present invention, the typical H.264 service is taken as an example to give the basic modular structure and protocol stack composition. For other protocols or future multimedia communication protocols or applications , it is only necessary to implement relevant technical details according to specific applications on the basis of the principles of the present invention to achieve the purpose of the invention without affecting the essence and scope of the present invention.

On the premise of giving the overall system architecture, the implementation details of each enhancement technology will be described in turn below.

Aiming at many problems existing in the prior art, the present invention proposes an improved RTP protocol supporting error resilience, which aims to integrate the error resilience mechanism into the transport layer protocol, which not only simplifies the transmission structure and reduces complexity, but also improves the error resilience mechanism Flexibility enhances delivery reliability. Due to the error resilience, the present invention claims this improved RTP protocol as Error Resilience Real-time Transport Protocol (Error Resilience Real-time Transport Protocol, referred to as "ERRTP" or "ER2TP"). The main difference between ERRTP and RTP is that the header information extension of the ERRTP protocol can carry information related to the error elastic coding scheme, such as FEC type, protection capability, and coding parameters.

On the basis of ERRTP, the present invention conveniently realizes unequal protection. Firstly, a variety of protection measures with different protection capabilities are provided for selection. These factors are used to select appropriate protection measures, so as to achieve the purpose of unequal protection and achieve a balance between protection capability and transmission efficiency. Since each ERRTP data packet carries the FEC-related information it adopts, the sending end only needs to fill in the information of the selected scheme into the ERRTP header information, and the receiving end can perform correct recovery or error correction based on it .

Finally, for the NALU data transmission application of H.264, a specific implementation method based on erasure code protection is given, including the steps of dividing, generating, encapsulating and decapsulating data nodes and check nodes. Divide a continuous string of NALUs into several data nodes of equal length, and then use Tornado codes to generate check nodes. All these nodes are distributed in several ERRTP packets for transmission, and the receiving end performs this reverse process.

In the second embodiment of the present invention, on the basis of the first embodiment, the sending and receiving parties realize unequal protection based on ERRTP, and the main steps are as follows:

The sending end selects the error elastic coding scheme to perform erasure coding on the multimedia data, encapsulates the encoded multimedia data with ERRTP, and carries the information about the error elastic coding scheme in the ERRTP header information, and then sends it to the receiving end;

The receiving end decapsulates the received ERRTP packet, extracts information about the error elastic coding scheme from the ERRTP header information, and then selects the error elastic coding scheme for error elastic decoding according to the information about the error elastic coding scheme to obtain multimedia data.

Among them, the unequal protection is embodied in that the sending end selects the error elastic coding scheme according to the current network transmission status and/or the service quality level of the multimedia data to be sent.

Firstly, the specific structure of ERRTP is introduced, and a specific embodiment of the header information structure of ERRTP is given below. Fig. 7 is a schematic diagram of the structure of ERRTP header information according to the first embodiment of the present invention. It can be seen from the figure that the version information field V takes a value of 3, indicating the ERRTP protocol, to distinguish it from the traditional RTP protocol (V=2). Wherein, the extension of the header information is at the end with relevant information fields about the error elastic coding scheme, including in this example: the error elastic coding type field, the error elastic coding parameter field, the data packet length field, and the data packet number field.

The error elastic coding type field is used to indicate the type of erasure code used by the error elastic coding scheme. It can also be called the FEC Type field, which indicates the FEC coding type, which occupies 4 bits and can represent 16 different FEC types. From practical application , is sufficient. The type defined here is actually a large type, which will be further subdivided into various schemes, called subtypes. The large types in practical applications are: 0010 means Tornado code, 0011 means RS code, etc. This field can identify 16 different large types of FEC codes. The communication parties need to agree in advance on a look-up table (Look-Up Table, referred to as "LUT") between the FEC encoding type and the encoding type code called FECTypeLUT.

The error elastic encoding subtype field is used to indicate the relevant parameter settings of the error elastic encoding scheme. For each type of FEC encoding, it is necessary to determine the settings of various parameters before it can be implemented. This field is to clarify the specific parameters. Due to the limited resources in the ERRTP header information, it is impossible to list the specific parameters and rules corresponding to various FEC encoding schemes. The first embodiment of the present invention indicates various alternative parameters by using the concept of subtype Set up the scheme. This field is also called the FEC encoding subtype field, FEC Subtype, and occupies 9 bits. This field mainly indicates the subtypes further subdivided under the major types defined in FECTypeLUT.

The data packet length field is used to indicate the length of the data node after erasure coding of the multimedia data by the error elastic coding scheme. It is called the Data Length field and occupies 11 bits. Since the length of each data packet should be less than the Maximum Transport Unit (MTU) transmitted by the network, and the current wired channel MTU<1500＝0 x 5DC bytes, the wireless channel MTU<100 bytes, so this field is 11 One bit is enough to store the length of the data packet.

The data packet number field is used to indicate the number of data nodes carried by the ERRTP packet. It is also called the Packet Number field and occupies 8 bits. In ERRTP, the number of data nodes carried in each ERRTP.

It can be seen that with these fields, the decoder or network node can check the received data packets according to the FEC code type and the data packet verification type given in this field, and recover the lost data packets.

It is worth noting that the 9 bits of the subtype FEC Subtype field mentioned above are used to encode and indicate various alternative parameter setting schemes. The following describes how to perform encoding instructions in the first embodiment of the present invention technical details.

First, the sending and receiving parties need to negotiate to determine the corresponding table indicating the relationship in this field. Before starting to transmit, the sending end and the receiving end negotiate and determine: for various FEC code types, the corresponding relationship between the value of FEC Subtype and the relevant parameter setting scheme of this kind of FEC code indicated, and various alternative schemes specific parameter settings.

Then, both the sending end and the receiving end establish a correspondence table according to the negotiation result, which is used to query the corresponding FEC encoding type or FEC encoding and decoding processing module according to the FECType and FEC Subtype fields;

In the process of sending and receiving, the sending end invokes the corresponding erasure coding processing module to perform erasure coding, and the receiving end calls the corresponding erasure decoding processing module to perform erasure decoding.

In practice, subtype information actually indicates two aspects:

A. Generation Rule of FEC encoding;

B. Protection Strength/Protection Ability.

The so-called generation rules are the rules or algorithms (Algorithm) of how to process the data nodes at the sending end to generate each check node. Of course, what is done at the receiving end is just the opposite. If packet loss occurs during transmission, that is, some nodes are lost, then the lost nodes can be restored or partially restored according to the generation rules. It can be seen that the generation rule is very important information, and according to it, the two parties in the communication can work based on the FEC mechanism. Each type of FEC type listed in FECTypeLUT has different generation rules; and in each type, such as Tornado code, the generation rules of the following subtypes also need to be combined with specific generation parameters (generation parameters) . So specific to each subclass here, the generation rules will be combined with the generation parameters.

For example, for Tornado codes, the generation parameters include the following data: the total number of data nodes, the total number of check nodes, the number of layers of check nodes, the scaling ratio of the number of nodes between two consecutive layers, and the association matrix representing the node association relationship between two consecutive layers , if there are L layers of check nodes, then there are L such association matrices, or an equivalent parametric mathematical representation of a bipartite graph (Bipartite) representing the association relationship between successive two layers of nodes.

Generally speaking, under the premise of the same large generation rules, generation parameters often determine the protection strength of subtypes. For example, in the Tornado code, among the generation parameters given above, the total number of data nodes and the total number of check nodes can basically determine the protection capability to a large extent (of course, strictly speaking, to completely determine the protection capability, all generation parameters are required ). In the present invention, for each large type of FEC, select some main parameters that determine the protection ability (the most decisive effect) as representative generation parameters (representative generation parameters). By using representative generation parameters, the subcategories under the general category can be arranged in order of protection ability from weak to strong (in ascending order). Thus create a LUT called FECSubTypeLUT.

Each large type specifically supports multiple subtypes, which can be determined by the specific application and communication capabilities of both parties (such as CPU processing speed, memory, program complexity, etc.) and needs. If the communication environment changes greatly and the performance of the network fluctuates widely, then generally there are more subtypes that need to be supported, and on the contrary, there can be fewer. This can be reached through the capability negotiation process before the communication starts, and the communication parties can reach a consensus. The negotiation can be performed through current mainstream multimedia communication framework protocols such as H.323 or Session Initial Protocol (Session Initial Protocol, "SIP" for short).

Assuming that for the subcategories under a certain category, if S subtypes (S≤2 ⁹ -1) need to be distinguished, there are k representative generation parameters, denoted by p ¹ , p ² , ..., p ^k , then Table 2 gives an example of the corresponding relationship. The superscript in the table indicates the large type of FEC, and the subscript indicates the specific parameter.

Table 2 Correspondence between FEC Subtype and parameter setting scheme

FEC Subtype FEC encoding subtype (parameter setting) 000000000 FEC subtype 0 (p⁰₁, p⁰₂, .., p⁰_k) 000000001 FEC subtype 1 (p¹₁, p¹₂, .., p¹_k) 000000010 FEC subtype 2 (p²₁, p²₂, .., p²_k) 000000011 FEC subtype 3 (p³₁, p³₂, .., p³_k) ………………… ……………… S(S≤2⁹-1) FEC subtype S(p^S₁, p^S₂, .., p^S_k)

For example, for the Tornado code, the corresponding relationship can be set as: 000000010-(24,20) (the total number of data nodes=20, the total number of check nodes=4), 000000011-(30,20), ..., 111111111-others.

For a subtype of FEC encoding with certain characteristics, a given set of generation rules combined with corresponding generation parameters corresponds to a unique encoding scheme, which uniquely determines how to generate check nodes from data nodes and how to recover lost nodes. A database can be created to store the generation parameters corresponding to each major type and subtype. And the generating rules themselves are realized by hardware or software modules. Therefore, each large type corresponds to a FEC processing module at the sending end, which is responsible for generating check nodes; at the receiving end, it also corresponds to an FEC processing module, which is responsible for restoring nodes. However, corresponding to each large type of module, specific generation parameters of each subtype need to be read from the above-mentioned generation parameter database for processing. Therefore, both sides of the communication decide which FEC processing module to call and which generation parameters to read based on the information of the two information fields of FEC Type and FEC Subtype.

Due to the current development of multimedia communication technology, the H.264 video coding standard has gradually become the mainstream media coding format, so the second embodiment of the present invention provides the NALU of H.264 using ERRTP on the basis of the first embodiment. The specific steps of performing FEC encoding and decoding on the data stream are as follows.

The sender merges multiple (assumed to be S) H.264 NALUs into a group for encoding and transmission, and first re-divides the S NALUs into equal-length blocks, assuming M, and these M are data nodes.

In this step, the S NALUs of H.264 are divided into one group; then the S NALUs are connected end to end (concatenated) to form a large block, and then the large block is equally divided into M data blocks, wherein The length of each data block is K bytes. Here, if the total number of bytes of the large block (set to TB) cannot be divisible by M, then a rounding operation should be performed so that the length of each data block is Ceiling(TB/M) bytes, and the Ceiling function indicates rounding , that is, Ceiling(x) is equal to the smallest integer not less than x, and x is any real number. Then in some data blocks, the operation of filling zero strings (zero padding) may be used to make the number of bytes equal to Ceiling (TB/M)).

Next, perform FEC encoding on the M data nodes to obtain N check nodes. Use the FEC code to encode M data blocks to generate N check blocks. The generation process uses the method described above. According to the FEC Type and FEC Subtype information, determine which FEC processing module to call to generate the check block.

Then, the sender encapsulates all data nodes and check node groups in ERRTP packets for transmission. In this example the fields should be set as follows:

Type field FEC Type = 0010, indicating that Tornado code is used;

The subtype field is selected by the sender according to the actual situation. For example, the value is FECSubtype=000000010, which means that the Tornado (24, 20) code is used, in which there are 20 data nodes, 4 check nodes, and the channel coding redundancy is 16.7 %; when the packet loss rate is less than or equal to 3%, the erasure code can completely recover the lost data packets;

Data packet length Data-Length = K Bytes;

Packet Number = (M+N)/P, indicating the number of data nodes carried in one ERRTP payload.

After receiving these ERRTP packets, the receiving end decapsulates to obtain data nodes and check nodes. The receiving end takes P data packets as a cycle, and starts decoding and recovery every time a group of P data packets are received. The number of data packets in a group is determined through negotiation between the two parties.

The receiving end performs error resilient decoding on the data nodes according to the check nodes. Each time after receiving the data packet P+1, it starts to detect whether there is any data packet loss in the previously received P data packets. If so, use the method described above to determine which FEC to call according to the FEC Type and FEC Subtype information. The processing module decodes and recovers or partially lost data.

Finally, after obtaining the complete data node, re-merge to obtain a large block, and use the same method as the sender to divide and obtain S NALUs.

In practical applications, it is found that the above example adopts the anti-packet loss algorithm based on ERRTP, which can greatly improve the anti-packet loss capability of the video code stream without increasing the codeword by 17%. Compared with the RTP load header structure, only 4 bytes are added, which shows that the transmission efficiency is basically not affected, and a significant practical effect has been achieved.

Another key technical point of the present invention mentioned above is the realization of unequal protection. It is mainly reflected in two aspects, one is to select the appropriate encoding scheme or parameters according to the multimedia data of different importance levels, that is, to determine the aforementioned FEC encoding type and subtype, and the other is to select according to the network conditions at different times. Corresponding to these two aspects, it is called mixing and alternately using various FEC coding schemes. The so-called hybrid (Hybrid) refers to the use of multiple FEC subtypes at the same time, which is mainly used to protect data of different importance; the so-called alternation (Alternation) refers to the use at different times (under different network conditions) Different FEC subtypes.

Therefore, for the H.264 NALU data stream, as mentioned earlier, its header byte reflects the importance of the data, so the sender can evaluate the QoS level according to the NRI field or Type field in the NALU header information, and then choose the error elastic coding scheme , that is, to determine the FEC Type field and the FEC Subtype field. As for the network status, general network transmission has a corresponding network status monitoring mechanism. The sending end can obtain the transmission report fed back by the receiving end based on these mechanisms, so as to evaluate the network transmission status, and then choose the error elastic coding scheme, that is, determine the FEC Type field and FEC Subtype field.

The H.264 code stream is transmitted or stored based on NALU, and NALU is composed of NAL header information and NAL payload. In the NALU of H.264, different NALU types have different effects on decoding and restoring images. For example, if NRI is 0, it means that a Slice or Slice data strip of a non-reference image is stored in the NALU, which will not affect subsequent decoding; and if NRI is not 0, it means that a sequence/image parameter set or a Slice of a reference image is stored in the NALU. Or Slice data strips will seriously affect subsequent decoding.

Therefore, when carrying out packet protection to the code stream of H.264, the data of H.264 can be divided into two classes according to the value of NRI or Nal_unit_type: one kind is relatively important image data (for example Nal_ref_idc equals 1); The other type is secondary image data (eg Nal_ref_idc equal to 0). Then, the important image data can be protected by FEC1 code with high redundancy and strong anti-packet loss ability; while the secondary image data can be protected by FEC2 code with small redundancy and weak anti-packet loss ability .

Through this unequal protection algorithm, the correct recovery of various important information is guaranteed in the environment of high data packet loss, and the image information that cannot be recovered by the FEC2 code adopts technologies such as error concealment and prevention of error diffusion. FEC1 and FEC2 are just general representation methods here, representing any two subtypes. The two subtypes can belong to the same macrotype or to different macrotypes.

Obviously, the above method can be extended to a more general situation, and the data is divided into more categories according to the value of NAL_unit-type, such as five categories: the most important data, the second important data, the general important data, the less important data, and the least important data. Important data; can also be divided into 7 categories or more, then, the same number of FEC subtypes can be used to protect, and each type of data corresponds to a different subtype. As long as the protection ability is from weak to strong, these subtypes do not necessarily belong to the same large type. For the image information that cannot be recovered after being protected by the most protective FEC code, technologies such as error concealment and prevention of error diffusion are used.

Another situation of unequal protection is also within the scope of the present invention, that is, FECs with different protection capabilities can be selected according to real-time network conditions. Then the two sides of the communication are notified through the header information of ERRTP, so that they can correctly decode the data and recover the lost data. The situation that the network is currently affected by the degradation of transmission performance can be divided into several levels. For example, there are five levels: most serious, less serious, generally serious, less serious, and least serious; it can also be divided into seven or more levels, then the same number of FEC subtypes can be used for protection, and each level corresponds to a different type Subtype. As long as the protection ability is from weak to strong, these subtypes do not necessarily belong to the same large type. For the image information that cannot be recovered after being protected by the most protective FEC code, technologies such as error concealment and prevention of error diffusion are used. Awareness of network conditions can be realized through various existing QoS monitoring methods.

More complex application schemes are also within the scope of the present invention, if there are a total of T kinds of FEC schemes (different types/subtypes) available (supported by both communication terminals). Deciding which FEC to use depends on the importance of data and the status of the network. Then a two-dimensional LUT method can be used, as shown in Table 3:

Table 3 Two-dimensional LUT with mixed and alternate use of multiple FEC mechanisms

In the above table, the data importance level and network status level are arranged in ascending order. The subscript of FEC is represented by a two-dimensional subscript, and the error resilience mechanism FEC(i, j) in the table, 0<i≤U, 0<j≤V, can be any one of the above T FEC schemes.

It should be mentioned that in the descriptions of the above-mentioned embodiments of the invention, FEC erasure codes, especially Tornado codes, are used as examples, but other similar error resilience mechanisms, especially FEC coding schemes other than Tornado codes, are applicable, and are not affect the spirit and scope of the present invention.

In another embodiment of the present invention, an improved Tornado erasure code is specially adopted, and this improved Tornado erasure code only generates one layer of check nodes for a group of data nodes, which can greatly reduce coding Delay, to meet the needs of real-time communication.

In real-time video communication, the use of FEC code data packet protection will introduce time delay, and the size of the time delay is related to the size of the image data data packet. Divide S NALUs into a group, where one NALU contains code stream data of one Slice. If a frame of image is divided into a Slice, there will be a delay of S frames at the encoding end, and there will be a delay of S frames at the decoding end. The relationship between NALU and the number of data nodes is shown in the following formula:

$\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; the\; s</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; NalSize</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; Pack\; Size</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; DataNodes</span>}$

In the formula, the sum of S NALU length values is equal to the number of data nodes multiplied by the size of each node's data packet. It can be seen from formula (1) that when the value of S is limited, the value of PackSize×DataNode will also be limited. In addition, due to the effectiveness of IP network transmission, the value of PackSize cannot be too small, so the value of DataNode is limited .

In real-time video communication on an IP network, the delay T _total of one frame of image is calculated as follows:

T _total ＝T _FEC +T _codec +T _trans

In this formula, T _FEC is the time delay introduced after adding FEC protection, and T _codec and T _trans are H.264 codec processing time delay and network transmission time delay respectively. Due to the rapid development of digital signal processing technology and IP network, it can be assumed that both T _codec and T _trans can meet the real-time requirements:

T _codec <= T _th , T _trans <= T _th , where T _th = 1/F _target

In the formula, F _target is the decoding target frame rate (possible value 10Hz, 30Hz, etc.), and a frame of image is divided into a Slice, then formula (2) can be changed to:

T _total <= S*T _th +2*T _th ＝(S+2)*T _th

It can be seen from the above two formulas that the delay T _total of a frame of image is basically determined by the value of S, and the DataNode greatly affects the value of S. Therefore, on the premise of ensuring the anti-packet loss capability of video communication, the delay introduced by FEC should be reduced as much as possible to further ensure the QoS of real-time video communication.

The present invention adopts an improved Tornado code protection algorithm under the condition that DataNode is limited. The improved Tornado method does not use the coding method of multi-level even graphs, but only uses the coding method of one layer of check nodes. Compared with the original Tornado encoding method, the improved encoding method greatly improves the flexibility of the algorithm, the number of data nodes and check nodes can be set arbitrarily, and the complexity of the encoding and decoding algorithm is also reduced, which can be used for real-time video communication resistance to packet loss. In addition, in the case of limited data nodes, the anti-packet loss performance of the improved Tornado code basically does not decrease. The specific principles and detailed steps of the improved Tornado encoding method will be described in detail later.

Note that the encapsulation of NALUs by ERRTP in the above example does not mention how to process the information headers of NALUs. On the basis of the third embodiment of the present invention and the second embodiment, ERRTP processes NALUs of the same type together and integrates the header information into In the ERRTP header information. The most basic difference from RTP is that in the ERRTP encapsulation process, the header information of NALU packets with the same header information is integrated into the ERRTP header information.

The structure of the NALU header information has been mentioned before, and here it is emphasized again that the NALU information contains in turn:

The 1-bit F field is used to indicate whether the NALU has an error;

The 2-bit NRI field is used to indicate the importance of the NALU;

The 5-bit Type field is used to indicate the type of the NALU.

The execution steps of the sending and receiving parties are as follows. The sender encapsulates multiple NALU data nodes or check nodes with the same header information in the same ERRTP packet according to the ERRTP encapsulation format. According to actual engineering experience, in general, because the H.264 bit stream always has the property that the corresponding NALU types of the adjacent parts are the same, this assumption can always be satisfied. Even if it cannot be satisfied in some cases, there are several countermeasures to deal with such a situation: the first one can accumulate NALUs of the same type until a certain number is met and then encapsulate them into ERRTP; If the number of NALUs of different types does not reach a certain number, the method of RTP filling is used. Although a bit of bandwidth is wasted, it is insignificant. Another method is that if there are many NALUs of different types, RTP encapsulation can be used. Anyway, it is receiving The end can be identified according to the ERRTP identifier, and corresponding processing is performed.

In the ERRTP encapsulation format mentioned above, the same header information of the NALU carried by it is integrated in the header information of the ERRTP packet, and the NALU carried by it is removed from the header information and then according to the aforementioned Process processing, performing division, encoding and encapsulation, filling into the payload of the ERRTP packet. So how to integrate the NALU header into the ERRTP header? Two sets of solutions will be given below to solve these problems.

In the ERRTP encapsulation format, the NRI field and the Type field in the NALU header information are filled in the PT field of the ERRTP header information. As mentioned above, the PT field is located in the last 7 bits of the second byte of the ERRTP header information. The format of such an ERRTP header has been given in Figure 7, where the parts different from RTP have been indicated in bold, and some parts in the figure will be explained later.

The other two points are: first, use the V field in the ERRTP header as the ERRTP identifier, as mentioned above; second, the F field in the NALU header information is filled in the M field of the ERRTP header information, and the M field is located in the ERRTP The first bit of the second byte of the packet header information, at the receiving end, judges whether the NALU carried by it is wrong according to the M field of the ERRTP packet, and realizes the forbidden bit function of the F field. It can be seen that the solution can tell the receiver of the RTP data packet that the RTP protocol is ERRTP through the difference of the version, so that the subsequent processing will be performed according to the processing flow for the ERRTP protocol.

In this solution, the NALU header information byte (8 bits) replaces 1 bit of the identification M field and 7 bits of the PT field in the original RTP header information with a total of 8 bits. The specific replacement order can be as follows:

F bits replace M bits;

NRI 2 bits replace the highest 2 bits of PT 7 bits;

Type 5 bits replace the lowest 5 bits of PT 7 bits;

In fact, such an alternative is reasonable. The 7 bits of PT can be used freely, as mentioned earlier. The purpose of the M field is stipulated in RTP (RFC 3550) as follows: a specific level (Profile) can stipulate that M bits are not used, but it is incorporated into PT, so that PT can have up to 8 bits, distinguishing 256 different type. Therefore, replacing M bits with F bits is completely in line with RTP regulations, and will not cause intercommunication problems between ERRTP and traditional RTP.

It can be easily seen that the encapsulation format of ERRTP of the present invention has three obvious advantages: the first, less overhead, especially when there are a plurality of NALUs in one RTP, it obviously saves the number of transmission bits; The relative importance of these NALUs can be judged by decoding the H.264 NALU data; thirdly, it is not necessary to decode the H.264 NALU data in the RTP data packet to identify whether the RTP packet is correct due to the loss of other bits decoding.

In order to further describe the technical details of the present invention in detail, a process description of ERRTP encapsulation and decapsulation is given below. After the above processing, the multiple H.264 NALU types in the same ERRTP packet are exactly the same, that is, their header information bytes are the same, then when they are divided, encoded, and encapsulated into the ERRTP packet, The original header information bytes can be stripped, so that if there are N NALUs, N bytes can be reduced. When decapsulating, it is to extract and decode the NALU from the ERRTP data packet, re-divide and restore it to the original form, that is, extract and decode the N NALUs from the ERRTP data packet in which they are located, and then extract and decode the PT in the ERRTP header information. The bits are copied to the lowest 7 bits of a byte H (8 bits), and the highest bit of H is set to 0 as the F bit. Then append the generated H byte to the front of each extracted NALU, thus restoring each NALU. Of course, if the F field in the ERRTP packet header is 1, it means that the NALU in the ERRTP packet is wrong, so it can be discarded directly, which also saves processing time.

The second solution is given below, which is the same as the first one, that is, the NRI and Type fields in the NALU header are filled into the 7 bits of the PT field in the ERRTP header. There are two different places: the M field is used to identify ERRTP, and a problem brought about by this is that there is no place to fill the F field. In this embodiment, the two types of NALUs whether the F is set or not are treated separately, and the error NALU for the F set The original RTP transmission is still used, and the normal ERRTP transmission is used, but the F bit is ignored. The specific details are described below.

The value of the M field is 1 to identify the ERRTP packet, and the M field is located in the first bit of the second byte of the ERRTP packet header information. As for the F bit, it is stipulated in the H.264 protocol that if there is a syntax conflict or an error, it is 1. When the network recognizes a bit error in this unit, it can be set to 1 so that the receiver discards the unit. It is mainly used to adapt to different types of network environments, such as the combination of wired and wireless environments. The specific usage principle is: in general, when the sending end and the receiving end of the communication perform H.264 encoding and decoding on the video, they do not perform a "write" operation on the bit, and the decoding end performs a "read" operation on the bit. If F=1 is found, the receiving end will discard the NALU during decoding. According to the current general application situation in the industry, the "write" operation for the F bit is mainly performed on the gateway between two different networks, such as the case of encoding conversion (MPEG-4 to H.264, H. 263 to H.264, etc.).

Therefore, the present invention ignores the F bit, which does not meet the purpose defined by the original H.264. Therefore, the M field originally used to fill the F bits can be reserved for future expansion to carry more information, which is used to identify the ERRTP packet here. The advantage of doing this is that there is no need to modify the version information V=2, and ERRTP still uses the value 2 of the original version V. This also saves the current only RTP version information resource.

However, it is unavoidable that there may be a small probability situation in which F bits need to be used in practical applications. For example, when the NALU syntax is wrong, the present invention handles this situation as follows: In the ERRTP encapsulation format, ignore the NALU header information However, at the sending end, RTP packet encapsulation is still used for the effective error NALU of the F field, and only the normal NALU is encapsulated with ERRTP; at the receiving end, it is judged whether the packet is ERRTP or RTP packet and then the corresponding encapsulation format is adopted Process the package. That is to say, when the F bit is used for the purpose defined in the original H.264 in some special cases, that is, it is used to indicate possible H.264 NALU syntax errors. When performing video encoding according to the H.264 protocol, if a NALU is found to have a syntax error, then the NALU must be encapsulated separately.

Summarize the method flow process of above-mentioned ERRTP and RTP alternate processing as follows:

The sender first judges whether the F field in the header information of at least one NALU is valid, and accordingly divides it into normal NALU and error NALU;

Then normal NALU is encapsulated into ERRTP packet by ERRTP encapsulation format, and ERRTP mark is established; Error NALU is encapsulated into RTP packet by RTP encapsulation format;

The receiving end first judges whether the header information of the received packet has an ERRTP flag, and divides it into an ERRTP packet and an RTP packet;

Then process the ERRTP packet according to the ERRTP encapsulation format, and process the RTP packet according to the RTP packet encapsulation format.

It can be seen that the gateway performs ERRTP encapsulation for normal NALUs according to the method described above, and for H.264 NALUs of the same type according to certain rules (determined by specific applications, mainly specifying how many similar NALUs are encapsulated in each ERRTP packet) , once a syntax error is found in a certain NALU, the conventional RTP encapsulation will be used for the NALU. At this time, the regular RTP packet may contain only one H.264 NALU.

The last point to be explained is that, noticing the type of NALU and the value of the corresponding Type field given in Table 1 mentioned above, it can be found that there are less than 16 existing types, that is to say, 5 bits of Type It can be reduced to 4, which does not affect the existing H.264 transmission. Therefore, in the ERRTP encapsulation format, when all types of NALU are less than 16, only the lower 4 bits of the Type field are used to represent, and the highest of Type Bits are reserved as extended bits, called the C field. Reserve the C bit for later use and continue with functional expansion. After bit C is reserved, the NALU type given in Table 1 needs to be modified accordingly: a total of 16 values, the values 0-12 are the same as those in Table 1, and the values 13-15 are reserved.

Of course, although there are only 13 NALU types in H.264 at present, H.264 will develop in the future, and more NALU types may be produced. If the NALU types increase to more than 16 in the future, then it is still necessary to use the lowest PT7 bits. 4 bits plus C bit for type indication.

It should be mentioned that the biggest advantage of integrating the NALU header information into the ERRTP header information here is that the multimedia transmission device can directly know the relevant information of the NALU carried by it according to the ERRTP header information, and implement H.264 multimedia data accordingly. QoS policy delivered in real time. This point cannot be realized in the existing RTP, because for the RTP layer, the NALU layer information is not concerned, and the header information of each NALU in the payload cannot be known, so that the QoS policy cannot be implemented.

On the basis of ERRTP, in order to realize the feedback of the receiving end, the enhanced technology of SEI carrying QoS report is adopted. From the previous description, it can be seen that RTCP undertakes the QoS reporting mechanism, but it is actually a general reporting method that can be used to report QoS , which can also be used to report other information. For specific video communication applications, using RTCP to report may not be the most appropriate. At some point, if both the sender and the receiver of the QoS information can use a higher layer protocol such as H.264 to communicate, it is entirely possible to consider using H.264 to carry the content of the report. Based on this starting point, the present invention directly adopts H.264 to bear QoS report information, avoids the use of extra channels, and realizes an "in-band" report mechanism.

Another basis for transmitting QoS reports by the H.264 high-level protocol is that in current video communication applications, adaptation measures for network transmission are mainly based on terminals, rather than network intermediate devices such as routers, switches or gateways. . Therefore, the encapsulation and extraction of the QoS report does not depend on the underlying protocol. Only the terminal can understand and extract the QoS report information carried in H.264 to realize QoS monitoring, so it does not depend on the underlying RTCP and other protocols. Of course, by adopting the "in-band" reporting mechanism of H.264, it does not mean that the application of the RTCP reporting mechanism is excluded. The two mechanisms can be used selectively or can coexist. The use of H.264 can reduce the reporting traffic of RTCP. In addition, if the H.264 "in-band" reporting method is used, multiple protection measures can be taken for the H.264 data packets, and the H.264 data packets carrying the QoS report can be considered as important data, depending on the According to the principle of Unequal Protection ("UEP"), high-intensity protection measures can be adopted for it. Therefore, the correct arrival of the report data can be ensured, and the reliability of QoS monitoring can be improved.

On the basis of the third embodiment, the fourth embodiment of the present invention carries the QoS report based on the extended message mechanism of H.264, which is roughly divided into the following three basic steps

First, each multimedia communication terminal statistically generates the QoS report of H.264 multimedia communication. The content of these reports can be the same as the SR and RR report content of RTCP, and of course it can also be different, but the described service quality of H.264 media communication and network status and other information are consistent;

Then, the terminal uses the H.264 extended message to carry these QoS reports and sends them to other communication terminals. The H.264 extended message mechanism has been mentioned above, typically SEI, etc., and what the present invention uses is basically the SEI message. In the future, the extension of H.264 can also be carried by other extended messages;

While sending QoS reports, the terminal also receives QoS reports from other terminals. In fact, each terminal will execute QoS policies according to these QoS reports.

The present invention uses the SEI message to carry the QoS report. Taking the existing RTCP QoS report as an example, the main content of the SR and RR reports of RTCP can be directly used as the load of the H.264 SEI message, so as to be carried by the extended SEI message these messages.

Based on this idea, in the fourth embodiment of the present invention, a specific SEI extension message is defined specifically for carrying the QoS report. According to H.264, SEI information is stored in a type of NALU, and its Type=6 as mentioned above. The invention stores the SR and RR report messages similar to RTCP in the SEI domain, which not only ensures the transmission efficiency, but also effectively feeds back the channel state and decoding information, and is convenient for the coding end and the decoding end to interactively resist data packet loss. The specific structure is shown in Figure 8, except that the header information is arranged according to the SEI message structure, other QoS report contents refer to the SR and RR report formats of RTCP.

The header information of the SEI message used to carry the QoS report contains the following fields:

The first byte (byte 0) is the load type field (SEI Type), which is used to indicate that the load is a corresponding QoS report. In this embodiment, SEI Type=200 means that what is stored in the SEI domain is a transmission similar to RTCP Report (SR), and SEI Type=201 indicates that it is a reception report (RR);

The second and third bytes (bytes 1 and 2) are the payload length field (SEI Packet-Length), which is used to indicate the length of the corresponding QoS report, which has the same definition as the length field in the RTCP QoS report;

The fourth and subsequent bytes are the load of the SEI message, which is used to fill the corresponding QoS report.

The QoS report is also divided into a sender report and a receiver report, which are distinguished by the payload type field indication, that is, the value of the SEI Type is different, and the specific content of the QoS report can be the same as the SR and RR reports of RTCP, as shown in Figure 2:

The version information field (V) occupies 2 bits, and the value of this example is binary 11, that is, V=3, indicating the difference from the previous version;

The padding field (P), which occupies 1 bit, is used to indicate whether there is padding content, which is the same as RTCP;

The received report number field (RC), which occupies 5 bits, is used to indicate the number of received report blocks reported in the QoS report;

The sender SSRC field, which occupies 32 bits, is used to identify the sender of the service quality report;

For the sender report, the sender information block is also included here, which is used to describe the relevant information of the sender of the report;

After that, it contains multiple receiving report blocks, which are used to describe the multimedia statistical information from different sources. Each block contains the identifier of the source and the relevant statistical indicators of the multimedia stream. The meanings of various indicators have been described in the previous RTCP;

Finally, it contains layer-specific extensions, which are reserved for layer-specific extensions.

It can be seen that the content of the QoS report given in Figure 8 is basically the same as that of RTCP. After the basic contents of RTCP, RR and SR, are written into the SEI field, no special logical channel is needed to transmit RTCP information, which saves some bandwidth overhead. In fact, the essence of the present invention is to use SEI messages for in-band bearer. As for how to generate QoS reports statistically, as long as the purpose of QoS monitoring can be realized, the essence and scope of the present invention will not be affected.

After the QoS report is realized, a variety of QoS strategies can be implemented on this basis, such as using the cumulative data packet loss field of RTCP, which can be used to feed back decoding information in two-way video communication (the terminal has both an encoder and a decoder) , to facilitate interactive anti-packet loss.

In addition, there are fields such as arrival delay jitter and sender byte count in the QoS report, which can be used to perceive the network status. Among them, the rate control algorithm can further ensure that the rate of the encoder is close to constant according to the information in the arrival delay jitter field; the byte count field of the sender can estimate the average rate of the load, which is convenient for the sender to reset the encoder parameters according to the network status. Including adjusting the target frame rate, restoring the image quality and resolution of the original image, and more.

In order to improve the reliability of RTCP transmission, after adopting the H.264 "in-band" reporting method, H.264 data packets can take multiple protection measures, and for H.264 data packets carrying QoS reports, it can be considered as For important data, according to the principle of unequal protection, high-intensity protection measures can be adopted for it. Thus, the correct arrival of the report data can be guaranteed. For example, the SEI used to carry the QoS report should be further carried by the NALU, and as mentioned above, the NALU has a header information that can set the importance of the content, so the communication terminal can set the NALU according to the reliability requirements of the QoS report transmission The nal_ref_idc field can be set to 1, 2, 3, etc. In error elastic coding, protection measures with different strengths will be taken according to the level of this field.

The communication terminal can also dynamically adjust the sending period of the QoS report based on the SEI message according to the current network status and high-level application requirements. By default, the time interval for writing RTCP information into the SEI field (that is, the reporting period) is consistent with the RTCP transmission interval recommended in RFC 3550. Of course, according to the needs of specific applications (specific protection methods, etc.), the reporting period may not necessarily be exactly the same as that stipulated in RFC 3550, but can be adjusted. The reporting period is determined by the needs of a particular application. For example, an important use of reporting data is to dynamically estimate the performance of the network: packet loss rate, delay, jitter, etc. If these data need to be detected frequently, the reporting period should be short, otherwise the reporting period can be long. When the network condition is good, the report can be stopped. In addition, the SEI message can not only transmit the QoS report of H.264 video, but also carry the QoS report of multiple media streams in a mixed way, just add the corresponding receiving report blocks of various media streams after the QoS report. Such as audio stream, etc., as long as the SSRC specific report block content of the source is added in the SR report. As mentioned earlier, in addition to using the SEI for in-band monitoring, the communication terminal can also choose the existing RTCP transmission, or use one or both of the H.264 extended message and RTCP to transmit the bearer QoS report.

After the use of SEI to realize the QoS report related to the network status fed back from the receiving end is given, based on this, it is easy to realize adaptive protection strategy adjustment, including multi-level protection and unequal protection. According to the problem that the prior art cannot adaptively adjust the network communication status, the fifth embodiment of the present invention provides an adaptively protected video transmission method that counts the current communication status and adaptively adjusts the protection strategy. First, according to the performance impact of the protection method, different parameter configurations are given, and multi-level protection strategies with different protection capabilities are set, which are used to be selected for efficient and reliable protection under different communication conditions; secondly, the network status is counted at the receiving end according to the communication conditions , communication quality, and send it back to the sender; finally, the sender adjusts according to the communication quality statistical information sent back, and selects the most appropriate protection strategy level.

The key of this scheme also lies in the method of statistical communication quality and the channel of sending back statistical information. Using the serial number loss of H.264 NALU, the packet loss rate and its location can be counted, and by defining the extended SEI message structure of the payload part in the NALU, it is used to carry the statistical information, and the statistical data is transmitted from the receiving end to the sending end . Although such a feedback mechanism is not the same as the SR/RR format of the QoS report, those skilled in the art can understand that the fundamental principles of the two methods are the same, but the content carried by the SEI is different, so the following descriptions are different. Then it specifically proposes the difference between the scheme of SEI bearer network packet loss rate and the scheme of QoS report.

Taking the Tornado erasure code as an example, the video stream data is protected according to the encoding and decoding method of the aforementioned Tornado erasure code. Tornado erasure code needs to set the parameters: the number of data nodes, the number of check nodes, the shrinkage ratio, the number of layers of check nodes, and the bipartite graph at all levels used to calculate the check nodes. In the process of video stream communication, the sending end divides the video stream data into data nodes, and then generates check nodes according to the Tornado encoding method, and sends them to the receiving end together; the receiving end performs error correction according to the Tornado decoding method to obtain video stream data.

Since factors such as the actual IP network bandwidth are often changing and unstable, a fixed protection strategy will bring problems such as low efficiency or high bit error rate. Therefore, this embodiment pre-sets a series of protection strategies with different levels of protection strength. They are respectively used to protect video stream data at different communication quality levels. It can be seen that different levels of protection strategies can adapt to changes in network communication quality, not only can meet the requirements of protection strength in the case of channel degradation, but also can properly reduce the protection strength in the case of signal improvement, so as to reduce system overhead and save processing and bandwidth resources .

In order to specify different levels of protection strategies, Tornado erasure codes with different parameters need to be set. According to the aforementioned parameters that affect the protection performance of Tornado erasure codes mainly include the number of data nodes, the number of check nodes, and the random distribution of node degree vectors on both sides of the bipartite graph, for the sake of simplicity, Tornado codes with different capabilities generally do not have a unified For bipartite graphs, different numbers of data nodes and check nodes are used to give Tornado erasure code protection strategies with different protection strengths. According to the principle of Tornado erasure codes, different numbers of data nodes and check nodes can determine Tornado erasure codes with different code rates or redundancy rates, thus giving different protection strengths and system overheads.

The receiving end receives the data and performs Tornado erasure code decoding to obtain the video stream data. At the same time, statistics are made according to the data loss, and the statistical information is obtained to represent the communication quality.

The sending end needs to adjust the protection strategy according to the communication quality status, so it needs to make statistics on the transmission status, and the receiving end calculates the transmission status according to the serial number of the NALU of the H.264 video flow data. In the two-way video communication based on H.264, each terminal of the communication system has both an encoder and a decoder. The NALUs are sequence numbered, that is, all NALUs sent by the sending end have a unified sequence number, so the receiving end can judge whether any NALU is lost according to the sequence number of the received NALU. If there are discontinuous NALU serial numbers, it means that there is a NALU loss. The interrupted NALU serial number is the serial number of the lost NALU, and its number is the number of lost NALUs. After a period of accumulation, the total number of NALUs lost during this period can be calculated, and then the number of all NALUs within this period can be normalized to obtain the Accumulated Lost Slice Rate (Accumulated Lost Slice Rate, referred to as "ALSR") "). Of course, the receiving end can also directly send the packet loss information back to the sending end, and the sending end will make statistics. The use of NALU serial numbers for statistics not only ensures the accuracy of statistical information, but also directly utilizes existing data information without additional bearer overhead.

The receiving end sends statistical information and other data loss information back to the sending end through the extended SEI message. After the statistical information about the transmission situation is obtained by the receiving end, it needs to be sent back to the sending end. This embodiment defines an extended SEI message structure, which is specially used to carry the statistical information about the transmission situation sent back from the receiving end. After finishing the statistics, the receiving end writes the information into the specially defined extended SEI message body, then writes it into the SEI field of the coded stream sent back by the terminal, and sends it back to the sending end. After receiving the SEI message, the sender can directly know the statistical information, or obtain the ALSR through statistics, so as to establish a true perception mechanism of the sender for the network packet loss rate.

As mentioned above, the SEI message is also carried by the basic unit NALU of the H.264 code stream. Each SEI field contains one or more SEI messages, and the SEI message is composed of SEI header information and SEI payload. The SEI header information includes two codewords: payload type and payload size. The length of the payload type is not necessarily the same. For example, when the type is between 0 and 255, it is represented by one byte, when the type is between 256 and 511, it is represented by two bytes 0xFF00 to 0xFFFE, and so on, so that users can customize Any number of load types. In the existing H.264 standard, types 0 to 18 have been defined as specific information, such as cache period, image timing, and the like. It can be seen that the SEI field defined in H.264 can store enough user-defined information according to requirements. In the first embodiment of the present invention, an extended SEI message for carrying statistical information is defined in the reserved SEI payload type.

Finally, the sender adjusts the Tornado erasure code according to the statistical information sent back, and uses a protection strategy that is more suitable for the current transmission situation. Finally, the sender will adjust the protection strategy according to the statistical information, that is, select a protection strategy with an appropriate level. Here, the sender also presets a series of judgment thresholds corresponding to different protection levels, sets the thresholds for entering each level, and then selects the corresponding level according to the threshold where the ALSR falls. The statistics, feedback and adjustment mechanism of the transmission situation thus established can accurately and timely adapt to the network transmission requirements and improve the protection capability.

Different series of protection strategies are adopted for data of different importance. Considering the different protection requirements of key data and non-key data, in order to further improve the adaptability, two different protection strategy series are set up, which are used to protect key data and non-key data respectively. In this way, the data of two different communication requirements can be processed independently, and the protection strategy can be selected according to the protection strength suitable for each requirement, so as to improve the system efficiency.

For example, different levels of Tornado codes are used as a series of protection schemes, and their protection capability levels are characterized by parameters n and 1, where n represents the number of data nodes and 1 represents the number of check nodes. Use TN(n+1, n) to represent the Tornado code protection scheme determined by parameters n and l. Therefore, the series of protection schemes corresponding to key data are: TN _K (n ₀ +l ₀ , n ₀ ), TN _K (n ₁ +l ₁ , n ₁ ), ......, TN _K (n _L-1 +l _L-1 , n _L-1 ); the same series of protection schemes for non-critical data are: TN _NK (n ₀ +l ₀ , n ₀ ), TN _NK (n ₁ +l ₁ , n ₁ ), ......, TN _NK (n _L-1 +l _L-1 , n _L-1 ). Set the threshold value series 0<G ₁ , G ₂ , . . . , G _L-1 <1, which is used to judge and select the protection level. When adjusting the protection strategy, the sending end performs the following operations according to the relationship between ALSR and the thresholds G ₁ , G ₂ , . . . , _GL-1 :

If 0<AlSR<G1, use TN _K (n ₀ +l ₀ , n ₀ ) to protect critical data, and use TN _NK (n ₀ +l ₀ , n ₀ ) to protect non-critical data;

If G _i <AlSR<G _i+1 , i=1, 2, ..., L-2, then use T _NK (n _i +l _i , n _i ) to protect key data, and use TN _NK (n _i +l _i , n _i ) protect non-critical data;

If G _L-1 <AlSR<1, then use TN _K (n _L-1 +l _L-1 , n _L-1 ) to protect key data, and use TN _NK (n _L-1 +l _L-1 , n _L-1 ) Protect non-critical data.

In addition, the sending end also resends the lost data information sent back by the receiving end. When the receiving end counts the lost NALU information, it also obtains the positioning information of the image frame corresponding to the lost NALU, and the information includes the sequence number of the frame and the position in the frame. The receiving end sends the positioning information back to the sending end, and the sending end can locate the corresponding video stream data and send it again. In real-time video communication, the video stream data with too long delay has lost its value, but under certain business requirements or a certain mechanism, data with a certain delay still has value, such as in a large buffer range In video communication, as long as the delayed video stream data still falls in the buffer, these data can be used to avoid interruption of video stream playback. It can be seen that the retransmission mechanism is of great value in improving the reliability and service quality of video stream communication.

In addition to adopting the error resilience protection strategy, the sixth embodiment of the present invention starts from the two aspects of error concealment and error diffusion elimination on the basis of the fifth embodiment, and combines the error concealment strategy of the receiving end and the error diffusion elimination of the sending end Strategy, in order to achieve the goal of minimizing the loss of video quality caused by bit errors and avoiding the spread caused by bit errors. For error concealment, a simple alternative can be used to achieve the effect of compensating bit error losses with as little complexity as possible; for bit error diffusion elimination, an error information feedback mechanism is established through the existing channels of H.264, and intra-frame implementation is implemented according to the feedback. Encoding to achieve the effect of diffusion elimination without adding additional network loads, ensuring the robustness of the video stream to bit errors, and thus avoiding error diffusion caused by error concealment.

The basic idea of this scheme is to discover the missing data information, such as the location of the Slice, by counting the NALU sequence numbers at the receiving end. On the one hand, an efficient algorithm is used to simply replace the missing Code information is fed back to the sender. Through the extended SEI message of H.264, an error information feedback channel from the receiving end to the sending end is established. After learning the bit error information, the sender immediately adopts the strategy of performing intra-frame coding segment by segment and refreshes the bit error Slice segment by segment to prevent bit error from spreading.

During the H.264 video communication process, the sending end encodes the video stream data to be sent to obtain the video code stream, then encapsulates the NALU and sends it to the receiving end through a packet message. The receiving end receives the message and decodes it. At this time, the receiving end needs to judge whether the video stream data is lost, so as to perform subsequent error elimination operations. The error elimination process is roughly divided into three major steps: cover, feedback, and diffusion elimination.

First, the receiving end judges whether data is lost according to the interruption of the NALU sequence number, and counts information about the lost data, that is, bit error information. As mentioned above, NALU is the basic unit of H.264 video stream data transmission, and each NALU has a unique serial number. Therefore, the receiving end knows which NALUs are lost according to whether the received NALU sequence numbers are interrupted. An error masking strategy for missing data can thus be implemented. The use of NALU serial numbers for statistics not only ensures the accuracy of statistical information, but also directly utilizes existing data information without additional bearer overhead.

First, the receiving end obtains the serial number by identifying the received NALU header information, and detects the occurrence of bit errors due to the discontinuity of the serial number, and learns the video data that the missing NALU should carry through the previous NALU, and locates the data loss caused by the bit error, for example The previous NALU of the lost NALU carried the first Slice of the Nth frame, and it can be inferred that the position of the Slice carried by the lost NALU should be the next Slice of the frame according to the transmission sequence.

Then the receiving end needs to re-synchronize the video information. Since the H.264 video stream is continuously transmitted, the receiving end and the data stream need to be synchronized before receiving it correctly. Once the data stream is interrupted, the receiving end needs to re-synchronize. Resynchronization of the decoder is done by finding the next NALU header after the break. In this process, the receiving end also needs to judge the number of NALUs lost in the middle and their location information through the serial number of the next NALU.

After that, the receiving end needs to perform error concealment, and the NALU with lost data is completely discarded, so the entire Slice carried by the NALU is lost. The error concealment strategy is to replace the lost data with adjacent data in the time domain or space domain through simple replacement, such as Use the Slice recovery image data of the frame corresponding to the frame where the missing data is located to cover up.

After obtaining the bit error information, the receiving end feeds it back to the sending end. Feedback error information requires a feedback channel. In order to reduce the network burden and simplify the implementation mechanism, the first embodiment of the present invention uses the existing H.264 communication mechanism to define extended SEI messages for carrying error information and establishing feedback. So that the sender can combine the bit error information to prevent bit error from spreading. In fact, only by combining the bit error information feedback mechanism and the bit error diffusion elimination strategy of the sending end, can the bit error diffusion caused by the previous error concealment strategy implemented by the receiving end be avoided.

In the previous embodiment, the extended SEI message of H.264 is used to provide an information feedback mechanism from the receiving end to the sending end, so that the sending end can know which NALUs are lost in time, so that effective bit error diffusion can be effectively eliminated in time, Prevent future error propagation caused by these lost data.

The advantage of establishing an information feedback mechanism within the H.264 system is to save network bandwidth overhead, save system processing resources, and not affect interoperability. The following describes how to define an extended SEI message. As mentioned above, the SEI message is also carried by the basic unit NALU of the H.264 code stream. Each SEI field contains one or more SEI messages, and the SEI message is composed of SEI header information and SEI payload. The SEI header information includes two codewords: payload type and payload size. The length of the payload type is not necessarily the same. For example, when the type is between 0 and 255, it is represented by one byte, when the type is between 256 and 511, it is represented by two bytes 0xFF00 to 0xFFFE, and so on, so that users can customize Any number of load types. In the existing H.264 standard, types 0 to 18 have been defined as specific information, such as cache period, image timing, and the like. It can be seen that the SEI domain defined in H.264 can store enough user-defined information according to requirements

Then, the sending end starts to eliminate bit error diffusion according to the fed back bit error information. The error diffusion elimination method of joint error information is better than the existing error diffusion elimination method without feedback. Using bit error information, such as the location of the lost slice, the sender can take preventive measures against the lost slice, for example, avoid using the lost slice as a reference frame in future encoding, so as to shorten the decoding time of the receiver as much as possible. dependency.

Since H.264 encoding is based on Slice, that is, the data of the same Slice in the previous and subsequent frames has a reference relationship, and the data of the same Slice in the subsequent frame is encoded through the Slice prediction of the previous frame, so the error diffusion will also be limited to the same Slice internal. In the second embodiment of the present invention, the strategy of performing intra-frame coding segment by segment is adopted, that is, after sending a bit error, the Slice area of the subsequent frame is segmented into new Slices, such as dividing P macroblocks as A new Slice is then intra-coded to remove any reference or dependency of the Slice to the previously lost Slice. In order to ensure the transmission quality, the H.264 video real-time transmission system uses a data rate control scheme to limit the fluctuation of each frame of data, so that the amount of data per frame is balanced and the stability of video transmission is improved. Therefore, the amount of data to be intra-coded once in each frame, that is, the number of macroblocks, cannot be too large, otherwise it will exceed the H.264 data rate control range.

Fig. 9 shows the principle of error diffusion elimination in segmented successive intra-frame coding. When an unrecoverable packet loss error occurs at the receiving end, it detects and feeds back the error information to the sending end, that is, the frame where the Slice of the lost data is located and the positioning information within the frame are sent back to the sending end through the extended SEI message. The sender extracts the missing slice positioning information from the SEI message. For example, each frame in Figure 9 is divided into three Slices, namely Slice#0, Slice#1, and Slice#2, and Slice#1 of the nth frame is being transmitted lost, after which segmental successive intra coding needs to be performed.

First, in the nth frame, the encoding end divides Slice#1 in the order of macroblock scanning, and divides P macroblocks from the starting position to form a new Slice#3, and the remaining macroblocks are still Slice#1. At this time, there are four Slices, in which the new Slice#3 is intra-coded.

Next, in the n+1th frame, the newly formed Slice#3 divided in the previous step is sent as Slice#3 after intra-frame encoding, while other Slices are still coded according to the routine.

After that, it is necessary to judge whether there are still remaining macroblocks in Slice#1. If there are any remaining macroblocks, return to the first step and continue to segment the remaining macroblocks of Slice#1 into a new frame in the next frame, perform intra-frame coding and Sent until all macroblocks have been processed.

The number P of macroblocks divided each time above should meet the following conditions and be as large as possible to avoid the number of divisions, reduce processing delay, and shorten the scope of influence, but it needs to meet the aforementioned H.264 data rate control range. The number of macroblocks divided each time may be different, but the number of macroblocks divided last time will make all the macroblocks in the lost slice be processed.

For example, a frame of video stream data consists of 240 macroblocks. Initially, every 80 macroblocks is divided into a Slice, that is, 1-80 macroblocks are Slice#0, 81-160 macroblocks are Slice#1, 161-240 The macroblock is Slice#2. According to the calculation of the data rate, the appropriate segment value P is determined to be a segment of 12 macroblocks. Then, after Slice#1 is lost in the nth frame, the 80 macroblocks of Slice#1 should be segmented and sequentially intraframe coded. First, the first 12 macroblocks in the n+1th frame are selected for intraframe coding to form Slice#3 , so that in the n+2th frame, Slice#3 can use conventional predictive coding, and then the next 12 macroblocks are intraframe coded to form Slice#4, until the n+7th frame, the last remaining 8 macroblocks The blocks are intra-frame coded to form Slice#9, and then the flow of the error diffusion method of segmental successive intra-frame coding is completed.

According to the experimental results, it is found that after adopting the combined method of error concealment and bit error diffusion elimination of the present invention, the effect of the obtained video image is very good.

Finally, an improved Tornado coding scheme is proposed, which is used as an error resilience protection strategy in the seventh embodiment of the present invention. The main difference between this Tornado coding scheme and the traditional coding scheme is briefly pointed out below.

In the process of using Tornado codes for data transmission protection, setting multi-layer Tornado code check node layers will enhance the data transmission protection capability to a certain extent, but setting multi-layer Tornado code check node layers will also make Tornado code check nodes layer The amount of calculation is large, so that the data is paid for a long time delay in the process of transmission protection. If the number of check node layers can be reduced while ensuring that the data transmission protection capability is not significantly reduced, the calculation load of the Tornado code can be effectively reduced, and the time delay in the data transmission process can be greatly reduced, thereby seeking a higher data transfer protection performance-cost ratio. Therefore, the seventh embodiment of the present invention is: setting an erasure code with only one check node layer, and performing data transmission protection according to the erasure code.

The Tornado erasure code scheme only has one layer of check nodes, and the intermediate check node layer of the Tornado code is removed. Similarly, the inherent requirement of generating the last layer of check nodes according to the Reed-Solomon code in the Tornado code is also removed. In this way, the erasure code of the present invention, as shown in Figure 10, has only one layer of data node layer and one layer of check node layer. It can be said that the erasure code of the present invention is a Tornado code with a simplified structure, which is a Improved Tornado code.

The data node size L1 of the improved Tornado code of the present invention, the number n of data nodes in the data node layer, and the number L of check nodes in the check node layer can be determined according to actual requirements. For example, determine the data node size L1 in the data node layer and the number of data nodes n contained in the data node layer based on factors such as data transmission rate, data type such as audio data/video data, data protection capability requirements, and the maximum network delay that can be received , the number L of check nodes included in the check node layer.

最后层与第m层之间的节点数目的等比递缩因子为则现有技术中Tornado码的总节点数Total_Node为:If it is assumed that the Tornado code in the prior art has m layers of intermediate check node layers, and from the data node layer to the mth intermediate layer, the proportional shrinkage factor of the number of nodes between adjacent two layers is

The proportional shrinkage factor of the number of nodes between the last layer and the mth layer is Then the total number of nodes Total _Node of Tornado code in the prior art is:

L不能够任意设定。由于需要保证Tornado码中每层节点的节点个数都是整数，因此需要以及

都是整数，这个条件叫做隐含整数节点数条件。根据该条件，如果给定Tornado码中的m和就可以计算出n需要满足的条件，如当m＝3，

则可以计算出n＝16k，其中k为任意自然数。由此可知，n能够取得的最小值为16，且现有技术中Tornado码的码率r为:

而冗余率1-r为:

Since Total _Node = n+L, therefore, the setting of L is limited,

L cannot be set arbitrarily. Since it is necessary to ensure that the number of nodes in each layer of the Tornado code is an integer, it is necessary as well as

are all integers, this condition is called the implicit integer node number condition. According to this condition, if m and You can calculate the conditions that n needs to meet, such as when m=3,

Then n=16k can be calculated, where k is any natural number. It can be seen that the minimum value that n can obtain is 16, and the code rate r of the Tornado code in the prior art is:

And the redundancy rate 1-r is:

可以任意设置，在给定数据节点个数n的条件下，L可以灵活设定。The improved Tornado code of the present invention does not need the condition of the above-mentioned implicit integer node number because there is no intermediate check node layer in the improved Tornado code, and the check node number L of the check node layer of the improved Tornado code of the present invention for: The proportional shrinkage factor of the data node layer and the node number of the check node layer of the improved Tornado code of the present invention

It can be set arbitrarily. Under the condition of a given number of data nodes n, L can be set flexibly.

本发明改进的Tornado码的冗余率1-r为:

The code rate r of the improved Tornado code of the present invention is:

The redundancy rate 1-r of the improved Tornado sign indicating number of the present invention is:

而码率r＝2/3＝66.7％。The improved Tornado code of the present invention can be represented as TN (n+L, n), such as TN (30, 20), indicating that the number of data nodes in the data node layer is n=20, and the number of check nodes in the check node layer is L=10 . At this point, the improved Tornado code of the present invention

And the code rate r=2/3=66.7%.

In summary, on the basis of synthesizing the above six enhancement technologies, the present invention modularizes the entire H.264/ERRTP transmission architecture, and combines each other on a protocol stack. Enhanced to reflect better reliability and quality of service.

Those skilled in the art can understand that the descriptions of the above seven embodiments involve various specific implementation details and parameter selections, which can be determined according to specific applications and do not affect the essence and scope of the present invention.

Although the present invention has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the present invention. The spirit and scope of the invention.

Claims (44)

1. a multimedia communication method is characterized in that, its communication process comprises following steps,

The A transmitting terminal is protected multi-medium data according to the error elasticity protection strategy based on real time transport protocol, and send it to receiving terminal, the relevant information of described error elasticity protection strategy is carried in the error elasticity real time transport protocol bag of the described multi-medium data of encapsulation, wherein, described error elasticity real time transport protocol bag is a kind of packet that includes multi-medium data simultaneously and the error elasticity that this multi-medium data is protected is protected the relevant information of strategy;

The described receiving terminal of B receives described multi-medium data, occur to transmit under the wrong situation, and the error elasticity protection strategy that carries by the error elasticity real time transport protocol bag of the described multi-medium data of described encapsulation recovers or part is recovered described multi-medium data.

2. multimedia communication method according to claim 1 is characterized in that its communication process also comprises following steps,

The described receiving terminal statistics of C communication quality generates service quality report, and it is sent back to described transmitting terminal;

In the described steps A, described transmitting terminal is adjusted described error elasticity protection strategy according to described service quality report.

3. multimedia communication method according to claim 2 is characterized in that its communication process also comprises following steps,

The described receiving terminal statistics of D transmits error message, and implementation mistake is covered strategy;

Among the described step C, described receiving terminal also feeds back to described transmitting terminal with described transmission error message;

The described transmitting terminal of E is implemented the error code diffusion according to described transmission error message and is eliminated strategy.

4. multimedia communication method according to claim 3 is characterized in that, described steps A comprises following substep:

The A1 transmitting terminal selects the error elasticity encoding scheme that multi-medium data is carried out forward error correction coding;

Multi-medium data after the described transmitting terminal of A2 is encoded with the encapsulation of error elasticity real time transport protocol, and in described error elasticity real time transport protocol header packet information, carry described error elasticity encoding scheme relevant information, send to receiving terminal then;

Described step B comprises following substep:

The described receiving terminal of B1 goes the error elasticity real time transport protocol bag of receiving to encapsulation, and extracts described forward error correction coding scheme relevant information from described error elasticity real time transport protocol header packet information;

If the losing of error elasticity real time transport protocol bag of back end correspondence taken place in B2 in transport process, so described receiving terminal is according to described error elasticity encoding scheme relevant information, select described decoding FEC scheme to carry out decoding FEC, recover or partly recover the described multi-medium data of losing; If losing of back end do not taken place, then do not need to carry out decoding FEC.

5. multimedia communication method according to claim 4 is characterized in that, the multi-medium data behind the described forward error correction coding is divided into back end and check-node two classes.

6. multimedia communication method according to claim 5 is characterized in that, in described steps A 1, described transmitting terminal is selected described forward error correction coding scheme according to one of following factor at least:

Current network transmission status and service of multimedia data credit rating to be sent, service of multimedia data credit rating wherein to be sent depends on the relative importance of different pieces of information.

7. multimedia communication method according to claim 6 is characterized in that, comprises in the described error elasticity real time transport protocol header packet information:

Error elasticity real time transport protocol identification field is used for indication to be different from real time transport protocol;

The forward error correction coding type field is used to the forward error correction code type of indicating described forward error correction coding scheme to adopt;

The forward error correction coding sub-type field is used to indicate the relative parameters setting of described forward error correction coding scheme;

The data packet length field is used to indicate the length of described forward error correction coding scheme at the node that described multi-medium data is carried out obtain behind the forward error correction coding;

The number of data packets field is used to indicate the number of the described node that this error elasticity real time transport protocol bag carried.

8. multimedia communication method according to claim 7, it is characterized in that, in described steps A 1, described transmitting terminal with at least one H.264 network abstraction layer unit be divided at least one isometric back end, then it is carried out forward error correction coding, obtain at least one check-node;

In described steps A 2, described transmitting terminal sends described back end and described check-node packet encapsulation at least one described error elasticity real time transport protocol bag;

In described step B1, described receiving terminal goes encapsulation to obtain described back end and described check-node after receiving described error elasticity real time transport protocol bag;

In described step B2, if the back end in the transport process has taken place to be lost, then described receiving terminal carries out recovering based on the recovery or the part of decoding FEC to the described back end of losing according to described check-node, and division obtains described H.264 network abstraction layer unit.

9. multimedia communication method according to claim 8 is characterized in that, before beginning transmission, also comprises step,

Described transmitting terminal and described receiving terminal are consulted to determine: for various described forward error correction code types, the corresponding relation of the relative parameters setting of this kind forward error correction that the value of described forward error correction sub-type field is indicated with it.

10. multimedia communication method according to claim 9, it is characterized in that, described transmitting terminal and described receiving terminal are all set up mapping table according to the corresponding relation of described forward error correction coding sub-type field indication, are used for inquiring about pairing forward error correction coding or decoding FEC processing module according to described forward error correction coding type field and described forward error correction coding sub-type field;

In described steps A 1, described transmitting terminal calls corresponding forward error correction coding processing module and carries out forward error correction coding;

In described step B2, described receiving terminal calls corresponding decoding FEC processing module and carries out decoding FEC.

11. multimedia communication method according to claim 10, it is characterized in that, in described steps A 1, described transmitting terminal is according to the combination of network abstract layer reference identification field in the header of described H.264 network abstraction layer unit and any one or both in the network abstraction layer unit type field, and any other can predefined rule assess the relative importance of corresponding data, thereby determine described service quality rating, and then select described forward error correction coding scheme, determine described forward error correction coding type field and forward error correction coding sub-type field.

12. multimedia communication method according to claim 10, it is characterized in that, in described steps A, described transmitting terminal is estimated described network transmission status according to the transmission report of described receiving terminal feedback, and then select described forward error correction coding scheme, determine described forward error correction coding type field and forward error correction coding sub-type field.

13. multimedia communication method according to claim 12 is characterized in that, the version information field value in the described error elasticity real time transport protocol header packet information is binary value " 11 " or decimal value " 3 ", to be different from real time transport protocol;

Described forward error correction coding type field is positioned at after the tabulation of contribution source identifier, accounts for 4 bits;

Described forward error correction coding sub-type field is positioned at after the described forward error correction coding type field, accounts for 9 bits;

Described data packet length field is positioned at after the described forward error correction coding sub-type field, accounts for 11 bits;

Described number of data packets field is positioned at after the described data packet length field, accounts for 8 bits.

14. multimedia communication method according to claim 8, it is characterized in that, in the described steps A, divide together again, encode after at least one network abstraction layer unit that described transmitting terminal is identical with header is removed its header and be encapsulated into described error elasticity real time transport protocol bag, and with identical header that this network abstraction layer unit had comprehensively in the header of this error elasticity real time transport protocol bag;

Among the described step B, described receiving terminal obtains the header that is carried from the header of the described error elasticity real time transport protocol bag that receives, and add the head of the network abstraction layer unit of having peeled off header that extracts from described error elasticity real time transport protocol bag to, obtain whole network level of abstraction unit; Transmit mistake if exist, then have the decoding FEC of carrying out to recover or part restore data node, and then therefrom extract network abstraction layer unit according to presetting strategy.

15. multimedia communication method according to claim 14, it is characterized in that, in described error elasticity real time transport protocol header, network abstract layer reference identification field in the described network abstraction layer unit header and type field are filled in the payload type field of described error elasticity real time transport protocol header packet information, and this payload type field is positioned at back 7 bits of the 2nd byte of described error elasticity real time transport protocol header packet information.

16. multimedia communication method according to claim 15, it is characterized in that, described error elasticity real time transport protocol identification field is the version information field of described error elasticity real time transport protocol header packet information, and this version information field is positioned at preceding 2 bits of the 1st byte of described error elasticity real time transport protocol header packet information.

17. multimedia communication method according to claim 16, it is characterized in that, in described error elasticity real time transport protocol encapsulation format, the bit field of forbidding in the described network abstraction layer unit header is filled in the tag field of described error elasticity real time transport protocol header packet information, and this tag field is positioned at preceding 1 bit of the 2nd byte of described error elasticity real time transport protocol header packet information;

And in described step B, receiving terminal judges according to the tag field of described error elasticity real time transport protocol bag whether its network abstraction layer unit of carrying makes mistakes.

18. multimedia communication method according to claim 15, it is characterized in that, in described steps A, B, described error elasticity real time transport protocol is designated the tag field value of described error elasticity real time transport protocol header packet information, and this tag field is positioned at preceding 1 bit of the 2nd byte of described error elasticity real time transport protocol header packet information.

19. multimedia communication method according to claim 18 is characterized in that, described steps A comprises following substep:

Described transmitting terminal at first judges to forbid whether bit field is effective in the header of at least one described network abstraction layer unit, in view of the above it is divided into proper network level of abstraction unit and the network abstraction layer unit of makeing mistakes;

By described error elasticity real time transport protocol encapsulation format described proper network level of abstraction unit package is become described error elasticity real time transport protocol bag then, and establish described error elasticity real time transport protocol sign;

By described real time transport protocol encapsulation format the described network abstraction layer unit of makeing mistakes is packaged into described real time transport protocol bag;

Described step B comprises following substep:

Described receiving terminal judges that at first whether the header of the bag that receives establishes described error elasticity real time transport protocol sign, is divided into described error elasticity real time transport protocol bag and described real time transport protocol bag with it;

Handle described error elasticity real time transport protocol bag according to described error elasticity real time transport protocol encapsulation format then, seal the described real time transport protocol bag of dress format analysis processing according to described real time transport protocol.

20. multimedia communication method according to claim 8 is characterized in that, the following substep of described step C,

The described receiving terminal statistics of C1 generates described service quality report;

The described receiving terminal of C2 carries described service quality report with supplemental enhancement information, issues described transmitting terminal.

21. multimedia communication method according to claim 20 is characterized in that,

The encapsulation format of described service quality report in described supplemental enhancement information is as follows:

The 1st byte is load type field, and being used to indicate load is the corresponding with service quality report;

2nd, 3 bytes are payload length field, are used to indicate corresponding with service quality report length;

After the 4th byte reaches is load, is used to fill the corresponding with service quality report.

22. multimedia communication method according to claim 21 is characterized in that, described service quality report is divided into transmit leg report and recipient's report, is distinguished by described load type field indication;

When described service quality report was filled in the load of described supplemental enhancement information, the load of described supplemental enhancement information comprised:

The version information field accounts for 2 bits;

Fill field, account for 1 bit, be used to indicate whether to have the filling content;

Receive the number of reports field, account for 5 bits, be used for indicating this service quality report to report and receive the report blocks number;

Transmit leg synchronous source identifier field accounts for 32 bits, is used to identify the transmit leg of this service quality report;

When described service quality report is transmit leg when report, also comprise the caller information piece, be used to describe the relevant information of the transmit leg of this service quality report;

Comprise at least one described reception report blocks, be used to describe from the multimedia statistical information of homology not;

Comprise specific aspect expansion, be used for the reservation function expansion of specific aspect.

23. multimedia communication method according to claim 21 is characterized in that, the described supplemental enhancement information that is used to carry described service quality report is further carried by abstract network layer unit;

The reliability requirement that described communication terminal transmits according to described service quality report is provided with the network abstract layer reference identification of this abstract network layer unit.

24. multimedia communication method according to claim 21 is characterized in that, described communication terminal is dynamically adjusted the statistics generation of described service quality report and the cycle that sends according to current network state and higher layer applications demand.

25. multimedia communication method according to claim 21 is characterized in that, when described communication terminal mixes the service quality report of at least a Media Stream of carrying with described additional enhancing message,

Comprise the corresponding described reception report blocks of at least a Media Stream that is carried in this service quality report.

26. multimedia communication method according to claim 20, it is characterized in that, among the described step C, described receiving terminal is according to the network abstraction layer unit sequence number of the described video stream data that receives, the described network abstraction layer unit number that statistics is lost, generate described service quality report, send back to described transmitting terminal;

In the described steps A, described transmitting terminal calculates described accumulative total packet loss according to described network abstraction layer unit sequence number of losing, and adjusts described error elasticity protection strategy in view of the above.

27. multimedia communication method according to claim 26 is characterized in that, described receiving terminal is according to the service quality report that receives, analytical calculation network condition parameter; Described parameter comprises instant bandwidth, time-delay and shake end to end.

28. multimedia communication method according to claim 27; it is characterized in that; described transmitting terminal is provided with the error elasticity protection strategy series of different brackets, selects to use corresponding described error elasticity protection strategy according to described service quality report in described steps A.

29. multimedia communication method according to claim 28, it is characterized in that among the described step C, described receiving terminal is according to the network abstraction layer unit sequence number of the described video stream data that receives, add up the locating information that obtains losing video stream data, and it is sent back to described transmitting terminal;

In the described steps A, described transmitting terminal resends the described video stream data of losing and gives described receiving terminal according to described locating information of losing video stream data.

30. multimedia communication method according to claim 8, it is characterized in that, in described step e, described transmitting terminal obtains described locating information of losing band according to described transmission error message, carry out segmentation intraframe coding one by one by this being lost band, eliminate strategy to realize described error code diffusion.

31. multimedia communication method according to claim 30 is characterized in that, described segmentation intraframe coding one by one comprises following steps,

E1 is cut apart one group of continuous macro block from described losing the band, form new band, and remaining described macro block still belongs to the described band of losing, and enters step e 2;

E2 carries out intraframe coding to described new band, sends when next frame, and this new band is done conventional coding after this, enters step e 3;

When E3 encodes at next frame, judge whether the described band of losing also comprises untreated macro block, if then return step e 1, otherwise finish intraframe coding.

32. multimedia communication method according to claim 31, it is characterized in that, in described step e 1, the size of each described one group of continuous macro block of separating satisfies: after this was organized continuous macro block and carries out intraframe coding, the data transfer rate of this frame was in data transfer rate control range H.264.

33. multimedia communication method according to claim 8 is characterized in that, described step D comprises following substep,

The described receiving terminal of D1 detects and transmits mistake, and statistics transmits error message;

The described receiving terminal of D2 carries out video information and weighs synchronously after taking place to transmit mistake;

The described receiving terminal of D3 is implemented described error concealment strategy according to described transmission error message.

34. multimedia communication method according to claim 33 is characterized in that, among the described step D1, described receiving terminal detects and adds up the transmission error message according to the discontinuous situation of network abstraction layer unit sequence number.

35. multimedia communication method according to claim 34, it is characterized in that, in described step D1, the locating information that described receiving terminal is lost band according to the interruption situation acquisition of described network abstraction layer unit sequence number, this locating information comprises described frame number and described position of losing band at this frame of losing the band place.

36. multimedia communication method according to claim 35 is characterized in that, described error concealment strategy comprises step: described receiving terminal substitutes this and loses band with described respective strap of losing the former frame of band place frame.

37. any described multimedia communication method in 36 is characterized in that described error elasticity encoding scheme comprises improved " Tornado " correcting and eleting codes according to Claim 8;

Described improved " Tornado " correcting and eleting codes only generates the described check-node of one deck for one group of described back end.

38., it is characterized in that the transmission mistake among the described step B comprises data-bag lost or random bit mistake according to each described multimedia communication method in the claim 2 to 36.

39. a multimedia communication terminal comprises the basic function module that is used to realize multimedia communication, wherein comprises the coding/decoding module that is used to realize the multi-media decoding and encoding function, it is characterized in that, also comprises with lower module:

Error elasticity transmits the control protocol module in real time; be used for and undertaken by the multi-medium data behind the described coding/decoding module coding transmitting at network side again after the error elasticity protection; described multi-medium data from network side is carried out passing to described coding/decoding module again after the error correction decodes; the relevant information of described error elasticity protection strategy is carried in the error elasticity real time transport protocol bag of the described multi-medium data of encapsulation; wherein, described error elasticity real time transport protocol bag is a kind of packet that includes multi-medium data simultaneously and the error elasticity that this multi-medium data is protected is protected the relevant information of strategy.

40. according to the described multimedia communication terminal of claim 39, it is characterized in that, also comprise with lower module:

Guard method and policy conferring module are used to be responsible for carry out error elasticity protection policy conferring between communicating pair, determine the protection strategy set, transmit the control protocol module in real time for described error elasticity and select;

Forward error correction block; be used to realize at least a forward error correction guard method; safeguard the relevant parameter of described forward error correction guard method; wherein said guard method and the described forward error correction block of policy conferring module controls are to realize unequal loss protection and adaptive hierarchical defencive function, and described error elasticity transmits the control protocol module in real time and realizes error elasticity protection and error correction by calling this forward error correction block.

41. according to the described multimedia communication terminal of claim 40, it is characterized in that, also comprise

The error concealment module is used to realize the error concealment function;

Described coding/decoding module is used to realize the H.264 encoding and decoding of encoding and decoding standard, also is used for the error code diffusion and eliminates function;

Also comprise network condition analytical calculation module, be used for the analytical calculation network condition, and provide information to described error concealment module and described coding/decoding module.

42. according to the described multimedia communication terminal of claim 41, it is characterized in that, also comprise

Replenish enhancing extension of message processing module, be used to realize service quality report and network condition function of reporting, and report is sent to described network condition analytical calculation module.

43., it is characterized in that its transport layer is used to realize supporting the multimedia transmitting function of error elasticity based on described error elasticity real time transport protocol/transmit control protocol in real time according to the described multimedia communication terminal of claim 42;

Its application protocol layer comprises protection mechanism and policy conferring sublayer, is used to realize cascade protection and unequal loss protection function;

Its H.264 the video coding layer comprise and replenish to strengthen extension of message report layer, be used to realize the function of reporting that strengthens extension of message based on replenishing;

H.264, it comprises the forward error correction coding layer in network abstract layer, is used to realize the forward error correction coding function.

44., it is characterized in that the described basic function module that is used to realize multimedia communication comprises one of following or its combination in any according to each described multimedia communication terminal in the claim 39 to 43:

Main control module is used for being responsible for the control of whole terminal;

Subscriber Interface Module SIM is used for the demonstration of the mutual and information of responsible user's input and output;

Network communication module is used for being responsible for and network communicates, and provides lower floor to transmit passage;

Input and output and bottom layer driving module are used for being responsible for driving for hardware device;

Business module is used to realize high-level business;

The communication course control module is used to control communication process;

The application protocol module is used to realize the application protocol function;

Transmit the control protocol module in real time, be used to realize transmit in real time the control protocol function;

The network abstract layer module is used to realize the network abstract layer function;

The audio coding decoding module is used to realize the audio coding decoding function.

Images