WO2025040105A1 - Video bitstream processing method and apparatus, and device and readable storage medium - Google Patents

Abstract

The present application relates to the technical field of video coding. Disclosed are a video bitstream processing method and apparatus, and a device and a readable storage medium, which aim to realize the support of an audio video coding standard for an RTP. The method comprises: performing RTP package on bitstream metadata or a bitstream metadata segment to obtain an RTP data packet; and sending the RTP data packet to a decoding end. The RTP data packet comprises an RTP header, and an RTP load of video bitstream data, wherein the RTP load of the video bitstream data comprises a universal load header, a decoding order indication identifier, load data headers corresponding to different types of RTP loads, and load data; or the RTP load of the video bitstream data comprises a universal load header, load data headers corresponding to different types of RTP loads, and load data.

Description

A video code stream processing method, device, equipment and readable storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Chinese Patent Application No. 202311071302.3 filed in China on August 23, 2023, the entire contents of which are incorporated herein by reference.

Technical Field

The present application relates to the field of video coding technology, and in particular to a video code stream processing method, device, equipment and readable storage medium.

Background Art

The Real-time Transport Protocol (RTP) is a transport protocol designed by the Internet Engineering Task Force (IETF) for real-time transmission of video and audio. The RTP protocol is located on top of the User Datagram Protocol (UDP) protocol, and usually uses the low-level UDP protocol to multicast or unicast real-time video and audio data, thereby achieving multi-point or single-point video and audio data transmission.

However, currently, for example, the Audio Video coding Standard (AVS) does not support the RTP protocol.

Summary of the invention

The embodiments of the present application provide a video code stream processing method, apparatus, device and readable storage medium to implement the support of audio and video coding standards for RTP.

In a first aspect, an embodiment of the present application provides a video code stream processing method, which is applied to an encoding end, including:

Perform RTP encapsulation on the meta-stream data or the meta-stream data fragments to obtain an RTP data packet;

Sending the RTP data packet to a decoding end;

Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data;

The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data;

or

The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data.

In a second aspect, an embodiment of the present application further provides a video code stream processing method, which is applied to a decoding end, comprising:

Receiving an RTP data packet, wherein the RTP data packet is obtained by performing RTP encapsulation on the meta-stream data or the meta-stream data fragments;

Decoding the RTP data packet;

Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data;

The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data;

or

The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data.

In a third aspect, an embodiment of the present application further provides a video code stream processing device, applied to an encoding end, comprising:

The first processing module is used to perform RTP encapsulation on the meta-stream data or the meta-stream data fragments to obtain an RTP data packet;

A first sending module, used for sending the RTP data packet to the decoding end;

Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data;

The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data;

or

The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data.

In a fourth aspect, an embodiment of the present application further provides a video code stream processing device, applied to a decoding end, comprising:

A first receiving module is used to receive an RTP data packet, wherein the RTP data packet is obtained by performing RTP encapsulation on the meta-stream data or the meta-stream data fragments;

A first processing module, used for decoding the RTP data packet;

Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data;

The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data;

or

The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data.

In a fifth aspect, an embodiment of the present application further provides a communication device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps in the video stream processing method described above when executing the program.

In a sixth aspect, an embodiment of the present application further provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the video stream processing method as described above are implemented.

In the embodiment of the present application, the meta-stream data or the meta-stream data fragments are RTP encapsulated to obtain an RTP data packet, and the RTP data packet is sent, thereby realizing the support of the audio and video coding standard for RTP.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flowchart of a video code stream processing method provided in an embodiment of the present application;

FIG2 is a schematic diagram of an RTP header in an embodiment of the present application;

FIG3 is a schematic diagram of an RTP data packet formed in an embodiment of the present application;

FIG4( a ) is a schematic diagram of a meta-code stream in an embodiment of the present application;

FIG4( b ) is a second schematic diagram of a meta-code stream in an embodiment of the present application;

FIG5 is one of schematic diagrams of a single load in an embodiment of the present application;

FIG6 is a second schematic diagram of a single load in an embodiment of the present application;

FIG7 is a schematic diagram of a shard load in an embodiment of the present application;

FIG8 is a second schematic diagram of a shard load in an embodiment of the present application;

FIG9 is a third schematic diagram of a shard load in an embodiment of the present application;

FIG10 is one of the schematic diagrams of the polymerized load in the embodiment of the present application;

FIG11 is a second schematic diagram of the polymer load in an embodiment of the present application;

FIG12 is a third schematic diagram of the polymerized load in an embodiment of the present application;

FIG13 is a fourth schematic diagram of the polymerized load in an embodiment of the present application;

FIG14 is a fifth schematic diagram of the polymer load in the embodiment of the present application;

FIG15 is a sixth schematic diagram of the polymerized load in the embodiment of the present application;

FIG16 is a seventh schematic diagram of the polymer load in the embodiment of the present application;

FIG17 is a schematic diagram of the eighth embodiment of the polymerized load in the present application;

FIG18 is a second flowchart of the video code stream processing method provided in an embodiment of the present application;

FIG19 is a structural diagram of a video code stream processing device provided in an embodiment of the present application;

FIG. 20 is a second structural diagram of the video code stream processing device provided in an embodiment of the present application.

DETAILED DESCRIPTION

In the embodiments of the present application, the term "and/or" describes the association relationship of the associated objects, indicating that there may be three relationships. For example, A and/or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects before and after are in an "or" relationship.

In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

Referring to FIG. 1 , FIG. 1 is a flowchart of a video code stream processing method provided in an embodiment of the present application, which is applied to an encoding end, as shown in FIG. 1 , and includes the following steps:

Step 101: perform RTP encapsulation on the meta-stream data or the meta-stream data fragments to obtain an RTP data packet.

In an embodiment of the present application, the meta-stream data or meta-stream data fragments may be meta-stream data or meta-stream data fragments supported by multiple standards, for example, the meta-stream data or meta-stream data fragments of AVS3. Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bitstream data, and the RTP payload of the video bitstream data includes: a general payload header, a decoding order indication mark, payload data headers corresponding to different types of RTP payloads, and payload data. Alternatively, the RTP payload of the video bitstream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data, that is, in this case, the RTP payload of the video bitstream data does not include the decoding order indication mark.

In the embodiment of the present application, a decoding order indicator sprop-decoder-order is added: this parameter is used to indicate the code stream characteristics of the sender, indicating the relationship between the transmission order and the decoding order. When its value is 0, it indicates that the transmission order is the same as the decoding order; when its value is greater than 0, it indicates that the transmission order is different from the decoding order. If this parameter is not transmitted (that is, the parameter is not carried in the formed RTP packet), the default value of this parameter is 0.

Here, taking the meta-stream data or meta-stream data fragment as an example, the meta-stream data or meta-stream data fragment of AVS3 is RTP encapsulated, which can be understood as the process of encapsulating the meta-stream data or meta-stream data fragment of AVS3 according to RTP, so that the RTP data packet formed in the above manner includes: RTP header and RTP payload of AVS3 video bitstream data. Here, according to the code stream characteristics of AVS3, a universal payload header is set, and different payload structures are designed for each different type of RTP payload, so as to achieve greater flexibility.

Optionally, in an embodiment of the present application, if the size of the meta-stream data transmitted at the IP layer is larger than the MTU, the meta-stream data transmitted at the IP layer is segmented to obtain meta-stream data fragments, thereby facilitating transmission.

In the embodiment of the present application, the RTP header is set on the basis of the existing protocol. As shown in Figure 2, it is a schematic diagram of the RTP header. As shown in Figure 3, it is a schematic diagram of the RTP data formed in the embodiment of the present application, wherein Figure 3 is described by taking the AVS3 load as an example.

In this embodiment of the present application, the RTP header includes one or more of the following:

The marker field (Marker bit, M) is used to indicate the boundary of the video frame. Specifically, this field is 1 bit, M specifies the boundary of the frame, the marker bit of the last RTP data packet of the video frame is set to 1, and the marker bits of other RTP data packets are set to 0.

The payload field (Payload Type, PT) is used to identify the RTP session and indicate the payload format. Specifically, this field is 7 bits. According to the definition, the upper-layer application should dynamically allocate a PT value in each RTP session (such as SDP) and maintain the mapping relationship between the PT value and the payload format. The value range of the PT value should comply with the provisions of the protocol.

The timestamp field Timestamp is used to indicate the sampling time of the RTP data packet; specifically, it means that the RTP timestamp is set to the sampling timestamp of the encoded content, and the clock used is 90000Hz, that is, the time unit of the RTP timestamp is 1/90000s. Like the sequence number, the initial value of the timestamp should also be random, and the timestamps of different RTP data packets of the same frame image should be consistent. For the sequence header, user extension data, and image header payloads that do not have time attributes, their timestamps should be consistent with the timestamps of the image data that follows them.

The sequence number field is used to indicate the transmission order of the RTP data packets. The sequence number is incremented by 1 for each RTP data packet sent. The receiver can use it to detect data packet loss and restore the data packet sequence. The initial value of the sequence number should be random.

The meanings of other fields may refer to the relevant provisions of the prior art, for example, V represents the version, P represents the fill identifier, X represents the extension, the SSRC segment represents the synchronization source, and the CSRC segment represents the action source.

In the embodiment of the present application, the RTP payload of the video bitstream data includes a single payload, a fragmented payload, or an aggregated payload. A single payload means that the payload data includes one metastream data, a fragmented payload means that the payload data includes one metastream data segmentation, and an aggregated payload means that the payload data includes multiple metastream data. In other words, the RTP payload of the video bitstream data may include one or more metastream data, or may be a metastream data segment obtained by further segmenting the metastream data.

Optionally, in an embodiment of the present application, in order to accurately distinguish meta-stream data of different levels, the encoder may determine indication information, wherein the indication information is used to indicate a formation mode of the meta-stream data or the meta-stream data fragment to which the meta-stream data belongs. The formation mode includes the aforementioned level 1 and level 2 modes. The indication information may be indicated to the decoder in an explicit manner or in an implicit manner.

The meta-codestream data or the meta-codestream data to which the meta-codestream data fragment belongs is a bit stream segment starting with a start code, including a sequence header, user data after the sequence header, extended data after the sequence header, an intra-frame predicted image, and an inter-frame predicted image; or, the meta-codestream data or the meta-codestream data to which the meta-codestream data fragment belongs is data between every two adjacent start code prefixes in a video bitstream and contains a start code prefix segment, including a sequence header, user data after the sequence header, extended data after the sequence header, an I-frame image header, an RL-frame image header, an inter-frame image header, user data after the image header, extended data after the image header, and frame slice data.

As shown in FIG4(a), the metastream is a bitstream segment starting with a start code, such as a sequence header (or a video sequence header), user data (after the sequence header), extended data (after the sequence header), intra-frame prediction images, inter-frame prediction images, etc. (i.e., layer 1). Alternatively, as shown in FIG4(b), the metastream data or the metastream data to which the metastream data segment belongs is the data between every two adjacent start code prefixes in the video bitstream and includes the start code prefix segment (i.e., layer 2). The metastream data slice is a segment of an integer number of consecutive bytes of the metastream data.

For example, for the RTP encapsulation type (single, fragmented, aggregated) defined later, the service party (encoder) can select a fixed level of meta-stream according to the application scenario, and describe it through the optional field video_bitstream_unit during Session Description Protocol (SDP) communication. This parameter is used to describe the formation method of the meta-stream data selected by the sender. If this parameter is explicitly sent to the decoder, when the parameter value is 1, it means that the meta-stream data defined in level 1 is selected; when the parameter value is 2, it means that the meta-stream data defined in level 2 is selected. If this parameter is not transmitted, the default value is 1 (that is, implicit mode).

When encapsulating, according to the payload type of the RTP data packet, encapsulation can be performed according to the following principles.

1) Meta-stream data (or bitstream data) of different types cannot be aggregated in the same RTP data packet. For example, meta-stream data of a dependent main bitstream and a knowledge bitstream cannot be aggregated in the same RTP data packet.

2) When the size of a meta-stream data exceeds the Maximum Transmission Unit (MTU) when transmitted at the IP layer, the meta-stream data is fragmented and the fragmented meta-stream data slices are transmitted in the fragmentation order using the fragmented RTP payload format.

3) The aggregated RTP payload does not contain slices of metastream data.

4) RTP payload data is not allowed to be nested, that is, the RTP payload cannot contain a complete RTP data packet.

5) For meta-codestream data defined in the layer 2 manner, some meta-codestream data, such as image header data, should be encapsulated in an aggregate packet with the subsequent image slice data as much as possible, provided that the size of the aggregate packet does not exceed the MTU.

6) Aggregate and encapsulate small-volume metastreams into an aggregate packet to prevent unnecessary packaging overhead. For example, sequence headers, video extensions, user extensions, and other data can be integrated and encapsulated into an aggregate packet as long as the size of the entire aggregate packet at the IP layer is smaller than the MTU.

The following describes in detail the process of obtaining RTP payloads of different types of video bit stream data.

1. Single payload: A single payload contains only one metastream data, and the value of the parameter payload_package_type in the general payload header is 0.

The payload data header of the single payload includes: a single payload header, used to indicate information related to the meta-codestream data; the payload data includes: one meta-codestream data.

The first fixed byte of the RTP payload of the video bit stream data is the general payload header, which indicates the RTP payload header data of the AVS3 video code stream. The descriptor syntax of the general payload header is shown in Table 1.

Table 1

in:

Payload package type field payload_package_type, this field is 2 bits, used to indicate the RTP payload type, the RTP payload type includes a single payload, a fragmented payload, or an aggregated payload; for example, a value of 0 indicates a single payload, a value of 1 indicates a fragmented payload, and a value of 2 indicates an aggregated payload (the payload contains more than or equal to 2 metastream data);

Temporal_id: This field is a 3-bit, unsigned integer used to indicate the time layer identifier to which the RTP payload belongs. The value range is 0 to 7, and its value should be the same as the value of temporal_id in the standard.

The library dependency field library_dependency is a 2-bit unsigned integer used to indicate the type of basic stream to which the RTP payload belongs; the basic stream type includes any of the following: independent main bit stream, dependent main bit stream, knowledge bit stream; for example, when the value is 0, it means that the payload only contains the independent main bit stream; when the value is 1, it means that the payload only contains the knowledge bit stream; when the value is 2, it means that the payload only contains the dependent main bit stream. When the value is 3, it is reserved.

x, this field is 1 bit, indicating whether there is an extension header. When the value is 0, it means that there is no extension header, otherwise a 1-byte extension header needs to be added. The extension header includes a fragment extension payload header or an aggregate extension payload header, that is, for a single payload, no extension payload header is defined.

The second byte of the RTP payload of the video bit stream data is a single payload header, which is used to describe the specific payload information of a single metastream. Its field structure and meaning are shown in Table 2.

Table 2

in:

The payload data type field payload_data_type is a 4-bit unsigned integer used to indicate the data type of the meta-codestream data; its value and type description are specifically shown in Table 3(a) or Table 3(b). Table 3(a) may correspond to the meta-codestream data divided by level 1, and Table 3(b) may correspond to the meta-codestream data divided by level 2.

The decode_order_size field of the payload is a 2-bit, unsigned integer, which is used to indicate the number of bytes required for the picture identifier (Picture ID, PID) that describes the decoding order of the meta-stream data; for example, if the value of the optional parameter sprop-decoder-order transmitted by SDP is the first value (such as 0) or the RTP payload of the video bitstream data does not include the decoding order indicator (that is, the default value is 0), it means that the decoding order of the meta-stream data is consistent with the transmission order (that is, the decoding order of the RTP data packet is consistent with the transmission order). If the value of the optional parameter sprop-decoder-order transmitted by SDP is the second value (such as a value greater than 0), it means that the decoding order of the meta-stream data is different from the transmission order (that is, the decoding order of the RTP data packet is different from the transmission order). At this time, the RTP payload of the video bitstream data also includes: N bytes of PID, which are used to indicate the decoding order of the meta-stream data. That is, in this case, decode_order_size+1 (the value of N) bytes of the parameter PID (picture ID) are added to indicate the decoding order of the RTP payload. Different bitstreams of the same frame data should have the same PID. The starting value of the PID is a random number, and is a continuous integer that increases by 1. After reaching the maximum value, you can choose to continue to expand the PID byte number DOS value (maximum 4 bytes), or you can start numbering from 0.

Reserved field (R), this field is 2 bits, unsigned integer, reserved.

Table 3(a)

Table 3(b)

Taking AVS3 as an example, according to the above description and the value of sprop-decoder-order, the composition structure of a single load is as follows:

When the value of sprop-decoder-order is 0, the RTP payload of the video bit stream data is shown in FIG5 . The RTP payload of the video bit stream data includes an AVS3 general payload header, an AVS3 single payload header, and a meta stream data (single source stream).

When the value of sprop-decoder-order is greater than 0, the RTP payload of the video bitstream data is as shown in Figure 6 (decode_order_size=0). The RTP payload of the video bitstream data contains an AVS3 general payload header, an AVS3 single payload header, a meta-stream data (single source stream), and decode_order_size+1 byte unsigned number PID, which is used to indicate the decoding order of the meta-stream data in the RTP data packet.

2. Fragmented payload: The fragmented payload (payload_package_type=1) contains one metastream data fragment. When the size of the metastream data at the IP layer after encapsulation is larger than the MTU, the metastream data needs to be segmented before RTP data is encapsulated. The data obtained after segmentation is the metastream data fragment, which can be transmitted as a fragmented RTP payload. The fragmented RTP payload of the metastream data fragment obtained by segmenting and encapsulating the same metastream data has a continuous and uninterrupted RTP sequence number (Sequence number) in the segmentation order and has the same timestamp.

The load data header of the slice load includes: a slice load header; the load data includes: a meta-stream data slice. When the extension byte identifier x in the general load header is 1, there is another byte of slice extension load header after the slice load header. The slice load header and/or the slice extension load header are used to indicate information related to the meta-stream data slice.

The first fixed byte of the RTP payload of the video bit stream data is the general payload header, which indicates the RTP payload header data of the AVS3 video code stream. The descriptor syntax of the general payload header is shown in Table 1.

The second byte of the RTP payload of the video bit stream data is the fragment payload header, and its field structure and meaning are shown in Table 4.

Table 4

in:

Payload data type field payload_data_type, this field is 4 bits, unsigned integer, used to indicate the data type of the meta-stream data to which the meta-stream data fragment belongs; its value and type description are shown in Table 3(a) or Table 3(b);

The fragment start field fragment_start (S) is used to indicate the starting fragment of the meta-stream data; this field is 1 bit, and when the value is 1, it indicates that it is the starting slice of a meta-stream data. Otherwise, the value is 0;

The fragment end field fragment_end is used to indicate the end fragment of the meta-stream data; this field is 1 bit, and when the value is 1, it indicates that it is the end slice of a meta-stream data. Otherwise, the value is 0.

The payload decoding order field decode_order_size is a 2-bit, unsigned integer used to indicate the number of bytes required for the PID that describes the decoding order of the meta-stream data fragments; if the value of sprop-decoder-order is a first value (such as 0) or the RTP payload of the video bitstream data does not include the decoding order indicator (that is, the default value is 0), it indicates that the decoding order of the meta-stream data fragments is consistent with the transmission order. If the value of sprop-decoder-order is a second value (such as a value greater than 0), it indicates that the decoding order of the meta-stream data fragments is different from the transmission order, and the RTP payload of the video bitstream data also includes: N bytes of PID, which are used to indicate the decoding order of the meta-stream data fragments. That is, at this time, decode_order_size+1 (the value of N) bytes of parameter PID are added to indicate the decoding order of the meta-stream data fragments. Different meta-stream data fragments of the same frame data should have the same PID. The starting value of PID is a random number and a continuous integer that increases by 1. After reaching the maximum value, you can choose to continue to expand the PID byte number DOS value (maximum 3 bytes), or you can start numbering from 0.

In one RTP data packet, fragment_start and fragment_end cannot be 1 at the same time.

Among them, the field structure and meaning of the fragment extension payload header are shown in Table 5.

Table 5

in:

pic_header: This field is 1 bit. When the meta-codestream data format of level 1 is selected, it indicates whether the complete image header data is included. The value is 1, indicating that the meta-codestream data fragment packet contains the complete image header data. The value is 0, indicating that the meta-codestream data fragment packet does not contain the complete image header data. When the meta-codestream data format of level 2 is selected, this field is a reserved field with no defined meaning.

full_patch_info: This field is 1 bit. If the value is 1, it indicates that the meta-stream data fragment is the frame data of an independent image. If the value is 0, it indicates that the meta-stream data fragment is not the slice data of an independent image.

R: Reserved field.

Taking AVS3 as an example, according to the above description and the value of sprop-decoder-order, the composition structure of the shard load is as follows:

When the value of sprop-decoder-order is 0, the RTP payload of the video bitstream data is shown in FIG7 (decode_order_size=0), and the RTP payload of the video bitstream data includes an AVS3 general payload header, an AVS3 fragment payload header, and a metastream data fragment (single metastream fragment). In the case shown in FIG7, x=0.

When the value of sprop-decoder-order is greater than 0, the RTP payload of the video bitstream data is as shown in Figures 8 and 9 (decode_order_size=0). In Figure 8, the RTP payload of the video bitstream data contains an AVS3 general payload header, an AVS3 slice payload header, decode_order_size+1 byte unsigned PID, which is used to indicate the decoding order of the meta-stream data slices in the RTP data packet, and one meta-stream data slice. In the case shown in Figure 8, x=0. In Figure 9, the RTP payload of the video bitstream data contains an AVS3 general payload header, an AVS3 slice payload header, a slice extension payload header, and decode_order_size+1 byte unsigned PID, which is used to indicate the decoding order of the meta-stream data slices in the RTP data packet, and one meta-stream data slice. In the case shown in Figure 9, x=1.

3. Aggregate Payload: The aggregate payload (payload_package_type=2 or 3) contains at least 2 metastream data, and each metastream data in the aggregate payload is preceded by a fixed 2-byte unsigned integer size to indicate the size of the aggregate RTP payload. There cannot be metastream slices in the aggregate RTP payload.

The payload data header of the aggregated payload includes: an aggregated payload header and a meta-codestream size field, or includes: an aggregated payload header, an aggregated extended payload header and a meta-codestream size field; or, the payload data header of the aggregated payload includes: an aggregated extended payload header, or, the payload data header of the aggregated payload includes: an aggregated extended payload header and a meta-codestream size field; or, the payload data header of the aggregated payload includes: a meta-codestream size field; or, the payload data header of the aggregated payload includes: an aggregated payload header;

Wherein, the aggregate payload header and/or the aggregate extended payload header is used to indicate information related to the meta-stream data;

The payload data includes: at least two meta-stream data.

For the above-mentioned type of aggregated payload, the first fixed byte of the RTP payload of the video bitstream data is the general payload header, which indicates the RTP payload header data of the video bitstream. The descriptor syntax of the general payload header is shown in Table 1.

Among them, the field structure and meaning of the aggregate payload header of the RTP payload of the video bit stream data are shown in Table 6.

Table 6

in:

Payload data type field payload_data_type, this field is 4 bits, unsigned integer, used to indicate the data type of the metastream data; its value and type description are shown in Table 3(a) or Table 3(b);

The payload decoding order field decode_order_size is a 2-bit, unsigned integer used to indicate the number of bytes required for the PID that describes the decoding order of the meta-stream data; if the value of sprop-decoder-order is the first value (such as 0) or the RTP payload of the video bitstream data does not include the decoding order indicator (that is, the default value is 0), it indicates that the decoding order of the meta-stream data is consistent with the transmission order. If the value of sprop-decoder-order is the second value (such as a value greater than 0), it indicates that the decoding order of the meta-stream data is different from the transmission order, and the RTP payload of the video bitstream data also includes: N bytes of PID, which are used to indicate the decoding order of the meta-stream data. That is, at this time, decode_order_size+1 (the value of N) bytes of parameter PID are added to indicate the decoding order of the meta-stream data fragments. Different meta-stream data fragments of the same frame data should have the same PID. The starting value of PID is a random number and a continuous integer that increases by 1. After reaching the maximum value, you can choose to continue to expand the value of decode_order_size (maximum 4 bytes) of the PID byte number, or you can start numbering from 0.

Reserved field (R), this field is 2 bits, unsigned integer, reserved.

Among them, the field structure and meaning of the aggregate extended payload header of the RTP payload of the video bit stream data are shown in Table 7.

Table 7

in:

A sequence header field sequence_header_index, which is 1 bit and an unsigned integer, is used to indicate whether the aggregated payload includes sequence header data;

Random access index field AU_index, this field is 1 bit, unsigned integer, used to indicate whether the aggregated load includes a random access frame. There are two types of random access frames for AVS3, one is an independent main bit stream (such as library_dependency=0), and the random access frame is the first I frame after the sequence header; the other is a non-independent main bit stream (such as library_dependency=2). For the non-independent main bit stream, the random access frame is the first P frame or B frame after the sequence header.

Reserved field (R), this field is 6 bits, unsigned integer, reserved.

Taking AVS3 as an example, in the embodiment of the present application, according to the value of decode_order_size, the composition structure of the aggregated load is as follows:

When sprop-decoder-order is 0:

When payload_package_type=2 and x=0, the RTP payload of the video bitstream data includes an AVS3 general payload header, at least 2 meta-stream data, a 1-byte aggregate payload header before each meta-stream data, and a 2-byte size indicating the size of the meta-stream, as shown in FIG10 .

When payload_package_type=2 and x=1, the RTP payload of the video bitstream data contains an AVS3 general payload header, an aggregate extended payload header, at least 2 meta-stream data, a 1-byte AVS3 aggregate payload header before each meta-stream data, and a 2-byte size indicating the size of the meta-stream, as shown in FIG11 .

When payload_package_type=3, x=0, it means that all meta-stream data belong to the same type of data (same payload_data_type). The aggregate payload includes a general payload header, an aggregate payload header, and at least 2 meta-stream payloads. The structure is shown in FIG12.

When payload_package_type=3 and x=1, the RTP payload of the video bitstream data includes an AVS3 general payload header, an AVS3 aggregate extended payload header, and at least two metastream data, as shown in FIG13 .

When the value of sprop-decoder-order is greater than 0 (taking the value of 1 as an example):

When payload_package_type=2 and x=0, the RTP payload of the video bitstream data contains an AVS3 general payload header, at least 2 meta-stream data, a 1-byte AVS3 aggregate payload header before each meta-stream data and a 2-byte size to identify the size of the meta-stream, as well as decode_order_size+1-byte unsigned number PID, as shown in Figure 14 (decode_order_size=0).

When payload_package_type=2 and x=1, the RTP payload of the video bitstream data contains an AVS3 general payload header, an aggregate extended payload header, at least 2 meta-stream data, a 1-byte AVS3 aggregate payload header before each meta-stream data and a 2-byte size to indicate the size of the meta-stream, as well as decode_order_size+1-byte unsigned number PID, as shown in Figure 15 (decode_order_size=0).

When payload_package_type=3 and x=0, the RTP payload contains a general payload header, an aggregate payload header, at least two meta-stream data, and a 2-byte size in front of each meta-stream data to identify the size of the meta-stream, and decode_order_size+1-byte unsigned number PID (picture ID) used to indicate the decoding order of the meta-stream data/meta-stream data fragments in the RTP data packet, as shown in Figure 16 (decode_order_size=0).

When payload_package_type=3 and x=1, the RTP payload contains a general payload header, an aggregate extended payload header, at least 2 meta-stream data, and a 2-byte size in front of each meta-stream to identify the size of the meta-stream, and decode_order_size+1-byte unsigned number PID (picture ID) used to indicate the decoding order of the meta-stream data/meta-stream data fragments in the RTP data packet, as shown in Figure 17 (decode_order_size=0).

In the embodiment of the present application, in the aggregated payload, when the extended payload header is not selected (payload_package_type=2), a design of 1 byte can be saved. The byte design of the expandable PID can flexibly adjust the number of PID bytes occupied according to the increase in quantity. When the PID is small, it can be described with 1 byte. When the PID increases, the byte can be expanded or rolled back to 0, which is more flexible and saves bytes.

In the above embodiment, the description is made by taking the aggregation of two elementary code stream data as an example. In actual application, when there are more than two elementary code stream data to be aggregated, the principle is the same as described above.

Step 102: Send the RTP data packet to the decoding end.

In the embodiment of the present application, the meta-stream data or the meta-stream data fragments are RTP encapsulated to obtain an RTP data packet, and the RTP data packet is sent, thereby realizing the support of the audio and video coding standard for RTP.

Referring to FIG. 18 , FIG. 18 is a flowchart of a video code stream processing method provided in an embodiment of the present application, which is applied to a decoding end, and as shown in FIG. 18 , includes the following steps:

Step 1801: Receive an RTP data packet, wherein the RTP data packet is obtained by performing RTP encapsulation on AVS3 meta-stream data or meta-stream data fragments;

Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data;

The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data;

or

The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data.

Step 1802: Decode the RTP data packet.

The unpacking process is to parse the code stream from the RTP code stream and transmit it to the decoder in the decoding order. Here, the RTP data packet is parsed to obtain the RTP header and the RTP payload of the video bit stream data. After that, decoding is performed based on the RTP payload of the video bit stream data.

Optionally, the decoding end may further obtain indication information, wherein the indication information is used to indicate a formation method of the meta-stream data or the meta-stream data to which the meta-stream data fragment belongs;

The meta-stream data or the meta-stream data to which the meta-stream data fragment belongs is a bit stream segment starting with a start code, including a sequence header, user data after the sequence header, extended data after the sequence header, an intra-frame prediction image, and an inter-frame prediction image;

or

The meta-codestream data or the meta-codestream data to which the meta-codestream data fragment belongs is the data between every two adjacent start code prefixes in the video bit stream and contains the start code prefix fragment, including a sequence header, user data after the sequence header, extended data after the sequence header, an I frame image header, an RL frame image header, an inter-frame image header, user data after the image header, extended data after the image header, and frame slice data.

The indication information may be an explicit indication or an implicit indication.

For example, the optional field video_bitstream_unit is obtained during SDP communication. When the parameter value is 1, it indicates that the meta-stream data defined in layer 1 is selected; when the parameter value is 2, it indicates that the meta-stream data defined in layer 2 is selected. If this parameter is not transmitted, the default value is 1 (i.e., implicit mode).

In the above decoding process, after determining the specific RTP payload type according to the general payload header, decoding can be performed in combination with the payload data header and payload data corresponding to different types of RTP payloads. Specifically, the decoding method of different types of RTP payloads is described below.

1. For a single load:

If it is determined according to the general payload header (such as the payload encapsulation type field payload_package_type) that the RTP payload of the video bitstream data is a single payload and the value of the decoding order indicator is the first value, obtaining the meta-stream data in the payload data, obtaining a meta-stream data packet, and decoding the meta-stream data packet;

If it is determined according to the universal load header that the RTP load of the video bit stream data is a single load and the decoding order indication identifier is a second value, obtain the meta-stream data in the load data to obtain a meta-stream data packet; obtain the PID of the RTP load of the video bit stream data, sort the meta-stream data packets according to the PID, and decode the sorted meta-stream data packets.

2. For shard load:

If it is determined according to the general payload header (such as the payload encapsulation type field payload_package_type) that the RTP payload of the video bit stream data is a fragmented payload and the value of the decoding order indicator is the first value, determine the starting fragment and the ending fragment according to the fragment start field fragment_start and the fragment end field fragment_end in the fragment payload header; form meta-stream data using the target code stream fragment data determined according to the starting fragment and the ending fragment, form a meta-stream data packet according to the meta-stream data, and decode the meta-stream data packet;

If it is determined according to the universal load header that the RTP load of the video bitstream data is a fragmented load and the decoding order indication identifier is a second value, determine the starting fragment and the ending fragment according to the fragment start field fragment_start and the fragment end field fragment_end in the fragment load header; use the target code stream fragment data determined according to the starting fragment and the ending fragment to form meta-code stream data, and form a meta-code stream data packet according to the meta-code stream data; obtain the PID of the RTP load of the video bitstream data, sort the meta-code stream data packets according to the PID, and decode the sorted meta-code stream data packets.

In the above process, if the extension byte identifier x in the general payload header is 1, it is also necessary to decode in combination with the slice extension payload header. In the process of decoding in combination with the slice extension payload header, it is mainly to determine whether the meta-code stream data slice includes complete image header data and whether the meta-code stream data slice is independent image slice data in combination with each field of the slice extension payload header, so as to facilitate processing in units of images.

3. For aggregated load:

If it is determined according to the general payload header (such as the payload encapsulation type field payload_package_type) that the RTP payload of the video bit stream data is an aggregate payload, decoding may be performed in any of the following ways:

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the metastream data is processed according to the aggregate payload header and the metastream size field located before each metastream data in the payload data header of the aggregate payload to form a metastream data packet, and the metastream data packet is decoded. In this process, if the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, the metastream data is obtained according to the metastream size field to obtain a metastream data packet; if the value of the decoding order indicator is the second value, the metastream data is obtained according to the metastream size field to obtain a metastream data packet, the PID of the RTP payload of the video bitstream data is obtained, the metastream data packets are sorted according to the PID, and the sorted metastream data packets are obtained.

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the metastream data is processed according to the aggregate extended payload header in the payload data header of the aggregate payload, the aggregate payload header located before each metastream data, and the metastream size field to form a metastream data packet, and the metastream data packet is decoded. In this process, if the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, the metastream data is obtained according to the metastream size field and the extended payload header to obtain a metastream data packet; if the value of the decoding order indicator is the second value, the metastream data is obtained according to the metastream size field and the extended payload header to obtain a metastream data packet, the PID of the RTP payload of the video bitstream data is obtained, the metastream data packets are sorted according to the PID to obtain sorted metastream data packets.

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate extended payload header in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded. In this process, if the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, the meta-stream data is obtained according to the aggregate extended payload header to obtain a meta-stream data packet; if the value of the decoding order indicator is the second value, the meta-stream data is obtained according to the meta-aggregate extended payload header to obtain a meta-stream data packet, the PID of the RTP payload of the video bitstream data is obtained, the meta-stream data packets are sorted according to the PID, and the sorted meta-stream data packets are obtained.

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate extended payload header and the meta-stream size field in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded. In this process, if the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, the meta-stream data is obtained according to the aggregate extended payload header and the meta-stream size field to obtain a meta-stream data packet; if the value of the decoding order indicator is the second value, the meta-stream data is obtained according to the aggregate extended payload header and the meta-stream size field to obtain a meta-stream data packet, the PID of the RTP payload of the video bitstream data is obtained, the meta-stream data packets are sorted according to the PID, and the sorted meta-stream data packets are obtained.

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the meta-stream size field in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded. In this process, if the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, the meta-stream data is obtained according to the meta-stream size field to obtain a meta-stream data packet; if the value of the decoding order indicator is the second value, the meta-stream data is obtained according to the meta-stream size field to obtain a meta-stream data packet, the PID of the RTP payload of the video bitstream data is obtained, the meta-stream data packets are sorted according to the PID, and the sorted meta-stream data packets are obtained.

If it is determined according to the general load header that the RTP load of the video bitstream data is an aggregate load, the meta-stream data is processed according to the aggregate load header in the load data header of the aggregate load to form a meta-stream data packet, and the meta-stream data packet is decoded. In this process, if the value of the decoding order indicator is the first value or the RTP load of the video bitstream data does not include the decoding order indicator, the meta-stream data is obtained according to the aggregate load header to obtain the meta-stream data packet; if the value of the decoding order indicator is the second value, the meta-stream data is obtained according to the aggregate load header to obtain the meta-stream data packet, and the PID of the RTP load of the video bitstream data is obtained. The meta-stream data packets are sorted according to the PID to obtain sorted meta-stream data packets.

Among them, the first value can be 0, the second value can be 1, etc.

In the embodiment of the present application, the meta-stream data or the meta-stream data fragments are RTP encapsulated to obtain an RTP data packet, and the RTP data packet is sent, thereby realizing the support of the audio and video coding standard for RTP.

Referring to FIG. 19 , FIG. 19 is a structural diagram of a video code stream processing device provided in an embodiment of the present application, which is applied to an encoding end. As shown in FIG. 19 , the video code stream processing device includes:

The first processing module 1901 is used to perform RTP encapsulation on the AVS3 meta-stream data or meta-stream data fragments to obtain an RTP data packet; the first sending module 1902 is used to send the RTP data packet to the decoding end. The RTP data packet includes: an RTP header and an RTP payload of video bit stream data;

The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data;

or

The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data.

Optionally, the device may further include:

The segmentation module is used to segment the meta-stream data transmitted at the IP layer to obtain meta-stream data fragments if the size of the meta-stream data transmitted at the IP layer is larger than the MTU.

Optionally, the device may further include:

A determination module, used for determining indication information, wherein the indication information is used for indicating a formation mode of the meta-stream data or the meta-stream data to which the meta-stream data fragment belongs;

The meta-stream data or the meta-stream data to which the meta-stream data fragment belongs is a bit stream segment starting with a start code, including a sequence header, user data after the sequence header, extended data after the sequence header, an intra-frame prediction image, and an inter-frame prediction image;

or

The meta-codestream data or the meta-codestream data to which the meta-codestream data fragment belongs is the data between every two adjacent start code prefixes in the video bit stream and contains the start code prefix fragment, including a sequence header, user data after the sequence header, extended data after the sequence header, an I frame image header, an RL frame image header, an inter-frame image header, user data after the image header, extended data after the image header, and frame slice data.

Optionally, if the RTP payload of the video bitstream data includes a single payload; the payload data header of the single payload includes: a single payload header, used to indicate information related to the meta-codestream data; the payload data of the single payload includes: one meta-codestream data.

Optionally, the single load head comprises:

The payload data type field payload_data_type is used to indicate the data type of the metastream data;

The payload decoding order field decode_order_size is used to indicate the number of bytes required by the PID that describes the decoding order of the meta-codestream data.

Optionally, if the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, it indicates that the decoding order of the meta-code stream data is consistent with the transmission order;

If the value of the decoding order indication identifier is the second value, it means that the decoding order of the meta-codestream data is different from the transmission order, and the RTP payload of the video bitstream data also includes: an N-byte PID, used to indicate the decoding order of the meta-codestream data, where N is equal to the sum of the number of bytes indicated by the payload decoding order field decode_order_size and 1.

Optionally, if the RTP payload of the video bitstream data includes a slice payload; the payload data header included in the slice payload includes: a slice payload header, or includes: a slice payload header and a slice extended payload header, wherein the slice payload header and/or the slice extended payload header are used to indicate information related to a meta-codestream data slice; the payload data includes: a meta-codestream data slice.

Optionally, the slice payload header includes:

The payload data type field payload_data_type is used to indicate the data type of the meta-stream data to which the meta-stream data fragment belongs;

The fragment start field fragment_start is used to indicate the starting fragment of the meta stream data;

The fragment end field fragment_end is used to indicate the end fragment of the meta stream data;

The payload decoding order field decode_order_size is used to indicate the number of bytes required by the PID that describes the decoding order of the meta-codestream data fragments.

Optionally, if the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, it indicates that the decoding order of the meta-stream data fragments is consistent with the transmission order;

If the value of the decoding order indication identifier is the second value, it means that the decoding order of the meta-stream data fragments is different from the transmission order, and the RTP payload of the video bitstream data also includes: an N-byte PID, used to indicate the decoding order of the meta-stream data fragments, where N is equal to the sum of the number of bytes indicated by the payload decoding order field decode_order_size and 1.

Wherein, the fragment extension payload header includes:

The pic_header field is used to indicate whether the meta-stream data fragment includes the complete picture header data or is reserved;

The full_patch_info field is used to indicate whether the meta-stream data fragment is independent image frame data.

Optionally, if the RTP payload of the video bitstream data includes an aggregate payload, the payload data header of the aggregate payload includes: an aggregate payload header and a meta-codestream size field, or the payload data header of the aggregate payload includes: an aggregate payload header, an aggregate extended payload header and a meta-codestream size field, or the payload data header of the aggregate payload includes: an aggregate extended payload header, or the payload data header of the aggregate payload includes: an aggregate extended payload header and a meta-codestream size field; or the payload data header of the aggregate payload includes: a meta-codestream size field; or the payload data header of the aggregate payload includes: an aggregate payload header; wherein the aggregate payload header and/or the aggregate extended payload header are used to indicate information related to meta-codestream data;

Optionally, the polymer load head comprises:

The payload data type field payload_data_type is used to indicate the data type of the metastream data;

The payload decoding order field decode_order_size is used to indicate the number of bytes required by the PID that describes the decoding order of the meta-codestream data.

Optionally, if the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, it indicates that the decoding order of the meta-code stream data is consistent with the transmission order;

If the value of the decoding order indicator is the second value, it means that the decoding order of the meta-stream data is different from the transmission order, and the RTP payload of the video bit stream data further includes: PID, which is used to indicate the decoding order of the meta-stream data.

Optionally, the aggregate extended payload header includes:

A sequence header field sequence_header_index is used to indicate whether the aggregated payload includes sequence header data;

The random access index field AU_index is used to indicate whether the aggregated payload includes a random access frame.

Optionally, the universal load head includes:

The payload encapsulation type field payload_package_type is used to indicate the RTP payload type, which includes a single payload, a fragmented payload, or an aggregated payload;

The temporal_id field is used to indicate the time layer identifier to which the RTP payload belongs.

The library dependency field library_dependency is used to indicate the type of elementary stream to which the RTP payload belongs;

The x field is used to indicate whether an extension header is included, and the extension header includes a fragment extension payload header or an aggregate extension payload header.

Optionally, the data type of the meta-stream data or the data type of the meta-stream data to which the meta-stream data fragment belongs includes:

Sequence header, video extension data, user data after the sequence header, I frame image, RL frame image, inter-frame image; or

The data type of the meta-stream data or the data type of the meta-stream data to which the meta-stream data fragment belongs includes:

Sequence header, video extension data, user data after the sequence header, I frame image header, RL frame image header, P frame image header, B frame image header, video extension data after the image header, user data after the image header, frame slice data.

Optionally, the RTP header includes one or more of the following:

A marker field M is used to indicate the boundary of a video frame;

The payload field PT is used to identify the RTP session and indicate the payload format;

The timestamp field Timestamp is used to indicate the sampling time of the RTP data packet;

The sequence number field, Sequence number, is used to indicate the transmission order of RTP data packets.

The device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

Referring to FIG. 20 , FIG. 20 is a structural diagram of a video code stream processing device provided in an embodiment of the present application, which is applied to a decoding end. As shown in FIG. 20 , the video code stream processing device includes:

The first receiving module 2001 is used to receive an RTP data packet, wherein the RTP data packet is obtained by RTP encapsulating the AVS3 meta-stream data or meta-stream data fragments; the first processing module 2002 is used to decode the RTP data packet.

Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data;

The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data;

or

The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data.

Optionally, the device may further include:

An acquisition module, used for acquiring indication information, wherein the indication information is used for indicating a formation mode of the meta-stream data or the meta-stream data to which the meta-stream data fragment belongs;

The meta-stream data or the meta-stream data to which the meta-stream data fragment belongs is a bit stream segment starting with a start code, including a sequence header, user data after the sequence header, extended data after the sequence header, an intra-frame prediction image, and an inter-frame prediction image; or

The meta-codestream data or the meta-codestream data to which the meta-codestream data fragment belongs is the data between every two adjacent start code prefixes in the video bit stream and contains the start code prefix fragment, including a sequence header, user data after the sequence header, extended data after the sequence header, an I frame image header, an RL frame image header, an inter-frame image header, user data after the image header, extended data after the image header, and frame slice data.

Optionally, the first processing module is further used for:

Parsing the RTP data packet to obtain an RTP packet header and an RTP payload of video bit stream data;

Decoding is performed based on the RTP payload of the video bit stream data.

Optionally, the first processing module is further used for:

If it is determined according to the universal payload header that the RTP payload of the video bitstream data is a single payload and the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, obtaining the meta-codestream data in the payload data, obtaining a meta-codestream data packet, and decoding the meta-codestream data packet;

If it is determined according to the universal load header that the RTP load of the video bit stream data is a single load and the decoding order indication identifier is a second value, obtain the meta-stream data in the load data to obtain a meta-stream data packet; obtain the PID of the RTP load of the video bit stream data, sort the meta-stream data packets according to the PID, and decode the sorted meta-stream data packets.

Optionally, the first processing module is further used for:

If it is determined according to the universal payload header that the RTP payload of the video bitstream data is a fragment payload and the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, determine the starting fragment and the ending fragment according to the fragment start field fragment_start and the fragment end field fragment_end in the fragment payload header; form meta-stream data using the target code stream fragment data determined according to the starting fragment and the ending fragment, form a meta-stream data packet according to the meta-stream data, and decode the meta-stream data packet;

If it is determined according to the universal load header that the RTP load of the video bitstream data is a fragmented load and the decoding order indication identifier is a second value, determine the starting fragment and the ending fragment according to the fragment start field fragment_start and the fragment end field fragment_end in the fragment load header; use the target code stream fragment data determined according to the starting fragment and the ending fragment to form meta-code stream data, and form a meta-code stream data packet according to the meta-code stream data; obtain the PID of the RTP load of the video bitstream data, sort the meta-code stream data packets according to the PID, and decode the sorted meta-code stream data packets.

Optionally, the first processing module is further used for:

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate payload header and the meta-stream size field located before each meta-stream data in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded; or

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate extended payload header in the payload data header of the aggregate payload, the aggregate payload header located before each meta-stream data, and the meta-stream size field to form a meta-stream data packet, and the meta-stream data packet is decoded; or

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate extended payload header in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded; or

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate extended payload header and the meta-stream size field in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded; or

If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the meta-stream size field in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded; or

If it is determined according to the general load header that the RTP load of the video bitstream data is an aggregate load, the meta-stream data is processed according to the aggregate load header in the load data header of the aggregate load to form a meta-stream data packet, and the meta-stream data packet is decoded.

Optionally, the first processing module is further used for:

If the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, obtaining the meta-code stream data according to the meta-code stream size field to obtain a meta-code stream data packet;

If the decoding order indication identifier is a second value, the meta-code stream data is obtained according to the meta-code stream size field to obtain a meta-code stream data packet, the PID of the RTP payload of the video bit stream data is obtained, and the meta-code stream data packets are sorted according to the PID to obtain sorted meta-code stream data packets.

Optionally, the first processing module is further used for:

If the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, obtaining the meta-codestream data according to the meta-codestream size field and the extended payload header to obtain a meta-codestream data packet;

If the decoding order indication identifier is a second value, the meta-codestream data is obtained according to the meta-codestream size field and the extended payload header to obtain a meta-codestream data packet, the PID of the RTP payload of the video bitstream data is obtained, and the meta-codestream data packets are sorted according to the PID to obtain sorted meta-codestream data packets.

Optionally, the first processing module is further used for:

If the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, obtaining the meta-stream data according to the aggregate extended payload header to obtain a meta-stream data packet;

If the decoding order indication identifier is a second value, the meta-code stream data is obtained according to the meta-aggregation extended payload header to obtain a meta-code stream data packet, the PID of the RTP payload of the video bit stream data is obtained, the meta-code stream data packets are sorted according to the PID to obtain sorted meta-code stream data packets.

Optionally, the first processing module is further used for:

If the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, obtaining the meta-stream data according to the aggregate extended payload header and the meta-stream size field to obtain a meta-stream data packet;

If the decoding order indication identifier is a second value, the meta-code stream data is obtained according to the aggregate extended payload header and the meta-code stream size field to obtain a meta-code stream data packet, the PID of the RTP payload of the video bit stream data is obtained, the meta-code stream data packets are sorted according to the PID to obtain sorted meta-code stream data packets.

Optionally, the first processing module is further used for:

If the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, obtaining the meta-code stream data according to the meta-code stream size field to obtain a meta-code stream data packet;

If the decoding order indication identifier is a second value, the meta-code stream data is obtained according to the meta-code stream size field to obtain a meta-code stream data packet, the PID of the RTP payload of the video bit stream data is obtained, and the meta-code stream data packets are sorted according to the PID to obtain sorted meta-code stream data packets.

Optionally, the first processing module is further used for:

If the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, obtaining the meta-stream data according to the aggregate payload header to obtain a meta-stream data packet;

If the decoding order indication identifier is a second value, the meta-stream data is obtained according to the aggregate load header to obtain a meta-stream data packet, the PID of the RTP load of the video bit stream data is obtained, the meta-stream data packets are sorted according to the PID, and the sorted meta-stream data packets are obtained.

The device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

An embodiment of the present application provides a communication device, including: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the video stream processing method as described above.

The embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by a processor, the various processes of the above-mentioned video code stream processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (Magneto Optical, MO)), etc.), optical storage (such as compact disc (Compact Disc, CD), digital video disc (Digital Video Disc, DVD), Blu-ray Disc (Blu-ray Disc, BD), high-definition versatile disc (High-definition Versatile Disc, HVD), etc.), and semiconductor memory (such as ROM, erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable read only memory, EEPROM), non-volatile memory (NAND (Non-volatile Memory Device) FLASH), solid-state drives (Solid State Drives, SSD)), etc.

It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present application and the claims, all of which are within the protection of the present application.

Claims (33)

A video code stream processing method, applied to an encoding end, comprising: Performing RTP encapsulation on the meta-stream data or the meta-stream data fragments to obtain an RTP data packet; Sending the RTP data packet to a decoding end; Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data; The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data; or The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data. The method according to claim 1, wherein the method further comprises: If the size of the meta-stream data transmitted at the IP layer is larger than the maximum transmission unit MTU, the meta-stream data transmitted at the IP layer is segmented to obtain the meta-stream data fragments. The method according to claim 1, wherein the method further comprises: Determine indication information, wherein the indication information is used to indicate a formation method of the meta-stream data or the meta-stream data to which the meta-stream data fragment belongs; The meta-stream data or the meta-stream data to which the meta-stream data fragment belongs is a bit stream segment starting with a start code, including a sequence header, user data after the sequence header, extended data after the sequence header, an intra-frame prediction image, and an inter-frame prediction image; or The meta-codestream data or the meta-codestream data to which the meta-codestream data fragment belongs is the data between every two adjacent start code prefixes in the video bit stream and contains the start code prefix fragment, including a sequence header, user data after the sequence header, extended data after the sequence header, an I frame image header, an RL frame image header, an inter-frame image header, user data after the image header, extended data after the image header, and frame slice data. The method according to claim 1, wherein if the RTP payload of the video bitstream data comprises a single payload; The payload data header of the single payload includes: a single payload header, used to indicate information related to the meta-stream data; The payload data of the single payload includes: a meta-code stream data. The method of claim 4, wherein the single load head comprises: The payload data type field payload_data_type is used to indicate the data type of the metastream data; The payload decoding order field decode_order_size is used to indicate the number of bytes required for the picture identifier PID that describes the decoding order of the meta-codestream data. The method according to claim 5, wherein If the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, it indicates that the decoding order of the metastream data is consistent with the transmission order; If the value of the decoding order indication identifier is the second value, it means that the decoding order of the meta-codestream data is different from the transmission order, and the RTP payload of the video bitstream data also includes: an N-byte PID, used to indicate the decoding order of the meta-codestream data, where N is equal to the sum of the number of bytes indicated by the payload decoding order field decode_order_size and 1. The method according to claim 1, wherein if the RTP payload of the video bit stream data includes a fragment payload; The load data header of the slice load includes: a slice load header, or includes: a slice load header and a slice extended load header, wherein the slice load header and/or the slice extended load header are used to indicate information related to the meta-codestream data slice; The load data of the slice load includes: a meta stream data slice. The method according to claim 7, wherein the slice payload header comprises: The payload data type field payload_data_type is used to indicate the data type of the meta-stream data to which the meta-stream data fragment belongs; The fragment start field fragment_start is used to indicate the starting fragment of the meta stream data; The fragment end field fragment_end is used to indicate the end fragment of the meta stream data; The payload decoding order field decode_order_size is used to indicate the number of bytes required by the PID that describes the decoding order of the meta-codestream data fragments. The method according to claim 8, wherein If the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, it indicates that the decoding order of the metastream data fragments is consistent with the transmission order; If the value of the decoding order indication identifier is the second value, it means that the decoding order of the meta-stream data fragments is different from the transmission order, and the RTP payload of the video bitstream data also includes: an N-byte PID, used to indicate the decoding order of the meta-stream data fragments, where N is equal to the sum of the number of bytes indicated by the payload decoding order field decode_order_size and 1. The method according to claim 8, wherein the slice extension payload header comprises: The pic_header field is used to indicate whether the meta-stream data fragment includes the complete picture header data or is reserved; The full_patch_info field is used to indicate whether the meta-stream data fragment is independent image frame data. The method according to claim 1, wherein if the RTP payload of the video bitstream data includes an aggregation payload; The payload data header of the aggregated payload includes: an aggregated payload header and a meta-codestream size field; or, the payload data header of the aggregated payload includes: an aggregated payload header, an aggregated extended payload header and a meta-codestream size field; or, the payload data header of the aggregated payload includes: an aggregated extended payload header and a meta-codestream size field; or, the payload data header of the aggregated payload includes: a meta-codestream size field; or, the payload data header of the aggregated payload includes: an aggregated payload header; Wherein, the aggregate payload header and/or the aggregate extended payload header is used to indicate information related to the meta-stream data; The payload data of the aggregate payload includes: at least two meta-stream data. The method according to claim 11, wherein the polymer loading head comprises: The payload data type field payload_data_type is used to indicate the data type of the metastream data; The payload decoding order field decode_order_size is used to indicate the number of bytes required by the PID that describes the decoding order of the meta-codestream data. The method according to claim 12, wherein If the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, it indicates that the decoding order of the metastream data is consistent with the transmission order; If the value of the decoding order indication identifier is the second value, it means that the decoding order of the meta-codestream data is different from the transmission order, and the RTP payload of the video bitstream data also includes: an N-byte PID, used to indicate the decoding order of the meta-codestream data, where N is equal to the sum of the number of bytes indicated by the payload decoding order field decode_order_size and 1. The method according to claim 11, wherein the aggregate extended payload header comprises: A sequence header field sequence_header_index is used to indicate whether the aggregated payload includes sequence header data; The random access index field AU_index is used to indicate whether the aggregated payload includes a random access frame. The method according to any one of claims 1 to 14, wherein the universal load head comprises: The payload encapsulation type field payload_package_type is used to indicate the RTP payload type, which includes a single payload, a fragmented payload, or an aggregated payload; The temporal_id field is used to indicate the time layer identifier to which the RTP payload belongs. The library dependency field library_dependency is used to indicate the type of elementary stream to which the RTP payload belongs; The x field is used to indicate whether an extension header is included, and the extension header includes a fragment extension payload header or an aggregate extension payload header. The method according to any one of claims 1 to 14, wherein The data type of the meta-stream data or the data type of the meta-stream data to which the meta-stream data fragment belongs includes: Sequence header, video extension data, user data after the sequence header, I frame image, RL frame image, inter-frame image; or The data type of the meta-stream data or the data type of the meta-stream data to which the meta-stream data fragment belongs includes: Sequence header, video extension data, user data after the sequence header, I frame image header, RL frame image header, P frame image header, B frame image header, video extension data after the image header, user data after the image header, frame slice data. The method according to claim 1, wherein the RTP header includes one or more of the following: A marker field M is used to indicate the boundary of a video frame; The payload field PT is used to identify the RTP session and indicate the payload format; The timestamp field Timestamp is used to indicate the sampling time of the RTP data packet; The sequence number field, Sequence number, is used to indicate the transmission order of RTP data packets. A video code stream processing method, applied to a decoding end, comprising: Receiving an RTP data packet, wherein the RTP data packet is obtained by performing RTP encapsulation on the meta-stream data or the meta-stream data fragments; Decoding the RTP data packet; Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data; The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data; or The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data. The method according to claim 18, wherein the method further comprises: Acquire indication information, wherein the indication information is used to indicate a formation method of the meta-stream data or the meta-stream data to which the meta-stream data fragment belongs; The meta-stream data or the meta-stream data to which the meta-stream data fragment belongs is a bit stream segment starting with a start code, including a sequence header, user data after the sequence header, extended data after the sequence header, an intra-frame prediction image, and an inter-frame prediction image; or The meta-codestream data or the meta-codestream data to which the meta-codestream data fragment belongs is the data between every two adjacent start code prefixes in the video bit stream and contains the start code prefix fragment, including a sequence header, user data after the sequence header, extended data after the sequence header, an I frame image header, an RL frame image header, an inter-frame image header, user data after the image header, extended data after the image header, and frame slice data. The method according to claim 18, wherein the decoding of the RTP data packet comprises: Parsing the RTP data packet to obtain an RTP packet header and an RTP payload of video bit stream data; Decoding is performed based on the RTP payload of the video bit stream data. The method according to claim 20, wherein the decoding based on the RTP payload of the video bit stream data comprises: If it is determined according to the universal payload header that the RTP payload of the video bitstream data is a single payload and the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, obtaining the meta-codestream data in the payload data, obtaining a meta-codestream data packet, and decoding the meta-codestream data packet; If it is determined according to the universal load header that the RTP load of the video bit stream data is a single load and the value of the decoding order indication identifier is the second value, obtain the meta-code stream data in the load data to obtain a meta-code stream data packet; obtain the PID of the RTP load of the video bit stream data, sort the meta-code stream data packets according to the PID, and decode the sorted meta-code stream data packets. The method according to claim 20, wherein the decoding based on the RTP payload of the video bit stream data comprises: If it is determined according to the universal payload header that the RTP payload of the video bitstream data is a fragment payload and the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, determine the starting fragment and the ending fragment according to the fragment start field fragment_start and the fragment end field fragment_end in the fragment payload header; form meta-stream data using the target code stream fragment data determined according to the starting fragment and the ending fragment, form a meta-stream data packet according to the meta-stream data, and decode the meta-stream data packet; If it is determined according to the universal load header that the RTP load of the video bitstream data is a fragmented load and the decoding order indication identifier is a second value, determine the starting fragment and the ending fragment according to the fragment start field fragment_start and the fragment end field fragment_end in the fragment load header; use the target code stream fragment data determined according to the starting fragment and the ending fragment to form meta-code stream data, and form a meta-code stream data packet according to the meta-code stream data; obtain the PID of the RTP load of the video bitstream data, sort the meta-code stream data packets according to the PID, and decode the sorted meta-code stream data packets. The method according to claim 20, wherein the decoding based on the RTP payload of the video bit stream data comprises: If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate payload header and the meta-stream size field located before each meta-stream data in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded; or If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate extended payload header in the payload data header of the aggregate payload, the aggregate payload header located before each meta-stream data, and the meta-stream size field to form a meta-stream data packet, and the meta-stream data packet is decoded; or If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate extended payload header in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded; or If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the aggregate extended payload header and the meta-stream size field in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded; or If it is determined according to the general payload header that the RTP payload of the video bitstream data is an aggregate payload, the meta-stream data is processed according to the meta-stream size field in the payload data header of the aggregate payload to form a meta-stream data packet, and the meta-stream data packet is decoded; or If it is determined according to the general load header that the RTP load of the video bitstream data is an aggregate load, the meta-stream data is processed according to the aggregate load header in the load data header of the aggregate load to form a meta-stream data packet, and the meta-stream data packet is decoded. The method according to claim 23, wherein the meta-stream data is processed according to the aggregate payload header and the meta-stream size field located before each meta-stream data in the payload data header of the aggregate payload to form a meta-stream data packet, comprising: If the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, obtaining the meta-code stream data according to the meta-code stream size field to obtain a meta-code stream data packet; If the value of the decoding order indication identifier is the second value, the meta-code stream data is obtained according to the meta-code stream size field to obtain a meta-code stream data packet, the PID of the RTP payload of the video bit stream data is obtained, and the meta-code stream data packets are sorted according to the PID to obtain sorted meta-code stream data packets. The method according to claim 23, wherein the meta-codestream data is processed according to the aggregate extended payload header in the payload data header of the aggregate payload, the aggregate payload header located before each meta-codestream data, and the meta-codestream size field to form a meta-codestream data packet, comprising: If the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, obtaining the meta-codestream data according to the meta-codestream size field and the extended payload header to obtain a meta-codestream data packet; If the decoding order indication identifier is a second value, the meta-codestream data is obtained according to the meta-codestream size field and the extended payload header to obtain a meta-codestream data packet, the PID of the RTP payload of the video bitstream data is obtained, and the meta-codestream data packets are sorted according to the PID to obtain sorted meta-codestream data packets. The method according to claim 23, wherein the processing of the meta-stream data according to the aggregate extended payload header in the payload data header of the aggregate payload to form a meta-stream data packet comprises: If the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, obtaining the meta-stream data according to the aggregate extended payload header to obtain a meta-stream data packet; If the decoding order indication identifier is a second value, the meta-code stream data is obtained according to the meta-aggregation extended payload header to obtain a meta-code stream data packet, the PID of the RTP payload of the video bit stream data is obtained, the meta-code stream data packets are sorted according to the PID to obtain sorted meta-code stream data packets. The method according to claim 23, wherein the processing of the meta-stream data according to the aggregate extended payload header and the meta-stream size field in the payload data header of the aggregate payload to form a meta-stream data packet comprises: If the value of the decoding order indicator is the first value or the RTP payload of the video bitstream data does not include the decoding order indicator, obtaining the meta-stream data according to the aggregate extended payload header and the meta-stream size field to obtain a meta-stream data packet; If the decoding order indication identifier is a second value, the meta-code stream data is obtained according to the aggregate extended payload header and the meta-code stream size field to obtain a meta-code stream data packet, the PID of the RTP payload of the video bit stream data is obtained, the meta-code stream data packets are sorted according to the PID to obtain sorted meta-code stream data packets. The method according to claim 23, wherein the processing of the meta-stream data according to the meta-stream size field in the payload data header of the aggregate payload to form a meta-stream data packet comprises: If the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, obtaining the meta-code stream data according to the meta-code stream size field to obtain a meta-code stream data packet; If the decoding order indication identifier is a second value, the meta-code stream data is obtained according to the meta-code stream size field to obtain a meta-code stream data packet, the PID of the RTP payload of the video bit stream data is obtained, and the meta-code stream data packets are sorted according to the PID to obtain sorted meta-code stream data packets. The method according to claim 23, wherein the processing of the meta-stream data according to the aggregate payload header in the payload data header of the aggregate payload to form a meta-stream data packet comprises: If the value of the decoding order indicator is the first value or the RTP payload of the video bit stream data does not include the decoding order indicator, obtaining the meta-stream data according to the aggregate payload header to obtain a meta-stream data packet; If the decoding order indication identifier is a second value, the meta-stream data is obtained according to the aggregate load header to obtain a meta-stream data packet, the PID of the RTP load of the video bit stream data is obtained, the meta-stream data packets are sorted according to the PID, and the sorted meta-stream data packets are obtained. A video code stream processing device, applied to an encoding end, comprising: The first processing module is used to perform RTP encapsulation on the meta-stream data or the meta-stream data fragments to obtain an RTP data packet; A first sending module, used for sending the RTP data packet to the decoding end; Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data; The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data; or The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data. A video code stream processing device, applied to a decoding end, comprising: A first receiving module is used to receive an RTP data packet, wherein the RTP data packet is obtained by performing RTP encapsulation on the meta-stream data or the meta-stream data fragments; A first processing module, used for decoding the RTP data packet; Wherein, the RTP data packet includes: an RTP header and an RTP payload of video bit stream data; The RTP payload of the video bit stream data includes: a general payload header, a decoding order indicator, payload data headers corresponding to different types of RTP payloads, and payload data; or The RTP payload of the video bit stream data includes: a general payload header, payload data headers corresponding to different types of RTP payloads, and payload data. A communication device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the video code stream processing method as described in any one of claims 1 to 29. A readable storage medium for storing a program, wherein when the program is executed by a processor, the steps in the video code stream processing method as described in any one of claims 1 to 29 are implemented.