CN104038844B - A mobile live broadcast system based on MPEG‑2 standard - Google Patents

Abstract

A kind of mobile live broadcast system based on the standards of MPEG 2, including mobile device end, streaming media server and player end, the mobile device end includes video data acquiring subsystem, and the video data acquiring subsystem includes H264 video compression codings module, the standard package modules of MPEG 2 and thread pool module；Described thread pool module is connected with RTSP signalling modules, RTP transport modules；The streaming media server includes live data forwarding module, RTCP rate control modules, video request program module and RTSP signalling modules, the mobile live broadcast system also includes RTSP signal servers, and for sending, signaling is opened in RTSP requests or RTSP requests stop signaling.The present invention improves video flowing package mode and transmission mode, and is applied to mobile phone terminal, sets up a new signaling system and carries out unified management, it is ensured that the various pieces communication stability of system is good.

Description

A Mobile Live Broadcasting System Based on MPEG-2 Standard

technical field

The invention relates to the technical field of video live broadcast in the field of network live broadcast, in particular to a mobile live broadcast system.

Background technique

With the popularization of cable digital TV, the status of MPEG-2 system multiplexing technology has become increasingly prominent. According to the quality of the transmission medium, the MPEG-2 standard defines two multiplexed information streams: transport stream (TS: Transport Stream) and Program stream (PS: Program Stream). The difference between the TS stream and the PS stream is that the packet structure of the TS stream is of fixed length, while the packet structure of the PS stream is of variable length. Since the length of the PS packet changes, once the synchronization information of a certain PS packet is lost, the receiver will enter an out-of-synchronization state, resulting in serious information loss events. Since the TS code stream adopts a fixed-length packet structure, when a transmission error destroys the synchronization information of a certain TS packet, the receiver can detect the synchronization information in the packet behind it at a fixed position, thereby restoring synchronization, avoiding Information is lost. Therefore, when the channel environment is relatively bad and the transmission error is high, the TS code stream is generally used, and when the channel environment is good and the transmission error is low, the PS code stream is generally used. While providing sufficient flexibility for data multiplexing, it also exhibits many advantages, such as dynamic bandwidth allocation, scalability, scalability, anti-interference, and low cost of the receiver.

Contents of the invention

In order to overcome the serious network packet loss and inaccurate playback speed of the existing streaming media live broadcast system, the audio and video are sent and received in two channels, and the audio and video are recorded at the same time. Inaccurate, unable to perfectly support mainstream players and other deficiencies, the present invention provides an improved video stream packetization method and transmission mode, and applies it to mobile terminals, establishes a new signaling system for unified management, and ensures system A mobile live broadcast system based on the MPEG-2 standard with good communication stability in each part.

The technical solution adopted by the present invention to solve its technical problems is:

A mobile live broadcast system based on the MPEG-2 standard, comprising a mobile device end, a streaming media server and a player end, the mobile device end comprising a video data acquisition subsystem, the video data acquisition subsystem comprising an H264 video compression encoding module , MPEG-2 standard packet module and thread pool module; In the H264 video compression encoding module, call the API interface of the IOS system layer, open the camera function, collect the original video data, and carry out H264 encoding to each frame collected , obtain the ES basic code stream; In the described MPEG-2 standard packet module, based on the video compression coding of the MPEG-2 standard, the ES stream encoded by H.264 is packaged into a PES stream, and the PES stream is then pressed to a fixed length of 188B Add auxiliary data to form the TS code stream, then put the TS code stream in the RTP buffer pool; the thread pool module is connected with the RTSP signaling module and the RTP transmission module; in the RTSP signaling module, start a monitoring RTSP signal In the service of order, in the described RTP transmission module, when detecting that the data in the RTP buffer pool is not empty, just read the TS stream data from the buffer pool, send the RTP packet to the streaming media server, until in the RTP buffer pool data is empty;

Described streaming media server comprises live data forwarding module, RTCP bit rate control module, video on demand module and RTSP signaling module, in described live data forwarding module, all video data is forwarded; In described RTCP bit rate control module , using the RTCP Internet protocol to count and feed back the data packets collected and sent by the IOS device live broadcast software and the data packets received by the streaming media server, and to control and adjust the code rate collected and sent by the IOS live broadcast software when network congestion occurs ; In the video-on-demand module, the video file uploaded by the IOS live broadcast software is saved and meets the playback of the IOS live broadcast software anytime and anywhere.

The mobile live broadcast system also includes an RTSP signaling server, which is used for sending RTSP request start signaling or RTSP request stop signaling.

Further, in the thread pool module, when the IOS device goes online, it will communicate with the RTSP signaling module, start a service to monitor the RTSP signaling, and when receiving the RTSP request from the remote RTSP signaling server to open the signaling, send The H264 video compression coding module sends a camera capture message to start video capture, and at the same time activates the RTP transmission module to open up an RTP buffer pool for storing the TS stream data generated by the MPEG-2 standard packet module; When the RTSP request of the signaling server stops the signaling, it sends a closing camera collection message to the H264 video compression coding module to stop the video collection.

Furthermore, the video data acquisition subsystem also includes an RTCP control module: establish another link with the RTCP code rate control module, cooperate with monitoring network conditions, and adjust the frame rate of the video captured by the camera in real time according to different network conditions to adjust the sending The outbound bit rate adapts to changes in the network.

The video data acquisition subsystem also includes a REGISTER module, which registers the device with the server when the software is started, and becomes an online device.

The video data collection subsystem also includes a local storage module, which saves the TS stream data collected and coded by the camera during the live broadcast, and the user can watch the historical video.

The technical idea of the present invention is: adopt RTP/UDP transmission technology, thread pool management technology, streaming media distribution technology, RTSP protocol, TCP protocol and MPEG-2 video coding standard technology, realize IOS mobile phone video collection software, streaming media server and PC The communication and data transmission between the three end video players.

TS code stream transmission has been well applied in cable digital TV, and it has also proved the superiority of TS code stream for transmitting video data. This invention will apply MPEG-2 technology to the network environment of mobile Internet to transmit video For the data, the TS code stream is generated by soft coding on the mobile device side, and forwarded by the streaming media server for network live broadcast, which also ensures the same superiority of the TS code stream as in cable digital TV.

The beneficial effects of the present invention are mainly manifested in: effectively reducing network packet loss, more accurate playback speed, good synchronization and rate during playback, and perfect support for mainstream players.

Description of drawings

Figure 1 is the overall framework of the mobile live broadcast system based on the MPEG-2 standard.

Figure 2 is a design diagram of the IOS device data acquisition terminal module.

Figure 3 is a PES packet structure diagram, wherein, ES (Elementary Streams): original stream; PES (PacketizedElementary Streams): original stream of packets; Packet Start Code Prefix: packet start code prefix; Stream ID: stream ID; PES Packet Length: packet length of packet code stream; Optional PES Header: optional header of packet code stream; PES Packet Data Bytes: payload data bytes of packet code stream; PES Scrambling Control: packet code stream interference control symbol; PES Priority: packet code Stream priority; Data Alignment Indicator: data verification identifier; Copyright: version; Original Or Copy: whether it is a copy; 7Flags: 7 flags; PES HeaderData Length: packet stream header data length; Optional Fields: optional field; 0xFF: 0xFF filling byte; PTS (Presentation Time Stamp): display time stamp; DTS (Decoding Time Stamp): decoding time stamp; ESCR (Elementary Stream Clock Reference): original stream reference clock; ES Rate: original stream rate ;DSM Trick Mode: DSM trick mode; Additional Copy Information: additional version information; PESCRC (Packetized Elementary Stream Cyclic Redundancy Check): packet stream cyclic redundancy check; PES Extension: packet stream extension; Optional Fields: optional fields ;5Flags: 5 flags; PESPrivate Data: packet stream private data; Packet Header Field: packet header field; Program PacketSequence Counter: program packet sequence counter; P-STD Buffer: P-STD cache; PES Extension FieldLength: packet stream Extended field length; PES Extension Field Data: Packet code stream extended field data.

Figure 4 is a TS packet structure diagram, where Header: header; Payload: load; Sync Byte: synchronization code; Transport Error Indicator: transmission error indicator; Payload Uint Start Indicator: load unit start indicator; Transport Priority: transmission priority ;PID (Packet ID): Packet ID; TransportScrambling Control: Transmission interference control; Adaptation Field Control: Adaptive field control; Continuity Counter: Continuous counter; Adaptation Field: Adaptive field; Adaptation FieldLength: Adaptive field length; Discontinuity Indicator: Non-continuous identifier; Random AccessIndicator: random access identifier; Elementary Stream Priority Indicator: original code stream priority identifier; 5Flags: 5 identifiers; Optional Fields: optional field; Stuffing Bytes: filling bytes; PCR (ProgramClock Reference): Program reference clock; OPCR (Original Program Clock Reference): original program reference clock; Splice Countdown: splicing countdown; Transport Private Data Length: transmission private data length; Transport Private Data: transmission private data; Adaptation Field ExtensionLength: adaptive field extension length ;3Flags: 3 flags; Optional Fields: optional field; LTW_valid flag(legal time window): legal time window flag; LTW offset(legal time window): legal time window offset; Piecewise Rate: segmentation rate; Splice Type : connector type; DTS_next_au (decoding timestamp next access unit): decoding timestamp next access unit.

Figure 5 is an interaction diagram of RTSP signaling, where, REGISTER+MD5: Digest-encrypted registration signaling; 200OK: Received and verified successfully; OPTION: Optional signaling; DESCRIBE: Description signaling; 401Unauthorized+nonce: Carrying a random number Authorization verification failure signaling, code 401; DESCRIBE+username+response: carrying username and random number encrypted response description signaling; 200OK+sdp: session description success signaling; SETUP+Transport: session establishment signaling carrying port number ;100Try: trying; 200OK+session+transport: session ID and port information; PLAY+session: signaling played on the session ID; 200OK+RTP Info: RTP transmission information; TS/RTP/UDP: use RTP packets to transmit TS video stream; TEARDOWN+session: Signaling of closing on session ID.

Figure 6 is the IOS device thread pool management diagram

detailed description

The present invention will be further described below in conjunction with the accompanying drawings.

With reference to Fig. 1～Fig. 6, a kind of mobile live broadcasting system based on MPEG-2 standard comprises mobile device end, streaming media server and player end, and described mobile device end comprises video data collecting subsystem, and described video data collecting sub-system System comprises H264 video compression coding module, MPEG-2 standard encapsulation module and thread pool module; In described H264 video compression coding module, call the API interface of IOS system layer, open camera function, collect original video data, come to collection Each frame of H264 is encoded to obtain the ES basic code stream; in the described MPEG-2 standard packet module, based on the video compression encoding of the MPEG-2 standard, the ES stream encoded by H.264 is packaged into a PES stream, and the PES The fixed length of 188B adds the auxiliary data to form the TS code stream, and then puts the TS code stream in the RTP buffer pool; the thread pool module is connected with the RTSP signaling module and the RTP transmission module; the RTSP signaling module , start a service to monitor RTSP signaling, in the RTP transmission module, when it is detected that the data in the RTP buffer pool is not empty, it will read the TS stream data from the buffer pool, and send RTP to the streaming media server packet until the data in the RTP buffer pool is empty;

Described streaming media server comprises live data forwarding module, RTCP bit rate control module, video on demand module and RTSP signaling module, in described live data forwarding module, all video data is forwarded; In described RTCP bit rate control module , using the RTCP Internet Protocol to make statistics and feedback on the data packets collected and sent by the IOS device live broadcast software and the data packets received by the streaming media server, in order to control and adjust the code rate collected and sent by the IOS live broadcast software when network congestion occurs ; In the video-on-demand module, the video file uploaded by the IOS live broadcast software is saved and meets the playback of the IOS live broadcast software anytime and anywhere.

The mobile live broadcast system also includes an RTSP signaling server, which is used for sending RTSP request start signaling or RTSP request stop signaling.

Further, in the thread pool module, when the IOS device goes online, it will communicate with the RTSP signaling module, start a service to monitor the RTSP signaling, and when receiving the RTSP request from the remote RTSP signaling server to open the signaling, send The H264 video compression coding module sends a camera capture message to start video capture, and at the same time activates the RTP transmission module to open up an RTP buffer pool for storing the TS stream data generated by the MPEG-2 standard packet module; When the RTSP request of the signaling server stops the signaling, it sends a closing camera collection message to the H264 video compression coding module to stop the video collection.

Furthermore, the video data acquisition subsystem also includes an RTCP control module: establish another link with the RTCP code rate control module, cooperate with monitoring network conditions, and adjust the frame rate of the video captured by the camera in real time according to different network conditions to adjust the sending The outbound bit rate adapts to changes in the network.

The video data acquisition subsystem also includes a REGISTER module, which registers the device with the server when the software is started, and becomes an online device.

The video data collection subsystem also includes a local storage module, which saves the TS stream data collected and coded by the camera during the live broadcast, and the user can watch the historical video.

In this embodiment, the basic process of the mobile live broadcast system includes the shooting of live video on the mobile device, compression encoding, video data format encapsulation, RTP encapsulation and data transmission, and preservation of local video data; the receiving end receives data, decapsulates, and stores data , The video signal is generated, mainly using the mainstream player that supports the RTSP protocol already in the market, which can be used for secondary development and expansion; the streaming media server realizes the forwarding of the video data of the mobile device according to the request of the player, and adopts the RTCP control protocol to optimize The code rate on the mobile device side adapts to the network bandwidth for the best user experience; the RTSP signaling server is responsible for the transmission and forwarding of signaling control messages in the system, which is a key part of the system communication protocol. The management of each terminal in the network completes the interactive session between the sending end, the receiving end and the RTSP signaling server. This platform-based design not only facilitates the unified management of mobile devices and players, but also enhances the overall load capacity of the system. The design framework of the mobile live broadcast system is shown in Figure 1.

For mobile devices, taking IOS mobile phones as an example, the video capture software needs to use the ffmpeg encoding library to encode the original image data into H264 video frames and convert them into TS stream data packets. The thread pool technology used reduces the coupling between modules, allowing Each module can be programmed independently, and the functional module design block diagram is shown in Figure 2. The module is designed to:

Camera: mainly responsible for real-time collection of raw data; preview: mainly for users to watch the real-time images currently collected; H.264 encoding module: compress and encode the original image data collected by the camera in H.264, which is convenient for transmission in the network and reduces transmission Bit rate to the network; TS stream encapsulation module: based on MPEG-2 standard video compression encoding, the ES stream encoded by H.264 is packaged into a PES stream, and the PES stream is composed of a fixed length of 188B plus auxiliary data TS code stream; RTP packaging: pack the packaged TS code stream into RTP packets, then send the packaged RTP packets to the streaming media server, and count the successfully sent RTP packets, and feed them back to the RTCP control module; 3G/4G/WIFI module: choose the best network conditions to send RTP packets. RTCP control module: This module will establish another link with the RTCP bit rate control module of streaming media, cooperate with monitoring network conditions, and adjust the frame rate of video captured by the camera in real time according to different network conditions to adjust the bit rate sent out to adapt Changes in the network can avoid more serious packet loss caused by network congestion and improve the viewing experience of users; REGISTER module: register the device with the server when the software is started, and become an online device; local save module: when the device is live, Save the TS stream data collected and encoded by the camera. In order for the user to watch the historical video, it mainly includes two files, one is the .M3U8 file, which is used for the HLS technology of the single bit rate adaptive stream. There is a CIF bit rate, one is a .ts file, which stores sliced media files of video data; WIFI upload: when the user is in a bad network condition, the user can also record the video first, and save it locally for later Upload to the remote HLS video-on-demand server under the WiFi network to realize video-on-demand video for other users anytime and anywhere; local HLS and local playback: establish a local HLS service, and watch the video files locally.

Encapsulating H264 video frame data into TS code stream requires wrapping H264 video frame data with a PES header to form a PES packet. Usually, the length of a PES packet is not fixed. Generally, one video frame generates a PES packet. PTS (Presentation Time Stamp) and DTS (Decoding Time Stamp) data are added to the PES packet, and the PES packet is attached with system information according to the MPEG-2 standard to become a TS packet, thereby forming a complete TS stream. The start code of the PES packet is composed of the start code prefix of the packet and the flow identifier. The start code prefix of the packet is a fixed code word structure, and its value is 0x000001, which is used to synchronize the PES packet at both the sending and receiving ends. The stream identifier sets the basic stream type, and each ES stream generated by the encoder is assigned a unique identification ID number. According to this ID number, up to dozens of video ES streams can be identified and reuse. As shown in Figure 3, PTS and DTS are two main functional domain parameters, and other identification bits can vary according to different applications. PTS is used to inform the decoder when to display a decoded image frame, and The DTS indicates when to decode the received coded stream of a frame of image. As shown in Figure 4, when PES is encapsulated into TS packets, PCR (Program ClockReference) is a very important indicator, indicating the instantaneous value time tag of the system clock itself, which is used by the decoding end to restore the system timing clock consistent with the encoding end STC (System Time Clock). PTS and DTS rely on STC to decode and display at a normal rate at the decoder. PCR occupies a total of 6Bytes in the TS header, of which 6bits are reserved, and the other 42bits are divided into two parts: 33bits of PCR-Base and 9bits of PCR-Ext. PCR-Ease is obtained by triggering the counter with a 90kHz pulse after 27MHz pulse divided by 300, and then sampling the state of the counter; PCR-Ext is obtained by directly triggering the counter with a 27MHz pulse, and then sampling the state of the counter.

When the device is turned on, it calls the REGISTER module to register the device with the RTSP signaling server. If the registration is successful, it becomes an online device for scheduling by the RTSP signaling server. Use the open source VLC player as a test player supporting the RTSP protocol on the PC side, open the network stream rtsp://192.168.1.167:8554/test, and send the RTSP request stream signaling to the RTSP signaling server. When the RTSP signal Make the server verify when it receives the signaling. If it succeeds, it will request the device to enable the live video function. If the device accepts the request, it will tell the device the destination IP and port of the RTP packet that needs to be sent, that is, the IP and receiving port of the streaming media server, and Return the request success and RTP information to the VLC player, so that a video stream transmission link from the device to the PC player is established through the intermediate streaming server. The basic RTSP signaling interaction process is shown in Figure 5.

Referring to Figure 6, the use of thread pools for unified management of RTSP signaling modules, H264 video encoding, TS packet encapsulation, RTP transmission modules, and allocation of RTP buffer space reduces the coupling of each module, making it more flexible and scalable. When the IOS device REGISTER goes online, the thread pool module will communicate with the RTSP module and start a service that monitors RTSP signaling. When receiving an RTSP request from a remote RTSP server to enable signaling, the thread pool will allocate resources and send H264 video The compression encoding module sends the camera acquisition message to start video acquisition, and at the same time activates the RTP transmission module to open up an RTP buffer pool for storing the TS code stream data generated by the MPEG-2 standard packet module; When the RTSP requests to stop signaling, send the closing camera acquisition message to the H264 video compression encoding module, stop the video acquisition, and perform thread recovery; when the RTP transmission module detects that the data in the RTP buffer pool is not empty, it reads from the buffer pool Fetch TS code stream data, send RTP packets to the streaming media server until the data in the RTP buffer pool is empty, if the RTSP stop signal is received, the RTP thread is recycled, and the RTP buffer area is recycled at the same time.

Claims (1)

1. a kind of mobile live broadcast system based on Moving Picture Experts Group-2, including mobile device end, streaming media server and player end, It is characterized in that：The mobile device end includes video data acquiring subsystem, and the video data acquiring subsystem includes H264 video compression codings module, Moving Picture Experts Group-2 package module and thread pool module；The H264 video compression codings module In, the api interface of IOS system layers is called, camera function is opened, collects original video data, what collection was come is each Frame carries out H264 codings, obtains ES elementary streams；In described Moving Picture Experts Group-2 package module, the video based on Moving Picture Experts Group-2 Compressed encoding, PES streams are packaged into by the ES streams H.264 encoded out, and PES streams add helper data set by 188B regular length again Into TS code streams, then TS code streams are put into RTP cache pools；Described thread pool module and RTSP signalling modules, RTP transmission moulds Block is connected；In the RTSP signalling modules, start in the service of a monitoring RTSP signaling, the RTP transport modules, work as detection Data in RTP cache pools are not space-time, and TS bit stream datas are just read from cache pool, RTP is sent to streaming media server Bag, until RTP cache pools mileage evidence is sky；

The streaming media server includes live data forwarding module, RTCP rate control modules, video request program module and RTSP In signalling module, described live data forwarding module, all video datas are forwarded；The RTCP rate control modules In, using RTCP Internet protocols, what packet and the streaming media server termination that software collection live to ios device is sent were subject to Packet is counted and fed back, when occurring network congestion, and the code sent is gathered to control and adjust the live software ends of IOS Rate；In the video request program module, the video file of the live software uploads of IOS is preserved and meet the live softwares of IOS with When playback everywhere；

The mobile live broadcast system also includes RTSP signal servers, and for sending, signaling is opened in RTSP requests or RTSP requests stop Only signaling；

In the thread pool module, when ios device is reached the standard grade, it can start one with the communication of RTSP signalling modules and monitor RTSP letters The service of order, when receiving the request unlatching signalings of the RTSP from long-range RTSP signal servers, to H264 video compression codings Module, which is sent, opens shooting collection message, carries out video acquisition, while activating RTP transport modules, opens up a RTP cache pool, For depositing the TS bit stream datas produced by Moving Picture Experts Group-2 package module；When receiving from long-range RTSP signal servers When RTSP requests stop signaling, sent to H264 video compression codings module and close shooting collection message, stop video acquisition；

The video data acquiring subsystem also includes RTCP control modules：Another chain is set up with RTCP rate control modules Connect, coordinate monitoring network condition, carry out the frame per second of adjustment camera collection video in real time to adjust according to different network conditions The code check seen off is haircuted to adapt to the change of network；

The video data acquiring subsystem also includes REGISTER modules, when software is opened to server registration equipment, into For the equipment reached the standard grade；

The video data acquiring subsystem also includes local preservation module, live while, and camera capturing and coding is gone out TS flow datas preserved, user can to history video recording watch.