CN116527981A - Multipath video synchronization method, system, device and storage medium - Google Patents

Abstract

The invention discloses a multipath video synchronization method, a multipath video synchronization system, a multipath video synchronization device and a multipath video synchronization storage medium, wherein the multipath video synchronization method comprises the following steps: acquiring an index file and a media fragment file sent by a streaming media server; the index file comprises reference time offset value information of a first frame image of each media slice file, and the media slice file comprises network delay time of a current video stream from a stream pushing end to a stream media server; determining a first media slicing file which is initially played by each path of video stream and a reference time offset value of a first frame image of the first media slicing file; determining a reference time offset value of a current frame of each path of video stream according to a preset time interval; synchronizing the normally played video streams according to the reference time offset value of the current frame of each video stream; the synchronization operation includes refresh, fast forward, or pause. The embodiment of the invention synchronizes multiple paths of video streams in a live broadcast scene, can independently control each path of video stream, and can be widely applied to the technical field of video processing.

Description

A multi-channel video synchronization method, system, device and storage medium

technical field

The present invention relates to the technical field of video processing, in particular to a multi-channel video synchronization method, system, device and storage medium.

Background technique

As a mature, stable and efficient audio and video transmission protocol, HLS (HTTP Live Streaming, an adaptive bit rate streaming media transmission protocol based on HTTP) has a wide range of application fields and superior performance characteristics. Communication plays an increasingly important role in providing better service and experience. HLS can be applied to various scenarios, such as video live broadcast, on-demand, online education, remote conferences, etc., to provide users with clear and smooth audio-visual effects, and also facilitate the release and management of content providers.

In related technologies, there are two modes for synchronizing multiple channels of HLS video streams. Traditional mode 1: The synchronization of multiple HLS video streams is realized by combining multiple video streams into one video stream on the streaming media server. , Zoom out of a certain video stream or change the page layout of several video streams, and at the same time, greatly increase the transcoding pressure of the streaming media server. Traditional mode 2: Synchronize the playback progress of video files that have been generated and ensure that the video source time of the first frame of multi-channel video is the same. For example, on-demand files in MP4 and other formats cannot meet various live broadcasts including HLS live broadcasts. Scenes.

Contents of the invention

In view of this, the purpose of the embodiments of the present invention is to provide a multi-channel video synchronization method, system, device and storage medium, which can synchronize multiple video streams in a live broadcast scene, and each video stream can be controlled independently.

On the one hand, the embodiment of the present invention provides a multi-channel video synchronization method, which is applied to the playback terminal and includes the following steps:

Obtain the index file and the media segment file sent by the streaming media server; wherein, the index file includes the reference time offset value information of the first frame image of each of the media segment files, and the media segment file includes the current The network delay time for the video stream to be pushed from the streaming end to the streaming media server;

According to the current time position played by the current video stream, the relative time relationship between the first frame image and the last frame image of each said media segment file, determine the first media segment file of the initial playback of each video stream, and from The index file obtains the reference time offset value of the first frame image of the first media segment file initially played by each video stream;

According to the preset time interval, according to the reference time offset value of the first frame image of the first media segment file initially played by each video stream, the current time of each video stream, and the first frame image of each video stream initially played The time of the first frame image of each media segment file determines the reference time offset value of the current frame of each video stream;

According to the preset time interval, calculate the maximum reference time offset value, the minimum reference time offset value and the average reference time offset value according to the reference time offset value of the current frame of each video stream, and calculate the maximum reference time offset value according to the maximum reference time offset value. The difference between the offset value and the minimum reference time offset value is used to synchronize the normally played video stream, or according to the difference between the average reference time offset value and the reference time offset value of the current frame of each video stream The relationship performs a synchronous operation on a normally played video stream; the synchronous operation includes refreshing, fast forwarding or pausing.

Optionally, according to the current time position played by the current video stream, and the relative time relationship between the first frame image and the last frame image of each media segment file, the first media to be initially played by each video stream is determined. Fragmented files, specifically including:

The relative time relationship between the first frame image and the last frame image of each media segment file in each video stream determines the duration of each media segment file in each video stream;

Accumulate the duration of each media segment file in each video stream in chronological order to obtain the cumulative time of each media segment file in each video stream;

The current time position played by each current video stream is matched with the cumulative time of each video stream to determine the first media segment file initially played by each video stream.

Optionally, according to the reference time offset value of the first frame image of the first media segment file initially played by each video stream, the current time of each video stream, and the first frame image of each video stream initially played, The time of the first frame image of each media segment file determines the reference time offset value of the current frame of each video stream, specifically including:

Calculate the difference between the current time of each video stream and the time of the first frame image of the first media segment file initially played by each video stream;

The reference time offset value of the current frame of each video stream is determined according to the difference value and the reference time offset value of the first frame image of the first media segment file initially played by each video stream.

Optionally, the normal playing video stream is synchronized according to the difference between the maximum reference time offset value and the minimum reference time offset value, or according to the average reference time offset value and each The relationship between the reference time offset values of the current frame of the video stream performs synchronous operations on the normally played video stream, specifically including:

For a normally played video stream, if the difference between the maximum reference time offset value and the minimum reference time offset value is greater than a preset threshold, refresh each video stream;

If the difference between the maximum reference time offset value and the minimum reference time offset value is less than the preset threshold, and the average reference time offset value is greater than the reference time offset value of the current frame, the path The video stream is fast forwarded to the first preset time period;

If the difference between the maximum reference time offset value and the minimum reference time offset value is smaller than the preset threshold, and the average reference time offset value is smaller than the reference time offset value of the current frame, the path The video stream is paused for a second preset time period.

Optionally, the method also includes:

For video streams that are not being played normally, the synchronous operation is suspended until normal playback is resumed; the abnormal playing includes any one of a paused state, a video loading state, or a video loading failure state.

On the other hand, the embodiment of the present invention provides a multi-channel video synchronization method, which is applied to the streaming terminal, including:

Calculate the network delay time of the current video stream from the streaming end to the streaming server;

Send the video stream and the network delay time to the streaming media server, so that the streaming media server generates index files and media fragment files, wherein the index files include the reference time of the first frame image of each media fragment file Offset information.

Optionally, the network delay time is calculated by the following method:

Send the first request to the network delay query interface of the streaming media server, receive the feedback information of the network delay query interface, record the first moment of sending the first request and the fourth moment of receiving the feedback information; the feedback information includes The second time when the first request enters the network delay query interface and the third time when the first request leaves the network delay query interface;

Calculate the network delay time according to the first moment, the fourth moment, the second moment and the third moment.

Optionally, the method also includes:

Calculate the network delay time at least 3 times;

Remove the maximum and minimum values of at least 3 network delay times to obtain the remaining network delay time;

Update the network delay time to the average of the remaining network delay times.

On the other hand, the embodiment of the present invention provides a multi-channel video synchronization method applied to a streaming media server, including:

Receive the video stream and network delay time sent by the streaming end;

Generate an index file and a media fragment file according to the video stream; wherein, the index file includes the reference time offset value information of the first frame image of each media fragment file, and the media fragment file includes the current video stream The network delay time for streaming to the streaming media server.

Optionally, the reference time offset value information is calculated by the following method:

Send a second request to the query interface of the time reference server;

Receive the reference time offset value information fed back by the query interface of the time reference server.

On the other hand, the embodiment of the present invention provides a multi-channel video synchronization system, which is applied to the playback terminal, including:

The first module is used to obtain the index file and the media segment file sent by the streaming media server; wherein, the index file includes the reference time offset value information of the first frame image of each of the media segment files, and the The media segment file includes the network delay time for the current video stream to be pushed from the streaming end to the streaming server;

The second module is used to determine the first media to be initially played by each video stream according to the current time position played by the current video stream, and the relative time relationship between the first frame image and the last frame image of each media fragment file Fragmentation file, and obtain the reference time offset value of the first frame image of the first media fragmentation file of the first media fragmentation file of each road video stream initial play from described index file;

The third module is used to, according to the preset time interval, according to the reference time offset value of the first frame image of the first media segment file initially played by each video stream, the current time of each video stream, and the current time of each video stream. The time of the first frame image of the first media fragment file played initially by the stream determines the reference time offset value of the current frame of each video stream;

The fourth module is used to calculate the maximum reference time offset value, the minimum reference time offset value and the average reference time offset value according to the reference time offset value of the current frame of each video stream according to the preset time interval, according to The difference between the maximum reference time offset value and the minimum reference time offset value performs a synchronous operation on the normally played video stream, or according to the average reference time offset value and the reference time of the current frame of each video stream The relationship between the offset values performs a synchronous operation on a normally played video stream; the synchronous operation includes refreshing, fast forwarding or pausing.

On the other hand, the embodiment of the present invention provides a multi-channel video synchronization system, which is applied to the streaming end, including:

The fifth module is used to calculate the network delay time of the current video stream from the streaming end to the streaming server;

The sixth module is used to send the video stream and network delay time to the streaming media server, so that the streaming media server can generate index files and media segment files.

On the other hand, an embodiment of the present invention provides a multi-channel video synchronization system applied to a streaming media server, including:

The seventh module is used to receive the video stream and network delay time sent by the streaming end;

The eighth module is used to generate an index file and a media fragment file according to the video stream; wherein, the index file includes the reference time offset value information of the first frame image of each media fragment file, and the media fragment The slice file includes the network delay time when the current video stream is pushed to the streaming media server.

On the other hand, an embodiment of the present invention provides a multi-channel video synchronization device, including:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor is made to implement the video synchronization method at any one of the streaming end, the streaming media server, or the playback end.

On the other hand, an embodiment of the present invention provides a computer-readable storage medium, which stores a processor-executable program, and the processor-executable program is used to execute the above streaming end, A video synchronization method at either end of the streaming media server or playback end.

On the other hand, an embodiment of the present invention provides a multi-channel video synchronization system, including a streaming media server and a streaming end and a playing end connected to the streaming media server; wherein,

The streaming end is used to execute the video synchronization method of the streaming end mentioned above;

The streaming media server is used to execute the video synchronization method of the streaming media server;

The playback terminal is used to execute the video synchronization method of the playback terminal.

Implementing the embodiment of the present invention includes the following beneficial effects: first, in this embodiment, the network delay time of the current video stream from the stream pushing end to the streaming media server is calculated by the streaming end, and the network delay time is sent to the streaming media server, considering The video time error caused by the network delay of the video stream from the streaming end to the streaming media server is eliminated; then, the media server generates an index file and a media segment file for the video stream, and sends the index file and media segment file For the player, the index file includes the reference time offset value information of the first frame image of each media segment file, and the media segment file includes the network delay time for the current video stream to be pushed to the streaming media server, passing the reference time The offset value information takes into account the video time error caused by the system time error between different streaming media servers, and can be applied to multiple different streaming media servers; then, after the player receives the index file and the media fragment file, it determines the video time error of each channel. The base time offset value of the first frame image of the first media fragment file played initially by the stream, and then determine the base time offset value of the current frame of each video stream, and according to the base time offset value of the current frame of each video stream The offset value is used to synchronize the normal playing video stream, and determine the base time offset value of the current frame of each video stream by obtaining the base time offset value of the first frame image of the first media segment file initially played by each video stream. Shift value, the calculation method is simple and does not need to analyze the data packet of the media fragment file, and the normal playback video stream is synchronized through the reference time offset value of the current frame of each video stream, the synchronization effect is good, and the time accuracy can reach the millisecond level , which can realize the synchronization of multiple video streams in the live broadcast scene, and can control each video stream independently.

Description of drawings

Fig. 1 is a schematic structural diagram of a multi-channel video synchronization system provided by an embodiment of the present invention;

Fig. 2 is a schematic flow chart of the steps of a multi-channel video synchronization method applied to a playback terminal provided by an embodiment of the present invention;

Fig. 3 is a schematic flow chart of steps for determining the first media segment file for initial playback of a video stream provided by an embodiment of the present invention;

FIG. 4 is a schematic flowchart of steps for determining a reference time offset value of a current frame of a video stream provided by an embodiment of the present invention;

FIG. 5 is a schematic flowchart of steps for synchronizing multiple channels of video provided by an embodiment of the present invention;

Fig. 6 is a schematic flow chart of the steps of a multi-channel video synchronization method applied to the streaming terminal provided by an embodiment of the present invention;

FIG. 7 is a schematic flow chart of the steps of a multi-channel video synchronization method applied to a streaming media server provided by an embodiment of the present invention;

FIG. 8 is a structural block diagram of a multi-channel video synchronization system applied to a playback terminal provided by an embodiment of the present invention;

FIG. 9 is a structural block diagram of a multi-channel video synchronization system applied to the streaming end provided by an embodiment of the present invention;

FIG. 10 is a structural block diagram of a multi-channel video synchronization system applied to a streaming media server provided by an embodiment of the present invention;

Fig. 11 is a structural block diagram of a road video synchronization system device provided by an embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, it is only set for the convenience of illustration and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art sexual adjustment.

It can be understood that terms such as "first" and "second" used in the embodiments of the present invention may be used to describe various concepts herein, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another.

The terms "at least one", "multiple", "each", "any", etc. used in the embodiments of the present invention, at least one includes one, two or more, and multiple includes two or more , each refers to each of the corresponding plurality, and any refers to any one of the plurality.

In related technologies, there are two modes for multi-channel HLS streaming media synchronization. One mode cannot independently operate each video stream, and the other mode is limited by many conditions and cannot be applied to live broadcast scenarios. The main reasons for the delay of HLS streaming media playback are: 1. The network delay of the streaming server pushing the stream to the streaming media server; 2. CDN distribution delay, playback end pull-down delay, decoding rendering delay, etc.

Based on the above situation, embodiments of the present invention provide a multi-channel video synchronization method, system, device, and storage medium. First, the network delay time of the current video stream from the streaming end to the streaming media server is calculated through the streaming end, And send the network delay time to the streaming media server, taking into account the video time error caused by the network delay of the video stream from the streaming end to the streaming media server; then, generate index files and media fragment files for the video stream through the media server , and send the index file and media segment file to the player, where the index file includes the reference time offset value information of the first frame image of each media segment file, and the media segment file includes the current video stream pushed to The network delay time of the streaming media server, through the reference time offset value information, considers the video time error caused by the system time error between different streaming media servers, and can be applied to multiple different streaming media servers; then, the player receives the index file and the media fragment file, determine the reference time offset value of the first frame image of the first media fragment file initially played by each video stream, and then determine the reference time offset value of the current frame of each video stream, and Synchronize the normally played video streams according to the reference time offset value of the current frame of each video stream, by obtaining the reference time offset value of the first frame image of the first media segment file initially played by each video stream Determine the reference time offset value of the current frame of each video stream. The calculation method is simple and does not need to parse the media fragment file data packet. The normal playback video stream is synchronized through the reference time offset value of the current frame of each video stream. It can realize the synchronization of multiple video streams in the live broadcast scene, the synchronization effect is good, the time accuracy can reach the millisecond level, and each video stream can be controlled separately.

Referring to Fig. 1, Fig. 1 is a schematic diagram of an implementation environment of a multi-channel video synchronization method provided by an embodiment of the present invention, the multi-channel video synchronization method is applied in a multi-channel video synchronization system, and the multi-channel video synchronization system includes a streaming terminal 110 , streaming media server 120 and player 130. The streaming terminal 110 calculates the network delay time of the current video stream, and sends the captured video stream and the network time of the current video stream to the streaming media server 120; the streaming media server 120 makes the received video stream into index files and media fragments file, and send the index file and the media fragment file to the playback terminal 130; after the playback terminal 130 receives the index file and the media fragment file, it simultaneously plays multiple video streams synchronously.

It should be noted that the streaming end 110 is connected to the streaming media server 120 by wire or wirelessly, and the streaming media server 120 is connected to the playback end 130 by wire or wirelessly. The streaming media server 120 may also be a cluster composed of multiple streaming media servers, and the number of streaming media servers in the cluster is determined according to actual application requirements. The push stream terminal 110 or the playback terminal 130 involved in the embodiment of the present application includes, but is not limited to, smart phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, and the like.

Referring to Fig. 2, the embodiment of the present invention provides a kind of multi-channel video synchronous method, is applied to the playback end, comprises the following steps:

S110, obtain the index file and the media segment file sent by the streaming media server; wherein, the index file includes the reference time offset value information of the first frame image of each media segment file, and the media segment file includes the current video stream from the pushed The network delay time when the streaming end pushes the stream to the streaming media server.

The playback end receives the index file and the media segment file sent by the streaming media server. In this embodiment, the index file includes but not limited to M3U8 file, and the media segment file includes but not limited to TS file. The position of the reference time offset value information of the first frame image of the media segment file in the index file is determined according to the actual application. The network delay time of the current video stream from the streaming end to the streaming server is within the The location is determined according to actual applications, and is not specifically limited in this embodiment.

S120, according to the current time position of the current video stream playback, the relative time relationship between the first frame image and the last frame image of each media segment file, determine the first media segment file for the initial playback of each video stream, and from The index file obtains the reference time offset value of the first frame image of the first media segment file initially played by each video stream.

When playing video on the HLS player side, the M3U8 file includes multiple TS files, and the player does not necessarily start playing from the first TS file in the TS file list in the M3U8 file, so it needs to analyze the current position of the current video playback The relationship between the relative time of the first frame and the relative time of the last frame of each TS slice, confirm which TS file is played at the beginning of each stream on the current player, and then obtain the corresponding TS file of the first frame of the video stream The base time offset value (in milliseconds).

S130. According to the preset time interval, according to the reference time offset value of the first frame image of the first media segment file initially played by each video stream, the current time of each video stream, and the initial playback time of each video stream The time of the first frame image of the first media segment file determines the reference time offset value of the current frame of each video stream.

It should be noted that the preset time interval refers to the time interval of video synchronization, and the preset time interval is determined according to the actual application. This embodiment does not make specific limitations. For example, the preset time interval is set to 1 second.

The playback end calculates the reference time offset value corresponding to the first frame of the TS file played at the beginning of each video stream, the current video stream playback time, and the reference time when the frame is generated corresponding to the latest frame played by the current TS file The offset value calculates the reference time offset value of the current frame of each video stream, and the obtained reference time has taken into account the delay factor of the streaming end, which is more accurate. Among them, the current video stream playback time can be directly obtained through the native js of the web page H5, and the reference time offset value corresponding to the latest frame played by the current TS file when generating this frame can be obtained through the file name of the current TS file. It should be noted that when the above time units are inconsistent, unit conversion is performed. For example, the unit of the reference time offset value corresponding to the first frame of the TS file played at the beginning is seconds, and the unit of the current video stream playback time is milliseconds. Calculate During the process, milliseconds and seconds need to be converted.

S140. According to the preset time interval, calculate the maximum reference time offset value, the minimum reference time offset value and the average reference time offset value according to the reference time offset value of the current frame of each video stream, and calculate the maximum reference time offset value according to the maximum reference time offset value The difference between the minimum reference time offset value and the normal playback video stream is synchronized, or according to the relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream. Streams perform synchronous operations; synchronous operations include flushing, fast-forwarding, or pausing.

Similarly, the preset time interval refers to the time interval of video synchronization, and the preset time interval is determined according to the actual application. This embodiment does not make specific limitations. For example, the preset time interval is set to 1 second.

Specifically, the player first determines characteristic parameters such as the maximum reference time offset value, the minimum reference time offset value, and the average reference time offset value based on the reference time offset value of the current frame of each video stream, and then determines the Synchronize the normally played video streams, or perform synchronous operations on the normally played video streams according to the relationship between these characteristic parameters and the reference time offset value of the current frame of each video stream.

Optionally, the multi-channel video synchronization method at the playback end also includes:

S150. For the video stream that is not being played normally, the synchronous operation is suspended until it is played normally again; the abnormal playing includes any one of a paused state, a video loading state, or a video loading failure state.

It should be noted that the video streams participating in the synchronization do not include the video streams that are not playing normally, such as those in the paused state, the video loading state, or the video loading failure. When these three video streams are playing normally, they can participate in the video synchronization. Pausing the synchronous operation of the video stream that is not played normally can reduce the unnecessary calculation amount and reduce the calculation pressure.

Optionally, referring to FIG. 3, the first media to be initially played by each video stream is determined according to the current time position played by the current video stream, and the relative time relationship between the first frame image and the last frame image of each media segment file. Fragmented files, specifically including:

S121. The relative time relationship between the first frame image and the last frame image of each media segment file in each video stream determines the duration of each media segment file in each video stream;

S122. Accumulate the duration of each media segment file in each video stream according to time sequence, to obtain the accumulated time of each media segment file in each video stream;

S123. Match the current time position played by each current video stream with the accumulated time of each video stream, and determine the first media segment file initially played by each video stream.

Specifically, the duration of each media segment file can be obtained by subtracting the playback time of the first frame image from the playback time of the last frame image; The accumulated time of each media fragment file in each video stream is accumulated; the current time position of the current video stream playback is within a certain accumulation time range, and the corresponding media fragment file within the accumulation time range is used as The first media slice file.

Specifically, the function to obtain the first TS file played by each video stream is as follows:

Fb＝fb(Tsc _x ,Tsd ₁ ,Tsf ₁ ,Tsd ₂ ,Tsf ₂ ,Tsd ₃ ,Tsf ₃ ...Tsd _n ,Tsf _n )

Among them, Fb is the file name of the TS file played first in the current stream, fb is a function to obtain the file name of the TS file played first in the current stream, and Tsc _x is the currentTime when the xth player initially plays (video The current initial position of playback), Tsd ₁ is the total duration of the first TS file, Tsf ₁ is the file name of the first TS file, Tsd _n is the total duration of the nth TS file, Tsf _n is the nth TS The file name of the file; when Tsc∈[0,Tsd ₁ ), then return Tsf ₁ , when Then return Tsf _n .

In a specific embodiment, the HLS player records the initial time of each HLS video stream:

When each HLS video stream starts to play, record the currentTime of each player (the currentTime of the initial video playback, this time is the time generated by the player itself). Exemplarily, there are 3 video streams, and the first video stream is marked as Tsc ₁ , the second video stream is marked as Tsc ₂ , and the third video stream is marked as Tsc ₃ .

Analyze the M3U8 file information of the currently playing HLS: When each HLS video starts to play, by analyzing the corresponding M3U8 file information, obtain the first played TS file (not necessarily starting from the first TS file).

In a specific embodiment, the content of the shili.m3u8 file is as follows:

#EXTM3U

#EXT-X-VERSION:3

#EXT-X-MEDIA-SEQUENCE:4176

#EXT-X-TARGETDURATION:10

#EXTINF: 9.493,1234

73e162672dd62b880192796a1816b998_20.ts

#EXTINF: 8.322,10727

d5b8473b826887491449319f277c7c06_40.ts

#EXTINF: 2.633,19049

e88dd1fd8aa1b880e16618de42e25230_125.ts

There are 3 TS file information in this M3U8 file, the first TS file information is:

#EXTINF: 9.493,1234

73e162672dd62b880192796a1816b998_20.ts

The first line includes 2 values, which are directly separated by commas. The first value indicates that the current TS file is 9.493 seconds long, that is, if the first frame time of the current TS file is 0 (in seconds), the last One frame is 9.493 (in seconds); the second value (here, 1234) represents the reference time offset value (in milliseconds) corresponding to the first frame of the current video file.

The second line contains the video file name of the current TS file, and the player downloads the ts video file through this name. (20 in the file name here) indicates the network delay (in milliseconds) of the current video stream push end to the streaming media server.

The second TS file information is:

#EXTINF: 8.322,10727

d5b8473b826887491449319f277c7c06_40.ts

The first line includes 2 values, the 2 values are directly separated by English commas, the first value indicates that the current TS file is 8.322 seconds long, that is, the time of the first frame of the current TS file is 9.493 (in seconds, in the current M3U8 If the first frame of the first TS file in the file is the start time), the last frame is 9.493+8.322=17.815 (unit second); the second value (here is 10727) indicates that the current video file is the first The reference time offset value corresponding to the frame (in milliseconds).

The second line contains the video file name of the current TS file. The player downloads the ts video file through this name (40 in the file name here) indicates the network delay (unit: millisecond).

The third TS file information is:

#EXTINF: 2.633,19049

e88dd1fd8aa1b880e16618de42e25230_125.ts

The first line includes 2 values, which are directly separated by English commas. The first value indicates that the current TS file is 2.633 seconds long, that is, the time of the first frame of the current TS file is 17.815 (in seconds, in the current M3U8 If the first frame of the first TS file in the file is the start time), the last frame is 9.493+8.322+2.633=20.448 (in seconds); the second value (19049 here) represents the current video file The reference time offset value corresponding to the first frame (in milliseconds). The second line contains the video file name of the current TS file, through which the player downloads the ts video file (125 in the file name here) indicates the network delay (unit: millisecond).

When each HLS video starts to play, judge whether the Tsc1 of the current player is within the start time of which of the above three TS files, and then judge which TS file is the first to play in the current stream. For example: when 0≤Tsc1<9.493, it means that the current player initially plays the first TS file; 17.815≤Tsc1<20.448, then it means that the current player initially plays the third TS file.

Optionally, referring to FIG. 4 , according to the reference time offset value of the first frame image of the first media segment file initially played by each video stream, the current time of each video stream, and the initial playback time of each video stream The time of the first frame image of the first media segment file determines the reference time offset value of the current frame of each video stream, including:

S131. Calculate the difference between the current time of each video stream and the time of the first frame image of the first media segment file initially played by each video stream;

S132. Determine the reference time offset value of the current frame of each video stream according to the difference value and the reference time offset value of the first frame image of the first media segment file initially played by each video stream.

In a specific embodiment, the formula for calculating the specific reference time offset value (measured in streaming media server time) played by each stream is:

T _x ＝Tss _x +(Tc _x -Tsc _x )×1000

Among them, Tx represents the reference time offset value of the latest frame played by the X-th video stream, and Tss _x represents the stream media server’s reference time offset corresponding to the first frame of the first TS file starting to play the X-th video stream Value, in milliseconds; Tc _x indicates the latest current time value of the Xth video stream, in seconds; Tsc _x indicates the current time currentTime value of the Xth video stream starting to play the first frame of the first TS file, in units for seconds.

For example, to obtain the current time (currentTime) and playback duration of each HLS stream, the current time of each HLS stream is as follows: the current time of the first video stream is recorded as Tc ₁ , and the current time of the second video stream is recorded as is Tc ₂ , the current time of the third video stream is recorded as Tc ₃ ; the current duration of each HLS stream is as follows: the current duration of the first video stream is recorded as Tt ₁ = Tc ₁ -Tsc ₁ , the second video stream The current duration of the stream is recorded as Tt ₂ = Tc ₂ -Tsc ₂ , and the current duration of the third video stream is recorded as Tt ₃ = Tc ₃ -Tsc ₃ ; the benchmark of the latest frame of the first TS file initially played by each stream The time offset value is as follows: the reference time offset value of the first frame of the first TS file initially played by the first video stream is recorded as Tss ₁ , and the first frame of the first TS file initially played by the second video stream is recorded as Tss 1 . The reference time offset value of the frame is recorded as Tss ₂ , and the reference time offset value of the first frame of the first TS file initially played by the third video stream is recorded as Tss ₃ . The reference time offset value of the current frame played by each HLS stream is calculated as follows: The reference time offset value of the latest frame played by the first video stream: T ₁ =Tss ₁ +(Tc ₁ -Tsc ₁ )×1000 , the reference time offset value of the latest frame played by the second video stream: T ₂ =Tss ₂ +(Tc ₂ -Tsc ₂ )×1000, the reference time offset value of the latest frame played by the third video stream Shift value: T ₃ =Tss ₃ +(Tc ₃ −Tsc ₃ )×1000.

Optionally, referring to FIG. 5 , the normal playing video stream is synchronized according to the difference between the maximum reference time offset value and the minimum reference time offset value, or according to the average reference time offset value and the current frame of each video stream The relationship between the reference time offset values of the normal playback video stream is synchronized, including:

S141. For a normally played video stream, if the difference between the maximum reference time offset value and the minimum reference time offset value is greater than a preset threshold, refresh each video stream;

S142. If the difference between the maximum reference time offset value and the minimum reference time offset value is less than the preset threshold, and the average reference time offset value is greater than the reference time offset value of the current frame, fast forward the video stream to the first preset time period;

S143. If the difference between the maximum reference time offset value and the minimum reference time offset value is less than the preset threshold, and the average reference time offset value is less than the reference time offset value of the current frame, suspend the video stream for the second preview Set time period.

It should be noted that the preset threshold is determined according to an actual application and is not specifically limited in this embodiment. For example, the preset threshold is 80 seconds. The first preset time period or the second preset time period is determined according to the magnitude of the average reference time offset value and the reference time offset value of the current frame.

Specifically, the maximum T _x of each video stream is calculated as Tmax, and the minimum T _x of each stream is recorded as Tmin by comparing the specific reference time offset value T _x of each video stream played to the latest frame. If Tmax-Tmin is greater than a certain threshold (for example, 100 seconds), the synchronization operation is performed, and the synchronization operation is divided into two schemes. First judge whether it meets the synchronization scheme 1: when Tmax-Tmin is too large and exceeds a set threshold, such as 3000 milliseconds, then use the player to refresh the video streams participating in synchronization for synchronization, otherwise use synchronization scheme 2; synchronization scheme 2: Calculate and obtain the average value of the specific reference time offset value of each stream played to the latest frame as Tpj, and then compare the T _x value of each video stream with Tpj, and fast-forward the slow stream to Tpj-T _x milliseconds, the fast-playing video stream starts playing after pausing for the duration of T _x- Tpj, so as to achieve the effect of synchronous playback of all streams. When Tmax-Tmin is too large, it exceeds a set threshold. In this case, the effect of using synchronization scheme 2 is not good, and it will cause frequent reloading of the video, so synchronization scheme 1 is used.

Referring to Fig. 6, the embodiment of the present invention provides a multi-channel video synchronization method, which is applied to the streaming end, including:

S210. Calculate the network delay time of the current video stream from the streaming end to the streaming media server;

S220. Send the video stream and network delay time to the streaming media server, so that the streaming media server generates an index file and a media segment file, wherein the index file includes the reference time of the first frame image of each media segment file Offset information.

Specifically, the streaming terminal first calculates the network delay time of the current video stream from the streaming terminal to the streaming server, and then sends the multiple video streams and network delay time to the streaming server; the streaming server receives multiple video streams After stream and network delay time, generate index file and media fragment file.

Optionally, the network delay time is calculated by the following method:

S211. Send the first request to the network delay query interface of the streaming media server, receive the feedback information from the network delay query interface, and record the first moment of sending the first request and the fourth moment of receiving the feedback information; the feedback information includes the first The second moment when the request enters the network delay query interface and the third moment when the first request leaves the network delay query interface;

S212. Calculate the network delay time according to the first moment, the fourth moment, the second moment and the third moment.

Specifically, the streaming end sends a request to the network delay query interface of the streaming media server, receives feedback information, and then calculates the network delay time according to the feedback information, sending time and receiving time. For example, calculate and record the number of milliseconds of network delay from the streaming end to the streaming media server for each stream (recorded as T _TLYS ). The calculation method of T _TLYS is to send the request from the streaming end to the network delay query interface provided by the streaming media server. The streaming end records the sending request time T _TLFS (in milliseconds) and the time of receiving the return value T _TLFH (in milliseconds). The data returned by the delay query interface includes the time T _TLJR (number of milliseconds) when the request enters the interface, and the time T _TLLK (number of milliseconds) when the request leaves the interface. The formula for calculating the network delay of streaming from the streaming end to the streaming media server is: T _TLYS =(T _TLFH -T _TLFS -(T _TLLK -T _TLJR ))/2.

Optionally, the method also includes:

S2121. Calculate the network delay time at least 3 times;

S2122. Remove the maximum and minimum values of at least three network delay times to obtain the remaining network delay time;

S2123. Update the network delay time to the average value of the remaining network delay time.

In order to improve the accuracy of the network delay from the streaming end to the streaming media server, T _TLYS needs to be tested more than or equal to 3 times, then remove a maximum value and a minimum value, and take the average of the remaining values as T _TLYSPJ . For example, calculate the network delay time 4 times as: 1s, 0.5s, 0.8s, 0.9s, remove the maximum value of 1s and the minimum value of 0.5s, the average value of the network delay time is (0.8+0.9)/2=0.85s, The final calculated network delay time is 0.85s.

Referring to Fig. 7, the embodiment of the present invention provides a kind of multi-channel video synchronous method, is applied to streaming media server, comprises:

S310. Receive the video stream and the network delay time sent by the streaming end;

S320, generate an index file and a media fragment file according to the video stream; wherein, the index file includes the reference time offset value information of the first frame image of each media fragment file, and the media fragment file includes the current video stream pushed to the stream The network latency of the media server.

Specifically, the streaming media server receives the video stream and network delay time sent by the streaming end, and then generates index files and media segment files according to the video stream and network delay time. It should be noted that the storage locations of the reference time offset value information and the network delay time are determined according to actual applications, and are not specifically limited in this embodiment.

In a specific embodiment, when the streaming server slices the video push stream (generates a TS file), the reference time offset value information corresponding to the first frame of each slice is written into the M3U8 file corresponding to the current slice. After the duration value of #EXTINF, the middle English comma is separated; at the same time, the number of milliseconds T _TLYS of the network delay for pushing the current video stream to the streaming media server is included in the name of the current TS file, separated by symbols and other file names in the middle . For example, 20 in the TS file name xxxx_20.ts is the number of milliseconds of network delay for the current video stream to be pushed to the streaming media server. The sample file of M3U8 is as follows:

The content of the shili.m3u8 file is:

#EXTM3U

#EXT-X-VERSION:3

#EXT-X-MEDIA-SEQUENCE:4176

#EXT-X-TARGETDURATION:10

#EXTINF: 9.493,1234

73e162672dd62b880192796a1816b998_20.ts

#EXTINF: 8.322,10727

d5b8473b826887491449319f277c7c06_40.ts

#EXTINF: 2.633,19049

e88dd1fd8aa1b880e16618de42e25230_125.ts

Optionally, the reference time offset value information is calculated by the following method:

S321. Send a second request to the query interface of the time reference server;

S322. Receive reference time offset value information fed back by the query interface of the time reference server.

Specifically, the time reference server provides a unified reference time offset value acquisition service for the stream media server cluster. First, make the time reference server consistent with a certain standard time through the ntp service or the chrony time synchronization server. Then provide the reference time offset value interface service on the time reference server to provide a consistent reference time offset value acquisition service for the current streaming media server cluster. The reference time offset value is how much time has passed since the current streaming media server cluster was enabled Milliseconds, which can reduce the amount of data transmitted, and at the same time unify the time of the streaming server cluster. By providing a unified reference time offset value interface query service, the problem of time inconsistency between different streaming media servers can be avoided.

Referring to Fig. 8, an embodiment of the present invention provides a multi-channel video synchronization system applied to a playback terminal, including:

The first module is used to obtain the index file and the media segment file sent by the streaming media server; wherein, the index file includes the reference time offset value information of the first frame image of each media segment file, and the media segment file includes the current The network delay time of the video stream from the streaming end to the streaming server;

The second module is used to determine the first media segment for initial playback of each video stream according to the current time position played by the current video stream, and the relative time relationship between the first frame image and the last frame image of each media segment file file, and obtain the reference time offset value of the first frame image of the first media segment file initially played by each video stream from the index file;

The third module is used to, according to the preset time interval, according to the reference time offset value of the first frame image of the first media segment file initially played by each video stream, the current time of each video stream, and the current time of each video stream. The time of the first frame image of the first media fragment file played initially by the stream determines the reference time offset value of the current frame of each video stream;

The fourth module is used to calculate the maximum reference time offset value, the minimum reference time offset value and the average reference time offset value according to the reference time offset value of the current frame of each video stream according to the preset time interval. The difference between the time offset value and the minimum reference time offset value is used to synchronize the normal playing video stream, or according to the relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream Synchronous operations are performed on the normal playing video stream; synchronous operations include refreshing, fast-forwarding, or pausing.

Referring to Figure 9, an embodiment of the present invention provides a multi-channel video synchronization system, which is applied to the push stream end, including:

The fifth module is used to calculate the network delay time of the current video stream from the streaming end to the streaming server;

The sixth module is used to send the video stream and network delay time to the streaming media server, so that the streaming media server can generate index files and media segment files.

Referring to Fig. 10, an embodiment of the present invention provides a multi-channel video synchronization system applied to a streaming media server, including:

The seventh module is used to receive the video stream and network delay time sent by the streaming end;

The eighth module is used to generate an index file and a media fragment file according to the video stream; wherein, the index file includes the reference time offset value information of the first frame image of each media fragment file, and the media fragment file includes the current video stream The network delay time for streaming to the streaming media server.

Referring to Figure 11, an embodiment of the present invention provides a multi-channel video synchronization device, including:

at least one processor;

at least one memory for storing at least one program;

When at least one program is executed by at least one processor, the at least one processor implements the video synchronization method at any one of the streaming end, the streaming media server, or the playback end.

Wherein, the memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. The memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include remote memory located remotely from the processor, and these remote memories may be connected to the processor through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

In addition, the embodiment of the present application also discloses a computer program product or computer program, and the computer program product or computer program is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above method. Similarly, the content in the above-mentioned method embodiments is applicable to this storage medium embodiment. The functions realized by this storage medium embodiment are the same as those of the above-mentioned method embodiments, and the beneficial effects achieved are the same as those achieved by the above-mentioned method embodiments. The beneficial effects are also the same.

An embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores a processor-executable program, and the processor-executable program is used to implement the above method when executed by the processor .

It can be understood that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware and an appropriate combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit . Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Referring to Fig. 1, an embodiment of the present invention provides a multi-channel video synchronization system, including a streaming media server and a streaming end and a playing end connected to the streaming media server; wherein,

The push stream end is used to execute the video synchronization method of the above push stream end;

The streaming media server is used to implement the video synchronization method of the streaming media server;

The playback terminal is used to implement the video synchronization method of the playback terminal.

Specifically, for the streaming end, it is mainly realized by a video shooting device, and it may specifically include at least one camera and a transmission component; the streaming media server may also be a streaming media server cluster, and the streaming media server includes a processor and a memory; for The playback end is mainly realized by a playback device, and may specifically include at least a player and a processor.

It can be seen that the content in the above-mentioned method embodiments is applicable to this system embodiment, and the functions realized by this system embodiment are the same as those of the above-mentioned method embodiments, and the beneficial effects achieved are also the same as those achieved by the above-mentioned method embodiments. same.

In a specific live shopping application scenario, the stream pusher calculates the network delay time of the current video stream before pushing the video stream, and then the stream pusher sends multiple video streams and network delay times of different sides of the same product to Streaming media server cluster; one or more streaming media servers in the streaming media server cluster receive the multi-channel video streams of the product, respectively obtain the reference time offset value information from the time reference server, and then, according to the multiple video streams, reference time Offset value information and network delay time generate index files and media fragment files, and then send the index files and media fragment files to the streaming end; after the streaming end receives the index files and media fragment files, first, determine the The first media segment file for the initial playback of the video stream, and the reference time offset value of the first frame image of the first media segment file, and then, according to the preset time interval, determine the reference of the current frame of each video stream Time offset value, and according to the reference time offset value of the current frame of each video stream, perform synchronous operations such as refreshing, fast forwarding, or pausing the normally played video stream.

The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the described embodiments, and those skilled in the art can also make various equivalent deformations or replacements without violating the spirit of the present invention. , these equivalent modifications or replacements are all within the scope defined by the claims of the present application.

Claims (13)

1. A multi-channel video synchronization method is characterized in that, being applied to the playback end, comprising: Obtain the index file and the media segment file sent by the streaming media server; wherein, the index file includes the reference time offset value information of the first frame image of each of the media segment files, and the media segment file includes the current The network delay time for the video stream to be pushed from the streaming end to the streaming media server; According to the current time position played by the current video stream, the relative time relationship between the first frame image and the last frame image of each said media segment file, determine the first media segment file of the initial playback of each video stream, and from The index file obtains the reference time offset value of the first frame image of the first media segment file initially played by each video stream; According to the preset time interval, according to the reference time offset value of the first frame image of the first media segment file initially played by each video stream, the current time of each video stream, and the first frame image of each video stream initially played The time of the first frame image of each media segment file determines the reference time offset value of the current frame of each video stream; According to the preset time interval, calculate the maximum reference time offset value, the minimum reference time offset value and the average reference time offset value according to the reference time offset value of the current frame of each video stream, and calculate the maximum reference time offset value according to the maximum reference time offset value. The difference between the offset value and the minimum reference time offset value is used to synchronize the normally played video stream, or according to the difference between the average reference time offset value and the reference time offset value of the current frame of each video stream The relationship performs a synchronous operation on a normally played video stream; the synchronous operation includes refreshing, fast forwarding or pausing. 2. The synchronization method according to claim 1, wherein the relative time relationship between the first frame image and the last frame image of said current time position according to the current video stream playback, each said media segment file Determine the first media segment file for initial playback of each video stream, including: The relative time relationship between the first frame image and the last frame image of each media segment file in each video stream determines the duration of each media segment file in each video stream; Accumulate the duration of each media segment file in each video stream in chronological order to obtain the cumulative time of each media segment file in each video stream; The current time position played by each current video stream is matched with the cumulative time of each video stream to determine the first media segment file initially played by each video stream. 3. The synchronization method according to claim 1, characterized in that, the reference time offset value of the first frame image of the first media segment file initially played according to each video stream, the reference time offset value of each video stream The current time and the time of the first frame image of the first media segment file initially played by each video stream determine the reference time offset value of the current frame of each video stream, specifically including: Calculate the difference between the current time of each video stream and the time of the first frame image of the first media segment file initially played by each video stream; The reference time offset value of the current frame of each video stream is determined according to the difference value and the reference time offset value of the first frame image of the first media segment file initially played by each video stream. 4. The method for synchronizing according to claim 1, characterized in that, the normal play video stream is synchronized according to the difference between the maximum reference time offset value and the minimum reference time offset value, or According to the relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream, the video streams played normally are synchronized, specifically including: For a normally played video stream, if the difference between the maximum reference time offset value and the minimum reference time offset value is greater than a preset threshold, refresh each video stream; If the difference between the maximum reference time offset value and the minimum reference time offset value is less than the preset threshold, and the average reference time offset value is greater than the reference time offset value of the current frame, the path The video stream is fast forwarded to the first preset time period; If the difference between the maximum reference time offset value and the minimum reference time offset value is smaller than the preset threshold, and the average reference time offset value is smaller than the reference time offset value of the current frame, the path The video stream is paused for a second preset time period. 5. The synchronization method according to any one of claims 1-4, wherein the method further comprises: For video streams that are not being played normally, the synchronous operation is suspended until normal playback is resumed; the abnormal playing includes any one of a paused state, a video loading state, or a video loading failure state. 6. A multi-channel video synchronization method, characterized in that it is applied to the streaming end, including: Calculate the network delay time of the current video stream from the streaming end to the streaming server; Send the video stream and the network delay time to the streaming media server, so that the streaming media server generates index files and media fragment files, wherein the index files include the reference time of the first frame image of each media fragment file Offset information. 7. The method according to claim 6, wherein the network delay time is calculated by the following method: Send the first request to the network delay query interface of the streaming media server, receive the feedback information of the network delay query interface, record the first moment of sending the first request and the fourth moment of receiving the feedback information; the feedback information includes The second moment when the first request enters the network delay query interface and the third moment when the first request leaves the network delay query interface; according to the first moment, the fourth moment, and the second moment Calculate the network delay time with the third moment. 8. The method according to claim 7, further comprising: Calculate the network delay time at least 3 times; Remove the maximum and minimum values of at least 3 network delay times to obtain the remaining network delay time; Update the network delay time to the average of the remaining network delay times. 9. A multi-channel video synchronization method, characterized in that it is applied to a streaming media server, comprising: Receive the video stream and network delay time sent by the streaming end; Generate an index file and a media fragment file according to the video stream; wherein, the index file includes the reference time offset value information of the first frame image of each media fragment file, and the media fragment file includes the current video stream The network delay time for streaming to the streaming media server. 10. The method according to claim 9, wherein the reference time offset value information is calculated by the following method: sending a second request to the query interface of the time reference server; Receive the reference time offset value information fed back by the query interface of the time reference server. 11. A multi-channel video synchronization device, characterized in that, comprising: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor is made to implement the method according to any one of claims 1-10. 12. A computer-readable storage medium, wherein a processor-executable program is stored, wherein the processor-executable program is used to perform any one of claims 1-8 when executed by a processor the method described. 13. A multi-channel video synchronization system, characterized in that it includes a streaming media server and a streaming end and a playback end connected to the streaming media server; wherein, The streaming end is used to execute the method according to any one of claims 6-8; The streaming media server is configured to execute the method according to any one of claims 9-10; The playback terminal is configured to execute the method according to any one of claims 1-5.