CN108174164A - A kind of real time video processing method and device - Google Patents

Abstract

The present invention provides a kind of real time video processing method and device, and this method includes：Transmitting terminal acquires current video frame；Detection is currently with the presence or absence of triggering control event；If in the presence of obtaining the corresponding control signaling of triggering control event, control signaling being inserted into current video frame, the current video frame after transmission insertion is to receiving terminal；If being not present, current video frame is directly transmitted to receiving terminal.Receiving terminal receives the current video frame of transmitting terminal transmission；Determine current video frame whether comprising control signaling；If so, current video frame is played according to control signaling；If not, directly play current video frame.Control signaling is inserted into video frame by transmitting terminal in the present invention, and control signaling is made to reach receiving terminal simultaneously with video frame.Realize that Video Rendering is synchronous with control signaling in receiving terminal.The triggering control event triggering that control signaling is actively triggered by user, increases transmitting terminal and the interactive means of receiving terminal two parties, improves the participation and Experience Degree of user in real-time video.

Description

A real-time video processing method and device

technical field

The present invention relates to the technical field of data processing, in particular to a real-time video processing method and device.

Background technique

At present, real-time video transmission is required in various application fields such as monitoring systems and instant messaging. The sending end also needs to transmit control signaling while transmitting real-time video, so that the receiving end can process real-time video according to the control signaling.

Currently, when real-time video is transmitted in the related art, the video stream and control signaling are respectively transmitted through different channels. Transmission through different channels is difficult to ensure that the video stream and its corresponding control signaling arrive at the receiving end at the same time. In order to realize the synchronization of the video stream and the control signaling, a buffer is added in the related technology, and time stamp synchronization is performed, but in a real-time video interaction scenario, a higher time delay will be caused. Moreover, when transmitting video streams and control signaling based on UDP (User Datagram Protocol, user datagram protocol), there must be packet loss. In order to ensure that both the video stream and its corresponding control signaling can reach the receiving end, the program will be increased accordingly. Design and implementation complexity.

Contents of the invention

In view of this, the purpose of the embodiments of the present invention is to provide a real-time video processing method and device to solve the problem in the prior art that the video stream and control signaling are transmitted separately in different channels, resulting in asynchronous video stream and control signaling .

In a first aspect, an embodiment of the present invention provides a real-time video processing method, the method comprising:

Capture the current video frame;

Detect whether there is currently a trigger control event;

If a trigger control event is detected, then acquire the control signaling corresponding to the trigger control event, insert the control signaling into the current video frame, and transmit the inserted current video frame to the receiving end;

If no trigger control event is detected, the current video frame is directly transmitted to the receiving end.

With reference to the first aspect, the embodiment of the present invention provides a first possible implementation manner of the above first aspect, wherein the inserting the control signaling into the current video frame includes:

Analyzing the current video frame, and locating a supplementary enhanced information unit SEI from the current video frame;

inserting the control signaling into the SEI.

In combination with the first aspect, the embodiment of the present invention provides a second possible implementation of the first aspect above, wherein the trigger control event is a preset user behavior; the detecting whether there is currently a trigger control event includes:

identifying a user image contained in the current video frame;

extracting image features of the user image;

According to the image features, determine whether there is a preset user behavior in the user image;

If yes, it is determined that a trigger control event currently exists; if not, it is determined that there is no trigger control event currently.

In combination with the first aspect, the embodiment of the present invention provides a third possible implementation of the first aspect above, wherein the trigger control event is the presence of a sound whose loudness is greater than a preset decibel threshold; the detection of whether there is currently a trigger control event events, including:

While collecting the current video frame, record the audio file corresponding to the current video frame;

judging whether the loudness of the sound contained in the audio file is greater than the preset decibel threshold;

If yes, it is determined that a trigger control event currently exists; if not, it is determined that there is no trigger control event currently.

In a second aspect, an embodiment of the present invention provides a real-time video processing method, the method comprising:

Receive the current video frame transmitted by the sender;

determining whether the current video frame contains control signaling;

If yes, playing the current video frame according to the control signaling;

If not, then directly play the current video frame.

With reference to the second aspect, the embodiment of the present invention provides the first possible implementation of the above second aspect, wherein the determining whether the current video frame contains control signaling includes:

Analyzing the current video frame, and locating the SEI from the current video frame;

Check whether the SEI contains control signaling.

With reference to the second aspect, the embodiment of the present invention provides a second possible implementation manner of the above second aspect, wherein the playing the current video frame according to the control signaling includes:

determining an operation instruction corresponding to the control signaling;

The operation instruction is executed while playing the current video frame.

In a third aspect, an embodiment of the present invention provides a real-time video processing device, the device comprising:

Acquisition module, is used for collecting current video frame;

A detection module, configured to detect whether there is currently a trigger control event;

An inserting module, configured to obtain control signaling corresponding to the triggering control event when the detection module detects the triggering control event, and insert the control signaling into the current video frame;

The transmission module is used to transmit the current video frame after the insertion operation to the receiving end; the transmission module is also used to directly transmit the current video frame to the receiving end when the detection module does not detect a trigger control event.

With reference to the third aspect, the embodiment of the present invention provides a first possible implementation manner of the above third aspect, wherein the plug-in module includes:

A positioning unit, configured to parse the current video frame, and locate a supplementary enhanced information unit SEI from the current video frame;

an inserting unit, configured to insert the control signaling into the SEI.

In a fourth aspect, an embodiment of the present invention provides a real-time video processing device, the device comprising:

The receiving module is used to receive the current video frame transmitted by the sending end;

A determining module, configured to determine whether the current video frame contains control signaling;

A playing module, configured to play the current video frame according to the control signaling when the determining module determines that the current video frame contains control signaling; and, when the determining module determines that the current video frame When the control signaling is not included, the current video frame is played directly.

In the embodiment of the present invention, the sending end collects the current video frame; detects whether there is a trigger control event; if it exists, obtains the control signaling corresponding to the trigger control event, inserts the control signaling into the current video frame, and transmits the inserted current video frame frame to the receiving end; if it does not exist, directly transmit the current video frame to the receiving end. The receiving end receives the current video frame transmitted by the sending end; determines whether the current video frame contains control signaling; if yes, plays the current video frame according to the control signaling; if not, directly plays the current video frame. In the present invention, the sending end inserts the control signaling into the video frame, so that the control signaling and the video frame arrive at the receiving end at the same time. Synchronization between video rendering and control signaling is realized at the receiving end. The control signaling is triggered by the trigger control event triggered by the user, which increases the interaction means between the sending end and the receiving end users, and improves the user's participation and experience in real-time video.

In order to make the above-mentioned objects, features and advantages of the present invention more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and thus It should be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings based on these drawings without creative work.

FIG. 1 shows a flowchart of a real-time video processing method provided by Embodiment 1 of the present invention;

FIG. 2 shows another flowchart of a real-time video processing method provided by Embodiment 1 of the present invention;

FIG. 3 shows a schematic diagram of real-time video transmission provided by Embodiment 1 of the present invention;

FIG. 4 shows a flowchart of a real-time video processing method provided by Embodiment 2 of the present invention;

FIG. 5 shows another flowchart of a real-time video processing method provided by Embodiment 2 of the present invention;

FIG. 6 shows a schematic structural diagram of a real-time video processing device provided by Embodiment 3 of the present invention;

FIG. 7 shows a schematic structural diagram of a real-time video processing device provided by Embodiment 4 of the present invention;

FIG. 8 shows a schematic structural diagram of a real-time video processing device provided by Embodiment 6 of the present invention;

FIG. 9 shows a schematic structural diagram of a real-time video processing device provided by Embodiment 8 of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is a part of embodiments of the present invention, but not all embodiments. The components of the embodiments of the invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present invention.

Considering that when real-time video is transmitted in the related art, the video stream and the control signaling are respectively transmitted through different channels, so that the video stream and the corresponding control signaling are easily out of sync. Moreover, in order to solve the out-of-synchronization problem in the prior art, a buffer is added and time stamp synchronization is performed, which will cause a high time delay. Based on this, embodiments of the present invention provide a real-time video processing method and device, which will be described below through embodiments.

Example 1

Referring to Fig. 1, the embodiment of the present invention provides a kind of real-time video processing method, and this method specifically comprises the following steps:

Step 101: Collect the current video frame.

The executor of the embodiment of the present invention is the real-time video sender. The sender can be a sender or a server. In application scenarios such as live video broadcasting, video monitoring, or video chatting, the sending end captures images of the user or the shooting area in real time, and collects the current video frame of the user or the shooting area.

Step 102: Detect whether there is currently a trigger control event, if yes, execute step 103, if not, execute step 104.

The trigger control event may be a preset user behavior, a sound with a loudness greater than a preset decibel threshold, or a preset moment.

When the trigger control event is a preset user behavior, after the current video frame is collected, image processing is performed on the current video frame to identify the user image contained in the current video frame; the image features of the user image are extracted; according to the image features, the user image is determined Whether there is a preset user behavior; if yes, it is determined that there is a trigger control event, and then the operation of step 103 is performed; if not, it is determined that there is no trigger control event, and the operation of step 104 is subsequently performed.

In the embodiment of the present invention, it is also possible to detect whether there is a preset user behavior in the current video frame in the following manner, specifically including: determining the body part corresponding to the specific user behavior. After collecting the current video frame, perform image processing on the current video frame, extract the user profile from the current video frame, intercept the profile image corresponding to the above limb parts from the user profile, and calculate the relationship between the profile image and the specific user behavior similarity coefficient. If the similarity coefficient is greater than the preset coefficient threshold, it is determined that a trigger control event currently exists. If the similarity coefficient is less than or equal to the preset coefficient threshold, it is determined that there is currently no trigger control event.

While collecting the current video frame, the sending end also records the audio file corresponding to the current video frame. When the trigger control event is a sound whose loudness is greater than the preset decibel threshold, it is judged whether the loudness of the sound contained in the audio file is greater than the preset decibel threshold; if yes, it is determined that there is currently a trigger control event, and the operation of step 103 is subsequently performed; if If not, it is determined that there is no triggering control event at present, and then the operation of step 104 is performed.

When the trigger control event is the preset time, while collecting the current video frame, obtain the system time, judge whether the system time is the above-mentioned preset time, if yes, determine that there is currently a trigger control event, and then perform the operation of step 103. If not, it is determined that there is no triggering control event at present, and then the operation of step 104 is performed.

In addition to the aforementioned preset user behavior, sound louder than the preset decibel threshold, or preset time, the triggering control event may also be other predetermined triggering conditions, for example, the triggering control event may also be the presence of a specific decibel sound. At this time, when detecting whether there is a trigger control event in the current video frame, determine the decibel distribution information of the audio file corresponding to the current video frame, and the decibel distribution information may be a decibel distribution scatter diagram. It is determined whether the above-mentioned specific decibel is included in the decibel distribution information, and if yes, it is determined that there is currently a trigger control event. If not, it is determined that there is no trigger control event currently. The embodiment of the present invention does not specifically limit the triggering control event, and specific triggering conditions for triggering the control event can be specified according to requirements in practical applications.

Step 103: Generate control signaling corresponding to the trigger control event, insert the control signaling into the current video frame, and transmit the inserted current video frame to the receiving end.

In the embodiment of the present invention, the corresponding relationship between the triggering control event and the control signaling is pre-configured in the sending end.

After it is determined that there is a trigger control event in the current video frame, the corresponding control signaling is acquired from the correspondence between the trigger control event and the control signaling. Insert the obtained control signaling into the current video frame.

In the embodiment of the present invention, the control signaling is inserted into the current video frame through the following operations, specifically including:

Parsing the current video frame, locating SEI (Supplemental Enhancement Information, Supplemental Enhancement Information Unit) from the current video frame; inserting control signaling into the SEI of the current video frame.

After the control signaling is inserted into the current video frame, the current video frame is transmitted to the receiving end. After receiving the video frame, the receiving end parses the control signaling from the current video frame, and executes the operation instruction corresponding to the control signaling when playing the current video frame.

In this embodiment of the present invention, the control signaling may be a predetermined signaling identifier, such as 0 or 1. For example, the corresponding relationship between the trigger control event "waving the right hand" and the control signaling "1" is configured in advance in the sending end, and when the image of the user in the current video frame is detected waving the right hand, the control signaling is inserted into the SEI of the current video frame "1", and then transmit the current video frame to the receiver. The corresponding relationship between the control signaling and the operation command is configured in the receiving end in advance. Assuming that the corresponding relationship between the control signaling "1" and the operating command "window vibration" is configured in the receiving end, the receiving end recognizes that the current video frame contains the control When the signal is "1", the window playing the current video frame will vibrate.

When the current video frame inserted with control signaling is transmitted to the receiving end, in order to reduce the loss of control signaling due to packet loss during transmission, the embodiment of the present invention adds a packet loss retransmission strategy during video transmission. Enable reliable UDP (User Datagram Protocol, User Datagram Transmission Protocol)-based transmission for the current video frame inserted into the control signaling, for example, transmit the current video frame inserted into the control signaling through KCP (Quick and Reliable Protocol).

When transmitting the current video frame inserted with control signaling through the KCP protocol, the sender inserts the control signaling into the current video frame, caches the current video frame, and assigns a unique identifier to the current video frame, and then passes the KCP protocol to the current video frame The current video frame and its unique identifier are transmitted to the receiving end. After receiving the current video frame and the unique identifier, the receiving end returns the unique identifier to the sending end. When the sending end receives the unique identifier, it confirms that the current video frame is successfully sent, and continues to process subsequent video frames. If the sending end does not receive the unique identifier returned by the receiving end within a period of time after sending the current video frame and its unique identifier to the receiving end, then it is confirmed that the current video frame is lost, and the sending end obtains the cached current video frame, through The KCP protocol resends the current video frame and its unique identifier to the receiving end. The sending end deletes the buffered current video frame until it is confirmed that the current video frame is successfully sent to the receiving end, and then continues to process subsequent video frames.

Step 104: Directly transmit the current video frame to the receiving end.

When step 102 detects that there is currently no trigger control event, the current video frame is directly transmitted to the receiving end. After receiving the current video frame, the receiving end finds that the current video frame does not contain control signaling through analysis, and then directly plays the current video frame.

When the current video frame does not contain control signaling, the sending end can directly transmit the current video frame to the receiving end through the unreliable UDP protocol, so as to increase the transmission speed of the video frame. Of course, the sending end can also transmit the current video frame without inserting the control signaling through the reliable UDP protocol.

In order to facilitate understanding of the real-time video processing solutions provided by the embodiments of the present invention, the following will be described in detail with reference to the accompanying drawings. As shown in FIG. 2, S1: collect the current video frame. S2: Encode the current video frame. S3: Write the control signaling into the SEI of the current video frame. S4: Determine whether the current video frame contains control signaling, if yes, execute step S5, if not, execute step S6. S5: Enable the packet loss retransmission mechanism, and use reliable UDP protocols such as the KCP protocol to transmit the current video frame. S6: Using the unreliable UDP protocol to transmit the current video frame.

As each frame of the real-time video is the same as the current video frame, the sending end processes each frame of the real-time video according to the operations of steps 101-104, and transmits it to the receiving end. FIG. 3 is a schematic diagram of real-time video transmission. In the real-time video, some video frames carry control signaling, and some video frames do not contain control signaling. In the video frame carrying the control signaling, the control signaling is located in the SEI of the video frame. In the embodiment of the present invention, the control signaling is inserted into the video frame, so that the control signaling can reach the receiving end at the same time as the video frame, and the synchronization of video rendering and control signaling is realized. In addition, the trigger control event can be actively triggered by the user, which improves the user's participation in real-time video and improves user experience.

In the embodiment of the present invention, the sending end collects the current video frame; detects whether there is a trigger control event; if it exists, obtains the control signaling corresponding to the trigger control event, inserts the control signaling into the current video frame, and transmits the inserted current video frame frame to the receiving end; if it does not exist, directly transmit the current video frame to the receiving end. In the present invention, the sending end inserts the control signaling into the video frame, so that the control signaling and the video frame arrive at the receiving end at the same time. Synchronization between video rendering and control signaling is realized at the receiving end. The control signaling is triggered by the trigger control event triggered by the user, which increases the interaction means between the sending end and the receiving end users, and improves the user's participation and experience in real-time video.

Example 2

Referring to Fig. 4, an embodiment of the present invention provides a real-time video processing method, which specifically includes the following steps:

Step 201: Receive the current video frame transmitted by the sending end.

The execution subject of the embodiment of the present invention is the receiving end of the real-time video. The receiving end receives the current video frame transmitted by the sending end.

Step 202: Determine whether the current video frame contains control signaling, if yes, execute step 203, if not, execute step 204.

After receiving the current video frame, the receiving end parses the current video frame and locates the SEI from the current video frame. Check whether the SEI of the current video frame contains control signaling. If it is included, step 203 will be subsequently performed, and if it is not included, then step 204 will be performed.

Step 203: Play the current video frame according to the control signal.

When it is determined in step 202 that the current video frame contains control signaling, an operation instruction corresponding to the control signaling is determined; and the operation instruction is executed while playing the current video frame.

In the embodiment of the present invention, the corresponding relationship between the control signaling and the operation instruction is pre-configured in the receiving end. When it is determined that the current video frame contains the control signaling, the corresponding operation instruction is obtained from the correspondence between the control signaling and the operation instruction, and the operation instruction is executed when the current video frame is played.

For example, assuming that the corresponding relationship between the control signaling "1" and the operation command "window vibration" is pre-configured in the receiving end, when it is determined that the current video frame contains the control signaling "1", when the current video frame is played through the playback window At the same time, vibrate the playback window.

In the above-mentioned embodiment 1, when sending the current video frame for inserting control signaling, in order to reduce the loss of control signaling due to packet loss in the transmission process, the sending end enables the packet loss retransmission strategy, that is, inserting control signaling The current video frame of the command enables the transmission based on the reliable UDP protocol, for example, the current video frame inserted into the control signaling is transmitted through the KCP protocol. When the sending end transmits the current video frame to the receiving end through the KCP protocol, it also transmits the unique identifier of the current video frame to the receiving end. After receiving the current video frame and the unique identifier, the receiving end returns the unique identifier to the sending end, so that the sending end confirms that the current video frame is successfully sent when receiving the unique identifier, and continues to process subsequent video frames. If packet loss occurs in the transmission of the current video frame, the receiving end cannot receive the current video frame and its unique identifier. If the sending end does not receive the unique identifier returned by the receiving end after a certain period of time, it will confirm that the current video frame is lost. package, the sender will retransmit the current video frame until it is confirmed that the current video frame is successfully sent to the receiver.

Step 204: Play the current video frame directly.

After receiving the current video frame, the receiving end finds that the current video frame does not contain control signaling through analysis, and then directly plays the current video frame.

In order to facilitate understanding of the real-time video processing method provided by the embodiment of the present invention, the following description will be made with reference to the accompanying drawings. As shown in FIG. 5, A1: analyze the current video frame transmitted by the sending end. A2: Determine whether the current video frame contains SEI, if yes, execute step A3, if not, execute step A6. A3: Determine whether the SEI of the current video frame contains control signaling, if yes, execute step A4, if not, execute step A5. A4: Return the control signaling to the business layer for processing, and execute the operation instructions corresponding to the control signaling at the business layer. A5: Display the current video frame.

As each frame of the real-time video is the same as the current video frame, the receiving end processes and displays each frame of the real-time video according to the operations of steps 201-204. The sending end inserts the control signaling into the video frame, so that the control signaling can reach the receiving end at the same time as the video frame, and the synchronization of video rendering and control signaling is realized at the receiving end. In addition, the trigger control event can be actively triggered by the user, which improves the user's participation in real-time video and improves user experience.

In the embodiment of the present invention, the receiving end receives the current video frame transmitted by the sending end; determines whether the current video frame contains control signaling; if yes, plays the current video frame according to the control signaling; if not, directly plays the current video frame. In the present invention, the sending end inserts the control signaling into the video frame, so that the control signaling and the video frame arrive at the receiving end at the same time. Synchronization between video rendering and control signaling is realized at the receiving end. The control signaling is triggered by the trigger control event triggered by the user, which increases the interaction means between the sending end and the receiving end users, and improves the user's participation and experience in real-time video.

Example 3

Referring to FIG. 6, an embodiment of the present invention provides a real-time video processing device, which is used to execute the real-time video processing method provided in Embodiment 1 above, and the device includes:

Acquisition module 30, is used for collecting current video frame;

A detection module 31, configured to detect whether there is currently a trigger control event;

The insertion module 32 is used to obtain the control signaling corresponding to the trigger control event when the detection module detects the trigger control event, and insert the control signaling into the current video frame;

The transmission module 33 is used to transmit the current video frame after the insertion operation to the receiving end; the transmission module 33 is also used to directly transmit the current video frame to the receiving end when the detection module 31 does not detect a trigger control event.

The above-mentioned plug-in modules 32 include:

The positioning unit is used to analyze the current video frame, and locate the supplementary enhanced information unit SEI from the current video frame;

The insertion unit is configured to insert the control signaling into the SEI.

In the embodiment of the present invention, the trigger control event is a preset user behavior; at this time, the detection module 31 is used to identify the user image contained in the current video frame; extract the image features of the user image; determine whether there is a user image in the user image according to the image feature Preset user behavior; if yes, determine that there is currently a trigger control event; if not, determine that there is currently no trigger control event.

In the embodiment of the present invention, the trigger control event can also be a sound whose loudness is greater than the preset decibel threshold; at this time, the detection module 31 is used to record the audio file corresponding to the current video frame while collecting the current video frame; determine the audio file Whether the loudness of the contained sound is greater than a preset decibel threshold; if yes, it is determined that a trigger control event currently exists; if not, it is determined that there is no trigger control event currently.

In the embodiment of the present invention, the sending end collects the current video frame; detects whether there is a trigger control event; if it exists, obtains the control signaling corresponding to the trigger control event, inserts the control signaling into the current video frame, and transmits the inserted current video frame frame to the receiving end; if it does not exist, directly transmit the current video frame to the receiving end. In the present invention, the sending end inserts the control signaling into the video frame, so that the control signaling and the video frame arrive at the receiving end at the same time. Synchronization between video rendering and control signaling is realized at the receiving end. The control signaling is triggered by the trigger control event triggered by the user, which increases the interaction means between the sending end and the receiving end users, and improves the user's participation and experience in real-time video.

Example 4

Referring to FIG. 7, an embodiment of the present invention provides a real-time video processing device, which is used to execute the real-time video processing method provided in Embodiment 2 above, and the device includes:

The receiving module 40 is used to receive the current video frame transmitted by the sending end;

Determining module 41, is used for determining whether current video frame comprises control signaling;

The playing module 42 is used for playing the current video frame according to the control signaling when the determining module 41 determines that the current video frame contains control signaling; and for directly playing when the determining module 41 determines that the current video frame does not contain the control signaling The current video frame.

The determination module 41 is configured to analyze the current video frame, locate the SEI from the current video frame, and check whether the SEI contains control signaling.

When the current video frame contains the control signaling, the playing module 42 is configured to determine the operation instruction corresponding to the control signaling; and execute the operation instruction while playing the current video frame. When it is determined that the current video frame does not contain control signaling, the current video frame is played directly.

In the embodiment of the present invention, the receiving end receives the current video frame transmitted by the sending end; determines whether the current video frame contains control signaling; if yes, plays the current video frame according to the control signaling; if not, directly plays the current video frame. In the present invention, the sending end inserts the control signaling into the video frame, so that the control signaling and the video frame arrive at the receiving end at the same time. Synchronization between video rendering and control signaling is realized at the receiving end. The control signaling is triggered by the trigger control event triggered by the user, which increases the interaction means between the sending end and the receiving end users, and improves the user's participation and experience in real-time video.

Example 5

An embodiment of the present invention provides a non-volatile computer storage medium, the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the real-time video processing method in Embodiment 1 above, and specifically execute the following methods: Collect the current video frame; detect whether there is a trigger control event; if a trigger control event is detected, obtain the control signaling corresponding to the trigger control event, insert the control signaling into the current video frame, and transmit the inserted current video frame to The receiving end; if no trigger control event is detected, the current video frame is directly transmitted to the receiving end.

When the computer-executable instructions stored in the computer storage medium are executed and it is determined that there is a trigger control event in the current video frame, the control signaling corresponding to the trigger control event is inserted into the current video frame, so that the control signaling and the current video frame arrive at the receiving end at the same time , realizing the synchronization of video rendering and control signaling. In addition, the trigger control event can be actively triggered by the user, which improves the user's participation in real-time video and improves user experience.

Example 6

As shown in FIG. 8 , an embodiment of the present invention provides a real-time video processing device. The real-time video processing device 60 includes: a processor 61, a memory 62 and a bus 63. The memory 62 stores execution instructions. When the device is running, Communication between the processor 61 and the memory 62 via the bus 63, the processor 61 executes the following execution instructions stored in the memory 62: collect the current video frame; detect whether there is a trigger control event; if a trigger control event is detected, then obtain the trigger control For the control signaling corresponding to the event, the control signaling is inserted into the current video frame, and the current video frame after the insertion operation is transmitted to the receiving end; if no trigger control event is detected, the current video frame is directly transmitted to the receiving end.

The real-time video processing device executes the above instructions through the processor 61, and when it is determined that there is a trigger control event in the current video frame, the control signaling corresponding to the trigger control event is inserted into the current video frame, so that the control signaling and the current video frame arrive at the receiving end at the same time. terminal to realize the synchronization of video rendering and control signaling. In addition, the trigger control event can be actively triggered by the user, which improves the user's participation in real-time video and improves user experience.

Example 7

An embodiment of the present invention provides a non-volatile computer storage medium, the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the real-time video processing method in Embodiment 2 above, and specifically execute the following methods: Receive the current video frame transmitted by the sending end; determine whether the current video frame contains control signaling; if yes, play the current video frame according to the control signal; if not, directly play the current video frame.

After the computer-executable instructions stored in the computer storage medium are executed, when the sending end determines that there is a trigger control event in the current video frame, it inserts the control signaling corresponding to the trigger control event into the current video frame, and inserts the current video of the control signaling into the current video frame. The frame is transmitted to the receiver. The control signaling and the current video frame arrive at the receiving end at the same time, and the synchronization of video rendering and control signaling is realized at the receiving end. In addition, the trigger control event can be actively triggered by the user, which improves the user's participation in real-time video and improves user experience.

Example 8

As shown in FIG. 9 , the embodiment of the present invention provides a real-time video processing device. The real-time video processing device 80 includes: a processor 81, a memory 82 and a bus 83. The memory 82 stores execution instructions. When the device is running, The processor 81 communicates with the memory 82 via the bus 83, and the processor 81 executes the following execution instructions stored in the memory 82: receiving the current video frame transmitted by the sending end; determining whether the current video frame contains control signaling; if so, according to The control signaling plays the current video frame; if not, the current video frame is played directly.

The real-time video processing device executes the above-mentioned instructions through the processor 81. When the sending end determines that there is a trigger control event in the current video frame, it inserts the control signaling corresponding to the trigger control event into the current video frame, and inserts the current video of the control signaling into the current video frame. The frame is transmitted to the receiver. The control signaling and the current video frame arrive at the receiving end at the same time, and the synchronization of video rendering and control signaling is realized at the receiving end. In addition, the trigger control event can be actively triggered by the user, which improves the user's participation in real-time video and improves user experience.

The real-time video processing apparatus provided in the embodiment of the present invention may be specific hardware on the device or software or firmware installed on the device. The implementation principles and technical effects of the device provided by the embodiment of the present invention are the same as those of the foregoing method embodiment. For brief description, for the parts not mentioned in the device embodiment, reference may be made to the corresponding content in the foregoing method embodiment. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the above-mentioned method embodiments, and will not be repeated here.

In the embodiments provided in the present invention, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments provided by the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .

It should be noted that like numerals and letters denote similar items in the following drawings, therefore, once an item is defined in one drawing, it does not require further definition and explanation in subsequent drawings, In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions, and should not be construed as indicating or implying relative importance.

Finally, it should be noted that: the above-described embodiments are only specific implementations of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting them, and the scope of protection of the present invention is not limited thereto, although referring to the foregoing The embodiment has described the present invention in detail, and those skilled in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention Changes can be easily imagined, or equivalent replacements can be made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (10)

1. A real-time video processing method, characterized in that the method comprises: Capture the current video frame; Detect whether there is currently a trigger control event; If a trigger control event is detected, then acquire the control signaling corresponding to the trigger control event, insert the control signaling into the current video frame, and transmit the inserted current video frame to the receiving end; If no trigger control event is detected, the current video frame is directly transmitted to the receiving end. 2. The method according to claim 1, wherein the inserting the control signaling into the current video frame comprises: Analyzing the current video frame, and locating a supplementary enhanced information unit SEI from the current video frame; inserting the control signaling into the SEI. 3. The method according to claim 1, wherein the trigger control event is a preset user behavior; and the detecting whether there is currently a trigger control event comprises: identifying a user image contained in the current video frame; extracting image features of the user image; According to the image features, determine whether there is a preset user behavior in the user image; If yes, it is determined that a trigger control event currently exists; if not, it is determined that there is no trigger control event currently. 4. The method according to claim 1, wherein the trigger control event is a sound whose loudness is greater than a preset decibel threshold; the detection of whether there is currently a trigger control event comprises: While collecting the current video frame, record the audio file corresponding to the current video frame; judging whether the loudness of the sound contained in the audio file is greater than the preset decibel threshold; If yes, it is determined that a trigger control event currently exists; if not, it is determined that there is no trigger control event currently. 5. A real-time video processing method, characterized in that the method comprises: Receive the current video frame transmitted by the sender; determining whether the current video frame contains control signaling; If yes, playing the current video frame according to the control signaling; If not, then directly play the current video frame. 6. The method according to claim 5, wherein the determining whether the current video frame contains control signaling comprises: Analyzing the current video frame, and locating the SEI from the current video frame; Check whether the SEI contains control signaling. 7. The method according to claim 5, wherein the playing the current video frame according to the control signaling comprises: determining an operation instruction corresponding to the control signaling; The operation instruction is executed while playing the current video frame. 8. A real-time video processing device, characterized in that the device comprises: Acquisition module, is used for collecting current video frame; A detection module, configured to detect whether there is currently a trigger control event; An inserting module, configured to obtain control signaling corresponding to the triggering control event when the detection module detects the triggering control event, and insert the control signaling into the current video frame; The transmission module is used to transmit the current video frame after the insertion operation to the receiving end; the transmission module is also used to directly transmit the current video frame to the receiving end when the detection module does not detect a trigger control event. 9. The device of claim 8, wherein the plug-in module comprises: A positioning unit, configured to parse the current video frame, and locate a supplementary enhanced information unit SEI from the current video frame; an inserting unit, configured to insert the control signaling into the SEI. 10. A real-time video processing device, characterized in that the device comprises: The receiving module is used to receive the current video frame transmitted by the sending end; A determining module, configured to determine whether the current video frame contains control signaling; A playing module, configured to play the current video frame according to the control signaling when the determining module determines that the current video frame contains control signaling; and, when the determining module determines that the current video frame When the control signaling is not included, the current video frame is played directly.