CN108111911A - Video data real-time processing method and device based on the segmentation of adaptive tracing frame - Google Patents

Abstract

The invention discloses a video data real-time processing method, device, computing device and computer storage medium based on adaptive tracking frame segmentation. The tracking frame corresponding to the t-1 frame image; according to the t-th frame image, the tracking frame corresponding to the t-1 frame image is adjusted to obtain the tracking frame corresponding to the t-frame image; according to the t-th frame image corresponding The tracking frame of the tth frame image is subjected to scene segmentation processing on the partial area of the tth frame image, and the segmentation result corresponding to the tth frame image is obtained; according to the segmentation result, the second foreground image of the tth frame image is determined; according to the second foreground image, adding Personalize special effects to obtain the processed t-th frame of image; cover the processed t-th frame of image with the t-th frame of image to obtain processed video data; display the processed video data. This technical solution can add personalized special effects to frame images more accurately and quickly.

Description

Video data real-time processing method and device based on adaptive tracking frame segmentation

technical field

The present invention relates to the technical field of image processing, in particular to a video data real-time processing method, device, computing device and computer storage medium based on adaptive tracking frame segmentation.

Background technique

In the prior art, when the user needs to perform personalized processing such as changing the background of the video, adding special effects, etc., the image segmentation method is often used to perform scene segmentation processing on the frame images in the video. Among them, the image segmentation method based on deep learning is adopted The segmentation effect at the pixel level can be achieved. However, when the existing image segmentation method performs scene segmentation processing, it is necessary to perform scene segmentation processing on all the contents of the frame image, the amount of data processing is large, and the processing efficiency is low; in addition, the existing image segmentation method is performing scene segmentation When processing, the proportion of the foreground image in the frame image is not considered, so when the proportion of the foreground image in the frame image is small, it is easy to divide the pixels that actually belong to the edge of the foreground image by using the existing image segmentation method is the background image, the resulting segmentation results have low segmentation accuracy and poor segmentation effect. Therefore, the image segmentation method in the prior art has the problems of a large amount of data processing for image scene segmentation, low processing efficiency and low segmentation accuracy, so the obtained segmentation results cannot be used to accurately and accurately classify the frames in the video. Add personalized special effects to the image, and the display effect of the obtained processed video data is relatively poor.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a method, device, computing device and computer storage medium for real-time processing of video data based on adaptive tracking frame segmentation to overcome the above problems or at least partially solve the above problems.

According to one aspect of the present invention, a method for real-time processing of video data based on adaptive tracking frame segmentation is provided, the method is used to process each group of frame images obtained by dividing every n frames in the video, and for one group of frames image, the method includes:

Obtain the tth frame image containing a specific object in a set of frame images and the tracking frame corresponding to the t-1th frame image, where t is greater than 1; the tracking frame corresponding to the first frame image is based on the corresponding to the first frame image Determined by the segmentation results;

According to the tth frame image, adjust the tracking frame corresponding to the t-1th frame image to obtain the tracking frame corresponding to the tth frame image; according to the tracking frame corresponding to the tth frame image, adjust the tth frame image Part of the area is subjected to scene segmentation processing to obtain a segmentation result corresponding to the image of the tth frame;

According to the segmentation result corresponding to the tth frame image, determine the second foreground image of the tth frame image;

Adding personalized special effects according to the second foreground image to obtain the processed t-th frame image;

Covering the processed t-th frame image with the t-th frame image to obtain processed video data;

Display the processed video data.

Further, according to the second foreground image, adding personalized special effects, the processed t-th frame image further includes:

Extract key information of the area to be processed from the second foreground image;

According to the key information, draw the effect map;

Fusing the effect map, the second foreground image and the preset background image to obtain the processed image of the tth frame; The two background images are fused to obtain the processed t-th frame image.

Further, the key information is key point information; drawing the effect map according to the key information further includes:

Find the basic effect texture corresponding to the key point information; or, obtain the basic effect texture specified by the user;

Calculating position information between at least two key points having a symmetrical relationship according to the key point information;

According to the location information, the basic effect map is processed to obtain the effect map.

Further, according to the second foreground image, adding personalized special effects, the processed t-th frame image further includes:

extracting key information of the area to be identified from the second foreground image;

According to the key information, the posture of the specific object is recognized, and the posture recognition result of the specific object is obtained;

According to the posture recognition result of the specific object, determine the corresponding effect processing command to be responded to the t-th frame of image, and obtain the processed t-th frame of image.

Further, according to the gesture recognition result of the specific object, determine the corresponding effect processing command to be responded to the t-th frame image, and obtain the processed t-th frame image further comprising:

According to the posture recognition result of the specific object and the interaction information with the interactive object contained in the t-th frame of image, determine the corresponding effect processing command to be responded to the t-th frame of image, and obtain the processed t-th frame of image.

Further, the effect processing command to be responded to includes an effect map processing command, a stylization processing command, a brightness processing command, a lighting processing command and/or a tone processing command.

Further, according to the tth frame image, adjusting the tracking frame corresponding to the t-1th frame image further includes:

Perform recognition processing on the tth frame image, and determine the first foreground image for a specific object in the tth frame image;

Apply the tracking frame corresponding to the t-1th frame image to the tth frame image;

According to the first foreground image in the tth frame image, the tracking frame corresponding to the t-1th frame image is adjusted.

Further, according to the first foreground image in the tth frame image, adjusting the tracking frame corresponding to the t-1th frame image further includes:

Calculate the ratio of the pixels belonging to the first foreground image in the tth frame image to all the pixels in the tracking frame corresponding to the t-1th frame image, and determine the ratio as the first foreground pixel ratio of the tth frame image ;

Obtain the second foreground pixel ratio of the t-1th frame image, wherein the second foreground pixel ratio of the t-1th frame image is the pixel point belonging to the first foreground image in the t-1th frame image and the t-1th frame image The proportion of all pixels in the tracking frame corresponding to the frame image;

Calculate the difference value between the first foreground pixel ratio of the tth frame image and the second foreground ratio of the t-1th frame image;

Judging whether the difference value exceeds a preset difference threshold; if so, adjusting the size of the tracking frame corresponding to the t-1th frame image according to the difference value.

Further, according to the first foreground image in the tth frame image, adjusting the tracking frame corresponding to the t-1th frame image further includes:

Calculate the distance between the first foreground image in the t-th frame image and the distance of each border of the tracking frame corresponding to the t-1-th frame image;

According to the distance and the preset distance threshold, the size of the tracking frame corresponding to the t-1th frame image is adjusted.

Further, according to the first foreground image in the tth frame image, adjusting the tracking frame corresponding to the t-1th frame image further includes:

According to the first foreground image in the tth frame image, determine the center point position of the first foreground image in the tth frame image;

According to the position of the center point of the first foreground image in the tth frame image, the position of the tracking frame corresponding to the t-1th frame image is adjusted so that the center point of the tracking frame corresponding to the t-1th frame image The position coincides with the center point position of the first foreground image in the tth frame image.

Further, according to the tracking frame corresponding to the t-th frame image, perform scene segmentation processing on the partial area of the t-th frame image, and obtain the segmentation result corresponding to the t-th frame image, which further includes:

According to the tracking frame corresponding to the tth frame image, extract the image to be segmented from the partial area of the tth frame image;

Scene segmentation processing is performed on the image to be segmented to obtain a segmentation result corresponding to the image to be segmented;

According to the segmentation result corresponding to the image to be segmented, the segmentation result corresponding to the t-th frame image is obtained.

Further, according to the tracking frame corresponding to the tth frame image, extracting the image to be segmented from the partial area of the tth frame image further includes:

Extract the image in the tracking frame corresponding to the t-th frame image from the t-th frame image, and determine the extracted image as the image to be segmented.

Further, performing scene segmentation processing on the image to be segmented, and obtaining a segmentation result corresponding to the image to be segmented further includes:

The image to be segmented is input into the scene segmentation network, and the segmentation result corresponding to the image to be segmented is obtained.

Further, displaying the processed video data further includes: displaying the processed video data in real time;

The method also includes: uploading the processed video data to a cloud server.

Further, uploading the processed video data to the cloud server further includes:

Upload the processed video data to the cloud video platform server for the cloud video platform server to display the video data on the cloud video platform.

Further, uploading the processed video data to the cloud server further includes:

Upload the processed video data to the cloud live broadcast server, so that the cloud live broadcast server can push the video data to the viewing user client in real time.

Further, uploading the processed video data to the cloud server further includes:

Upload the processed video data to the cloud official account server, so that the cloud official account server can push the video data to the official account follower client.

According to another aspect of the present invention, a video traversal processing device based on adaptive tracking frame segmentation is provided, the device is used to process each group of frame images obtained by dividing every n frames in the video, the device includes:

The acquisition module is adapted to acquire the tth frame image containing a specific object in a group of frame images and the tracking frame corresponding to the t-1th frame image, wherein t is greater than 1; the tracking frame corresponding to the first frame image is based on the Determined by the segmentation result corresponding to the first frame image;

The segmentation module is adapted to adjust the tracking frame corresponding to the t-1th frame image according to the tth frame image to obtain the tracking frame corresponding to the tth frame image; according to the tracking frame corresponding to the tth frame image, the Scene segmentation processing is performed on a part of the t-th frame image to obtain a segmentation result corresponding to the t-th frame image;

The determination module is adapted to determine the second foreground image of the t-th frame image according to the segmentation result corresponding to the t-th frame image;

The processing module is adapted to add personalized special effects according to the second foreground image to obtain the processed t-th frame image;

An overlay module, adapted to cover the processed t-th frame image with the processed t-frame image to obtain processed video data;

The display module is suitable for displaying the processed video data.

Further, the processing module is further adapted to:

Extract key information of the area to be processed from the second foreground image;

According to the key information, draw the effect map;

Fusing the effect map, the second foreground image and the preset background image to obtain the processed image of the tth frame; The two background images are fused to obtain the processed t-th frame image.

Further, the key information is key point information; the processing module is further suitable for:

Find the basic effect texture corresponding to the key point information; or, obtain the basic effect texture specified by the user;

Calculating position information between at least two key points having a symmetrical relationship according to the key point information;

According to the location information, the basic effect map is processed to obtain the effect map.

Further, the processing module is further adapted to:

extracting key information of the area to be identified from the second foreground image;

According to the key information, the posture of the specific object is recognized, and the posture recognition result of the specific object is obtained;

According to the posture recognition result of the specific object, determine the corresponding effect processing command to be responded to the t-th frame of image, and obtain the processed t-th frame of image.

Further, the processing module is further adapted to:

According to the posture recognition result of the specific object and the interaction information with the interactive object contained in the t-th frame of image, determine the corresponding effect processing command to be responded to the t-th frame of image, and obtain the processed t-th frame of image.

Further, the effect processing command to be responded to includes an effect map processing command, a stylization processing command, a brightness processing command, a lighting processing command and/or a tone processing command.

Further, the segmentation module is further adapted to:

Perform recognition processing on the tth frame image, and determine the first foreground image for a specific object in the tth frame image;

Apply the tracking frame corresponding to the t-1th frame image to the tth frame image;

According to the first foreground image in the tth frame image, the tracking frame corresponding to the t-1th frame image is adjusted.

Further, the segmentation module is further adapted to:

Calculate the ratio of the pixels belonging to the first foreground image in the tth frame image to all the pixels in the tracking frame corresponding to the t-1th frame image, and determine the ratio as the first foreground pixel ratio of the tth frame image ;

Obtain the second foreground pixel ratio of the t-1th frame image, wherein the second foreground pixel ratio of the t-1th frame image is the pixel point belonging to the first foreground image in the t-1th frame image and the t-1th frame image The proportion of all pixels in the tracking frame corresponding to the frame image;

Calculate the difference value between the first foreground pixel ratio of the tth frame image and the second foreground ratio of the t-1th frame image;

Judging whether the difference value exceeds a preset difference threshold; if so, adjusting the size of the tracking frame corresponding to the t-1th frame image according to the difference value.

Further, the segmentation module is further adapted to:

Calculate the distance between the first foreground image in the t-th frame image and the distance of each border of the tracking frame corresponding to the t-1-th frame image;

According to the distance and the preset distance threshold, the size of the tracking frame corresponding to the t-1th frame image is adjusted.

Further, the segmentation module is further adapted to:

According to the first foreground image in the tth frame image, determine the center point position of the first foreground image in the tth frame image;

According to the position of the center point of the first foreground image in the tth frame image, the position of the tracking frame corresponding to the t-1th frame image is adjusted so that the center point of the tracking frame corresponding to the t-1th frame image The position coincides with the center point position of the first foreground image in the tth frame image.

Further, the segmentation module is further adapted to:

According to the tracking frame corresponding to the tth frame image, the image to be segmented is extracted from the partial area of the tth frame image;

Scene segmentation processing is performed on the image to be segmented to obtain a segmentation result corresponding to the image to be segmented;

According to the segmentation result corresponding to the image to be segmented, the segmentation result corresponding to the t-th frame image is obtained.

Further, the segmentation module is further adapted to:

Extract the image in the tracking frame corresponding to the t-th frame image from the t-th frame image, and determine the extracted image as the image to be segmented.

Further, the segmentation module is further adapted to:

The image to be segmented is input into the scene segmentation network, and the segmentation result corresponding to the image to be segmented is obtained.

Further, the display module is further adapted to: display the processed video data in real time;

The device also includes: an upload module, suitable for uploading the processed video data to a cloud server.

Further, the upload module is further adapted to:

Upload the processed video data to the cloud video platform server for the cloud video platform server to display the video data on the cloud video platform.

Further, the upload module is further adapted to:

Upload the processed video data to the cloud live broadcast server, so that the cloud live broadcast server can push the video data to the viewing user client in real time.

Further, the upload module is further adapted to:

Upload the processed video data to the cloud official account server, so that the cloud official account server can push the video data to the official account follower client.

According to yet another aspect of the present invention, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete mutual communication through the communication bus;

The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the operations corresponding to the above video data real-time processing method based on adaptive tracking frame segmentation.

According to still another aspect of the present invention, a computer storage medium is provided, and at least one executable instruction is stored in the storage medium, and the executable instruction causes the processor to perform the above-mentioned method corresponding to the video data real-time processing method based on adaptive tracking frame segmentation operate.

According to the technical solution provided by the present invention, for each group of frame images, the tracking frame corresponding to the t-th frame image is obtained based on the tracking frame corresponding to the t-1th frame image, and the scene segmentation is performed on the t-th frame image using the tracking frame , can quickly and accurately obtain the segmentation result corresponding to the t-th frame image, effectively improving the segmentation accuracy of the image scene segmentation. Compared with performing scene segmentation processing on all contents of the frame image in the prior art, the present invention only performs scene segmentation processing on a part of the frame image, which effectively reduces the amount of data processing for image scene segmentation and improves processing efficiency. The image scene segmentation processing method is optimized; and based on the obtained segmentation results, personalized special effects can be added to the frame image more accurately and quickly, and the video data display effect is beautified.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same parts. In the attached picture:

FIG. 1 shows a schematic flow diagram of a method for real-time processing of video data based on adaptive tracking frame segmentation according to an embodiment of the present invention;

2 shows a schematic flow diagram of a method for real-time processing of video data based on adaptive tracking frame segmentation according to another embodiment of the present invention;

FIG. 3 shows a block diagram of a real-time processing device for video data based on adaptive tracking frame segmentation according to an embodiment of the present invention;

Fig. 4 shows a schematic structural diagram of a computing device according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

The present invention provides a real-time processing method of video data based on adaptive tracking frame segmentation, considering that during the process of video shooting or video recording, the number of specific objects photographed or recorded may change due to movement and other reasons. Changes, taking a specific object as a human body as an example, the number of photographed or recorded human bodies may increase or decrease. In order to quickly and accurately perform scene segmentation processing on the frame images in the video, this method divides every n frames in the video Each group of frame images obtained by division is processed, and for each group of frame images, based on the tracking frame corresponding to the t-1th frame image, the tracking frame corresponding to the t-th frame image is obtained, and the t-th frame image is tracked using the tracking frame Perform scene segmentation. In the present invention, the foreground image may only contain specific objects, and the background image is an image other than the foreground image in the frame image. In order to distinguish the foreground image in the frame image before the segmentation processing from the foreground image in the frame image after the segmentation processing, in the present invention, the foreground image in the frame image before the segmentation processing is referred to as the first foreground image, the foreground image in the segmented frame image is referred to as the second foreground image. Similarly, the background image in the frame image before the segmentation processing is called the first background image, and the background image in the frame image after the segmentation processing is called the second background image. Wherein, the tracking frame can be a rectangular frame, which is used to frame the first foreground image in the frame image, and realize the tracking of a specific object in the frame image. Those skilled in the art can set n according to actual needs, and there is no limitation here . Wherein, n can be a fixed preset value. For example, when n is 20, then the frame images in the video are divided every 20 frames to obtain each group of frame images, and the method processes each group of divided frame images.

Fig. 1 shows a schematic flow chart of a method for real-time processing of video data based on adaptive tracking frame segmentation according to an embodiment of the present invention, the method is used to process each group of frame images obtained by dividing every n frames in the video, such as As shown in Figure 1, for one set of frame images, the method includes the following steps:

Step S100, acquiring the t-th frame image containing a specific object in a group of frame images and the tracking frame corresponding to the t-1-th frame image.

Wherein, the frame image contains a specific object, and the specific object may be a human body, a vehicle, or the like. Those skilled in the art may set specific objects according to actual needs, which is not limited here. When it is necessary to perform scene segmentation on the tth frame image in a group of frame images, where t is greater than 1, in step S100, acquire the tth frame image and the tracking frame corresponding to the t-1th frame image. Wherein, the tracking frame corresponding to the t-1th frame image can completely select the first foreground image frame in the t-1th frame image. Specifically, the tracking frame corresponding to the first frame image is determined according to the segmentation result corresponding to the first frame image.

Step S101, according to the tth frame image, adjust the tracking frame corresponding to the t-1th frame image to obtain the tracking frame corresponding to the tth frame image; A part of the frame image is subjected to scene segmentation processing to obtain a segmentation result corresponding to the t-th frame image.

In the process of using the tracking frame to track the first foreground image, the tracking frame needs to be adjusted according to each frame image, then for the tth frame image, the size and position of the tracking frame corresponding to the t-1th frame image can be adjusted Adjustment is made so that the adjusted tracking frame can be applied to the image of the tth frame, so as to obtain the tracking frame corresponding to the image of the tth frame. Since the tracking frame corresponding to the t-th frame image can frame the first foreground image in the t-th frame image, the scene can be performed on a part of the t-th frame image according to the tracking frame corresponding to the t-th frame image Segmentation processing to obtain the segmentation result corresponding to the t-th frame image. For example, the scene segmentation process may be performed on the area framed by the tracking frame corresponding to the t-th frame of image in the t-th frame of image. Compared with performing scene segmentation processing on all contents of a frame image in the prior art, the present invention only performs scene segmentation processing on a partial area of a frame image, which effectively reduces the amount of data processing for image scene segmentation and improves processing efficiency.

Step S102: Determine the second foreground image of the t-th frame of image according to the segmentation result corresponding to the t-th frame of image.

According to the segmentation result corresponding to the t-th frame image, it can be clearly determined which pixels in the t-th frame image belong to the second foreground image and which pixels belong to the second background image, thereby determining the second foreground image of the t-th frame image .

Step S103, adding personalized special effects according to the second foreground image to obtain the processed t-th frame image.

After the second foreground image is determined, a personalized special effect can be added according to the second foreground image to obtain the processed t-th frame image. Persons skilled in the art may set personalized special effects according to actual needs, which is not limited here. For example, according to the second foreground image, an effect map can be added at the edge of the specific object. The effect map can be a static effect map or a dynamic effect map. Specifically, when the specific object is a human body, the effect map can be Effect maps such as flames, jumping musical notes, waves, etc.; when the specific object is a human head, the effect maps can be effect maps such as hair crowns, shaking ears, etc., which are set according to the actual situation, and are not limited here.

Step S104, overlaying the processed t-th frame image on the t-th frame image to obtain processed video data.

The processed image of the tth frame is used to directly overwrite the original image of the tth frame, and the processed video data can be obtained directly. At the same time, the recorded user can also directly see the processed t-th frame image.

Step S105, displaying the processed video data.

When the processed t-th frame of image is obtained, the processed t-th frame of image will directly overwrite the original t-th frame of image. The speed of covering is relatively fast, and it is generally completed within 1/24 second. For the user, since the overlay processing time is relatively short, the human eye does not notice it obviously, that is, the human eye does not perceive the process that the original t-th frame image in the video data is overlaid. In this way, when the processed video data is subsequently displayed, it is equivalent to the processed video data being displayed in real time while shooting and/or recording and/or playing the video data, and the user will not feel that the frame image in the video data is covered display effect.

According to the video data real-time processing method based on adaptive tracking frame segmentation provided in this embodiment, for each group of frame images, based on the tracking frame corresponding to the t-1th frame image, the tracking frame corresponding to the t-th frame image is obtained, and using The tracking frame performs scene segmentation on the t-th frame image, and can quickly and accurately obtain the segmentation result corresponding to the t-th frame image, effectively improving the segmentation accuracy of the image scene segmentation. Compared with performing scene segmentation processing on all contents of the frame image in the prior art, the present invention only performs scene segmentation processing on a part of the frame image, which effectively reduces the amount of data processing for image scene segmentation and improves processing efficiency. The image scene segmentation processing method is optimized; and based on the obtained segmentation results, personalized special effects can be added to the frame image more accurately and quickly, and the video data display effect is beautified.

Fig. 2 shows a schematic flowchart of a method for real-time processing of video data based on adaptive tracking frame segmentation according to another embodiment of the present invention, the method is used to process each group of frame images obtained by dividing every n frames in the video, As shown in Figure 2, for one set of frame images, the method includes the following steps:

Step S200, acquiring the tth frame image containing a specific object in a group of frame images and the tracking frame corresponding to the t-1th frame image.

where t is greater than 1. For example, when t is 2, in step S200, the second frame image containing a specific object in a group of frame images and the tracking frame corresponding to the first frame image are acquired, specifically, the tracking frame corresponding to the first frame image The frame is determined according to the segmentation result corresponding to the first frame image; when t is 3, in step S200, the third frame image containing a specific object in a group of frame images and the second frame image corresponding to The tracking frame, wherein the tracking frame corresponding to the second frame image is obtained by adjusting the tracking frame corresponding to the first frame image during the scene segmentation process on the second frame image.

Step S201, perform recognition processing on the tth frame image, determine the first foreground image for a specific object in the tth frame image, apply the tracking frame corresponding to the t-1th frame image to the tth frame image, and according to the tth frame image For the first foreground image in the frame image, the tracking frame corresponding to the t-1th frame image is adjusted.

Specifically, image processing tools such as AE (Adobe After Effects) and NUKE (The FoundryNuke) in the prior art can be used to identify and process the image of the tth frame, which pixels in the image of the tth frame can be identified as belonging to the first foreground image, so as to determine the first foreground image for a specific object in the tth frame image. After determining the first foreground image, the tracking frame corresponding to the t-1th frame image can be set on the tth frame image, so as to adjust the tracking frame according to the first foreground image in the tth frame image, thus obtaining Tracking box corresponding to image frame t.

Specifically, it is possible to calculate the ratio of the pixels belonging to the first foreground image in the tth frame image to all the pixels in the tracking frame corresponding to the t-1th frame image, and determine this ratio as the tth frame image The first foreground pixel ratio, and then obtain the second foreground pixel ratio of the t-1th frame image, wherein the second foreground pixel ratio of the t-1th frame image is the pixel belonging to the first foreground image in the t-1th frame image Points in the proportion of all pixels in the tracking frame corresponding to the t-1th frame image, and then calculate the ratio between the first foreground pixel ratio of the t-th frame image and the second foreground ratio of the t-1th frame image Difference value, judge whether the difference value exceeds the preset difference threshold, if it is judged that the difference value exceeds the preset difference threshold, it means that the tracking frame corresponding to the t-1th frame image does not match the first foreground image in the tth frame image , then adjust the size of the tracking frame corresponding to the t-1th frame image according to the difference value. If it is determined that the difference value does not exceed the preset difference threshold, the size of the tracking frame corresponding to the t-1th frame image may not be adjusted. Those skilled in the art may set the preset difference threshold according to actual needs, which is not limited here.

Assume that after applying the tracking frame corresponding to the t-1th frame image to the t-th frame image, although the tracking frame corresponding to the t-1th frame image can completely frame the first foreground image in the t-th frame image However, the difference between the first foreground pixel ratio of the tth frame image and the second foreground ratio of the t-1th frame image exceeds the preset difference threshold, indicating that for the first foreground image in the tth frame image, The tracking frame corresponding to the t-1th frame image may be larger or smaller, so it is necessary to adjust the size of the tracking frame corresponding to the t-1th frame image. For example, when the first foreground pixel ratio of the tth frame image is 0.9, the second foreground pixel ratio of the t-1th frame image is 0.7, and the difference between the two ratios exceeds the preset difference threshold, then the difference can be based on the difference value Adaptively enlarge the size of the tracking frame corresponding to the t-1th frame image; for another example, when the first foreground pixel ratio of the t-th frame image is 0.5, the second foreground ratio of the t-1th frame image is 0.7 , and the difference value between the two ratios exceeds the preset difference threshold, then the size of the tracking frame corresponding to the t-1th frame image can be adaptively reduced according to the difference value.

Optionally, calculate the distance between the first foreground image in the tth frame image and the borders of the tracking frame corresponding to the t-1th frame image; according to the calculated distance and the preset distance threshold, for the t-1th frame image The size of the tracking frame corresponding to the frame image is adjusted. Those skilled in the art may set the preset distance threshold according to actual needs, which is not limited here. For example, if the calculated distance is less than the preset distance threshold, then the size of the tracking frame corresponding to the t-1th frame image can be adaptively enlarged, so that the distance between the first foreground image in the tth frame image and the tracking frame The distance of each frame meets the preset distance threshold; for another example, if the calculated distance is greater than the preset distance threshold, then the size of the tracking frame corresponding to the t-1th frame image can be adaptively reduced, so that the tth frame image The distance between the first foreground image in and each frame of the tracking frame meets the preset distance threshold.

In addition, the center point position of the first foreground image in the tth frame image can also be determined according to the first foreground image in the tth frame image; according to the center point position of the first foreground image in the tth frame image, pair and The position of the tracking frame corresponding to the t-1 frame image is adjusted so that the center point position of the tracking frame corresponding to the t-1 frame image coincides with the center point position of the first foreground image in the t frame image, Therefore, the first foreground image can be located in the middle of the tracking frame.

Step S202, according to the tracking frame corresponding to the t-th frame image, extract the image to be segmented from the partial area of the t-th frame image.

Specifically, the image in the tracking frame corresponding to the t-th frame image may be extracted from the t-th frame image, and the extracted image is determined as the image to be segmented. Since the tracking frame corresponding to the tth frame image can completely select the first foreground image frame in the tth frame image, then the pixels outside the tracking frame in the tth frame image belong to the second background image , so after obtaining the tracking frame corresponding to the t-th frame image, the image in the tracking frame corresponding to the t-th frame image can be extracted from the t-th frame image, and the image is determined as the image to be segmented, and then only The scene segmentation processing is performed on the image to be segmented, which effectively reduces the data processing amount of the image scene segmentation and improves the processing efficiency.

Step S203, performing scene segmentation processing on the image to be segmented to obtain a segmentation result corresponding to the image to be segmented.

Since the tracking frame corresponding to the t-th frame image can completely select the first foreground image frame in the t-th frame image, there is no need to perform scene segmentation processing on the pixels outside the tracking frame in the t-th frame image That is, it can be determined that the pixels outside the tracking frame belong to the second background image, so scene segmentation processing can only be performed on the extracted image to be segmented.

Wherein, when performing scene segmentation processing on the image to be segmented, a deep learning method can be used. Deep learning is a method based on representation learning of data in machine learning. Observations (such as an image) can be represented in a variety of ways, such as a vector of intensity values for each pixel, or more abstractly as a series of edges, regions of a specific shape, etc. And it is easier to learn tasks from examples with some specific representations. The segmentation method of deep learning can be used to perform scene segmentation processing on the image to be segmented, and the segmentation result corresponding to the image to be segmented can be obtained. Among them, the scene segmentation network obtained by the deep learning method can be used to perform scene segmentation processing on the image to be segmented, and the segmentation result corresponding to the image to be segmented is obtained. According to the segmentation result, it can be determined which pixels in the image to be segmented belong to the second foreground image. Which pixels belong to the second background image.

Specifically, the image to be segmented may be input into the scene segmentation network to obtain a segmentation result corresponding to the image to be segmented. In the prior art, in order to facilitate the scene segmentation processing of the input image by the scene segmentation network, it is necessary to adjust the size of the image to a preset size, for example, the preset size is 320×240 pixels, and in general Below, the size of the image is mostly 1280×720 pixels, so it needs to be resized to 320×240 pixels first, and then the scene segmentation process is performed on the resized image. However, when the scene segmentation network is used to perform scene segmentation processing on the frame images in the video, if the proportion of the first foreground image in the frame image is small, such as the proportion of the first foreground image in the frame image is 0.2, then according to The existing technology still needs to reduce the size of the frame image, and then perform scene segmentation processing on it, so when performing scene segmentation processing, it is easy to divide the pixels that actually belong to the edge of the second foreground image into the second background image, the resulting segmentation results have low segmentation accuracy and poor segmentation effect.

According to the technical solution provided by the present invention, the image in the tracking frame corresponding to the t-frame image extracted from the t-frame image is determined as the image to be divided, and then the scene segmentation process is performed on the image to be divided. When the proportion of the first foreground image in the tth frame image is small, the size of the extracted image to be segmented will also be far smaller than the size of the tth frame image, so the image to be segmented and adjusted to the preset size are adjusted to Compared with frame images with preset sizes, foreground image information can be preserved more effectively, so the segmentation results obtained have higher segmentation accuracy.

In step S204, according to the segmentation result corresponding to the image to be segmented, the segmentation result corresponding to the t-th frame image is obtained.

The image to be segmented is the image in the tracking frame corresponding to the t-th frame image, and which pixels in the image to be segmented belong to the second foreground image and which pixels belong to the second background can be clearly determined according to the segmentation result corresponding to the image to be segmented image, and the pixels outside the tracking frame in the tth frame image belong to the second background image, so the segmentation corresponding to the tth frame image can be obtained conveniently and quickly according to the segmentation result corresponding to the image to be segmented As a result, it can be clearly determined which pixels in the t-th frame image belong to the second foreground image and which pixels belong to the second background image. Compared with performing scene segmentation processing on all contents of frame images in the prior art, the present invention only performs scene segmentation processing on images to be segmented extracted from frame images, which effectively reduces the amount of data processing for image scene segmentation and improves processing efficiency.

Step S205: Determine the second foreground image of the t-th frame of image according to the segmentation result corresponding to the t-th frame of image.

Step S206, adding personalized special effects according to the second foreground image to obtain the processed t-th frame image.

In a specific implementation manner, the key information of the area to be processed can be extracted from the second foreground image, and the effect map is drawn according to the key information. The key information can be specifically key point information, key area information, and/or key line information, etc. The embodiment of the present invention is described by taking the key information as the key point information as an example, but the key information in the present invention is not limited to the key point information. The use of key point information can improve the processing speed and efficiency of drawing effect maps based on key point information, and can directly draw effect maps based on key point information, without the need for subsequent calculations, analysis and other complex operations on key information. At the same time, key point information is easy to extract, and the extraction is accurate, making the effect of drawing effect maps more accurate. Specifically, the key point information of the edge of the region to be processed may be extracted from the second foreground image. Those skilled in the art can set the area to be treated according to actual needs, which is not limited here.

In order to draw the effect map conveniently and quickly, many basic effect maps can be drawn in advance, then when drawing the effect map, you can first find the corresponding basic effect map, and then process the basic effect map to get the effect quickly stickers. These basic effect maps may include different clothing effect maps, decorative effect maps, texture effect maps, etc. For example, the decorative effect maps may be effect maps such as flames, dancing musical notes, waves, hair crowns, and shaking ears. In addition, in order to facilitate the management of these basic effect textures, an effect texture library can be established, and these basic effect texture maps can be stored in the effect texture library.

Specifically, taking the key information as the key point information as an example, after extracting the key point information of the area to be processed from the second foreground image, the basic effect map corresponding to the key point information can be searched, and then according to the key point information, calculate The position information between at least two key points having a symmetrical relationship, and then according to the position information, the basic effect map is processed to obtain the effect map. In this way, the effect map can be accurately drawn. Wherein, the method can automatically search the basic effect map corresponding to the key point information from the effect map library according to the extracted key point information. In addition, in practical applications, in order to facilitate the use of users and better meet the individual needs of users, the basic effect textures contained in the effect texture library can be displayed to users, and users can specify the basic effect textures according to their own preferences. Then In this case, this method obtains a user-specified base effect texture.

After the effect map is drawn, the effect map, the second foreground image and the preset background image may be fused to obtain the processed t-th frame image. Those skilled in the art can set a preset background image according to actual needs, which is not limited here. The preset background image can be a 2D scene background image, or a 3D scene background image, such as a 3D seabed scene background image, a 3D volcano scene background image, and other 3D scene background images. In addition, the effect map, the second foreground image and the second background image determined according to the segmentation result corresponding to the t-th frame image (that is, the original background image of the t-th frame image) can also be fused to obtain the processed first t frames of images.

Optionally, in another specific implementation manner, the key information of the area to be recognized can be extracted from the second foreground image, and then the gesture of the specific object is recognized according to the key information to obtain the gesture recognition result of the specific object, Then, according to the posture recognition result of the specific object, determine the corresponding effect processing command to be responded to the t-th frame of image, and obtain the processed t-th frame of image.

When recognizing the posture of a specific object, the key information can be matched with the preset posture key information to obtain the posture recognition result; in addition, the trained posture recognition network can also be used to recognize the posture of a specific object, because the recognition network is trained , so the gesture recognition result of a specific object can be obtained conveniently and quickly. After the gesture recognition result of the specific object is obtained, according to different gesture recognition results of the specific object, the corresponding effect processing command to be responded to the t-th frame image is determined. Specifically, gesture recognition results may include different shapes of facial gestures, gestures, leg movements, whole-body gestures, etc. According to different gesture recognition results, different application scenarios (the scene where the video data is located, the scene where the video data is used) ), one or more corresponding effect processing commands to be responded to can be determined for different gesture recognition results. Wherein, the same gesture recognition result can determine different effect processing commands to be responded to in different application scenarios, and different gesture recognition results can also determine the same effect processing commands to be responded to in the same application scenario. For a gesture recognition result, the determined effect processing commands to be responded to may include one or more processing commands. It is specifically set according to the implementation situation, and is not limited here. After determining the effect processing command to be responded to, in response to the effect processing command to be responded, the t-th frame of image is processed according to the effect processing command to be responded to obtain the processed t-th frame of image.

The effect processing commands to be responded to may include, for example, various effect map processing commands, stylization processing commands, brightness processing commands, lighting processing commands, tone processing commands, and the like. The effect processing command to be responded can include the above multiple processing commands at one time, so that when the t-th frame image is processed according to the effect processing command to be responded to, the effect of the t-th frame image after processing is more realistic and more coordinated overall.

For example, when the user is taking a selfie, live broadcasting or recording a quick video, if the gesture recognition result obtained through recognition is a hand-to-heart shape, then the determined effect processing command to be responded to the t-th frame image can be the t-th frame image Add the heart-shaped effect texture processing command in , the heart-shaped effect texture can be a static texture or a dynamic texture; The effect processing command to be responded to the t-th frame image may include an effect map command for adding sunflowers on the head, a stylized processing command for modifying the style of the t-th frame image to a pastoral style, and performing a stylization command on the lighting effect of the t-th frame image. Handled lighting processing commands (sunny lighting effects), etc.

Optionally, according to the gesture recognition result of the specific object and the interaction information with the interactive object contained in the t-th frame image, the corresponding effect processing command to be responded to the t-th frame image can be determined, and the processed t-th frame image can be obtained. t frames of images.

For example, when the user is live broadcasting, the t-th frame of image not only contains the user (that is, a specific object), but also contains interactive information with interactive objects (such as the audience watching the live broadcast), such as the audience watching the live broadcast giving the user an ice cream , an ice cream will appear on the tth frame image. Combined with the interaction information, when the obtained gesture recognition result is that the user is eating ice cream, and the effect processing command to be responded is determined to remove the original ice cream effect map and add the effect map for reducing the bite of the ice cream, then the t-th frame image is processed according to The effect processing command to be responded is processed to increase the interactive effect with the audience watching the live broadcast and attract more viewers to watch the live broadcast.

In step S207, the processed t-th frame of image is overlaid with the t-th frame of image to obtain processed video data.

The processed image of the tth frame is used to directly overwrite the original image of the tth frame, and the processed video data can be obtained directly. At the same time, the recorded user can also directly see the processed t-th frame image.

Step S208, displaying the processed video data.

After the processed video data is obtained, it can be displayed in real time, and the user can directly see the display effect of the processed video data.

Step S209, uploading the processed video data to the cloud server.

The processed video data can be directly uploaded to the cloud server, specifically, the processed video data can be uploaded to one or more cloud video platform servers, such as iQiyi, Youku, Kuai Video and other cloud video platform servers, For the cloud video platform server to display video data on the cloud video platform. Or the processed video data can also be uploaded to the cloud live broadcast server. When a user with a live viewing terminal enters the cloud live broadcast server to watch, the cloud live broadcast server can push the video data to the viewing user client in real time. Or the processed video data can also be uploaded to the cloud official account server. When a user pays attention to the official account, the cloud official account server will push the video data to the official account follower client; further, the cloud official account server can also according to Pay attention to the viewing habits of the users of the official account, and push the video data in line with the user's habits to the official account follower client.

According to the video data real-time processing method based on adaptive tracking frame segmentation provided in this embodiment, for each group of frame images, according to the first foreground image in the tth frame image, the tracking frame corresponding to the t-1th frame image is processed Adjust the processing to obtain the tracking frame corresponding to the t-th frame image, and use the tracking frame to extract the image to be segmented. According to the segmentation result corresponding to the image to be segmented, the segmentation result corresponding to the t-th frame image can be quickly and accurately obtained , which effectively improves the segmentation accuracy of image scene segmentation. Compared with performing scene segmentation processing on all contents of the frame image in the prior art, the present invention only performs scene segmentation processing on the image to be segmented extracted from the frame image, effectively reducing the data processing amount of the image scene segmentation, The processing efficiency is improved, and the image scene segmentation processing method is optimized; and based on the obtained segmentation results, personalized special effects can be added to the frame image more accurately and quickly, and the video data display effect is beautified; in addition, based on the obtained segmentation results It can also identify gestures more accurately, quickly and accurately determine the effect processing command to be responded to process the frame image, and optimize the video data processing method.

Fig. 3 shows a structural block diagram of a real-time video data processing device based on adaptive tracking frame segmentation according to an embodiment of the present invention, the device is used to process each group of frame images obtained by dividing every n frames in the video, such as As shown in FIG. 3 , the device includes: an acquisition module 310 , a segmentation module 320 , a determination module 330 , a processing module 340 , an overlay module 350 and a display module 360 .

The acquisition module 310 is adapted to: acquire the t-th frame image containing a specific object in a group of frame images and the tracking frame corresponding to the t-1-th frame image.

Where t is greater than 1; the tracking frame corresponding to the first frame image is determined according to the segmentation result corresponding to the first frame image.

The segmentation module 320 is adapted to: adjust the tracking frame corresponding to the t-1th frame image according to the tth frame image to obtain the tracking frame corresponding to the tth frame image; according to the tracking frame corresponding to the tth frame image, Scene segmentation processing is performed on a part of the t-th frame image to obtain a segmentation result corresponding to the t-th frame image.

Optionally, the segmentation module 320 is further adapted to: perform recognition processing on the t-th frame image, determine the first foreground image for a specific object in the t-th frame image; apply the tracking frame corresponding to the t-1-th frame image to the t-th frame image t-frame image; according to the first foreground image in the t-th frame image, adjust the tracking frame corresponding to the t-1-th frame image.

Specifically, the segmentation module 320 is further adapted to: calculate the ratio of pixels belonging to the first foreground image in the tth frame image to all pixels in the tracking frame corresponding to the t-1th frame image, and determine the ratio as The first foreground pixel ratio of the tth frame image; obtain the second foreground pixel ratio of the t-1th frame image, wherein the second foreground pixel ratio of the t-1th frame image is the first foreground pixel ratio in the t-1th frame image The ratio of the pixels of the foreground image to all the pixels in the tracking frame corresponding to the t-1th frame image; calculate the first foreground pixel ratio of the tth frame image and the second foreground ratio of the t-1th frame image The difference value between; determine whether the difference value exceeds the preset difference threshold; if so, adjust the size of the tracking frame corresponding to the t-1th frame image according to the difference value.

The segmentation module 320 is further adapted to: calculate the distance between the first foreground image in the t-th frame image and the distance between each frame of the tracking frame corresponding to the t-1-th frame image; The size of the tracking frame corresponding to the frame image is adjusted.

The segmentation module 320 is further adapted to: according to the first foreground image in the tth frame image, determine the center point position of the first foreground image in the tth frame image; according to the center point position of the first foreground image in the tth frame image , adjust the position of the tracking frame corresponding to the t-1th frame image, so that the center point position of the tracking frame corresponding to the t-1th frame image is the same as the center point of the first foreground image in the t-th frame image The location coincides.

Optionally, the segmentation module 320 is further adapted to: extract the image to be segmented from the partial area of the t-th frame image according to the tracking frame corresponding to the t-th frame image; perform scene segmentation processing on the image to be segmented, and obtain the corresponding The segmentation result of ; according to the segmentation result corresponding to the image to be segmented, the segmentation result corresponding to the t-th frame image is obtained.

The segmentation module 320 is further adapted to: extract the image in the tracking frame corresponding to the t-th frame image from the t-th frame image, and determine the extracted image as the image to be segmented.

The segmentation module 320 is further adapted to: input the image to be segmented into the scene segmentation network, and obtain a segmentation result corresponding to the image to be segmented.

The determining module 330 is adapted to: determine the second foreground image of the t-th frame of image according to the segmentation result corresponding to the t-th frame of image.

The processing module 340 is adapted to: add personalized special effects according to the second foreground image to obtain the processed t-th frame of image.

Optionally, the processing module 340 is further adapted to: extract key information of the area to be processed from the second foreground image; draw an effect map according to the key information; perform fusion processing on the effect map, the second foreground image and the preset background image , to obtain the processed t-th frame image; or, perform fusion processing on the effect map, the second foreground image and the second background image determined according to the segmentation result corresponding to the t-th frame image, to obtain the processed t-th frame image.

The key information may specifically be key point information, key area information, and/or key line information. The embodiments of the present invention are described by taking key information as key point information as an example. The processing module 340 is further adapted to: find the basic effect map corresponding to the key point information; or obtain the basic effect map specified by the user; calculate the position information between at least two key points with a symmetrical relationship according to the key point information; The position information is used to process the basic effect map to obtain the effect map.

Optionally, the processing module 340 is further adapted to: extract the key information of the area to be recognized from the second foreground image; recognize the gesture of the specific object according to the key information, and obtain the gesture recognition result of the specific object; As a result of gesture recognition, determine the corresponding effect processing command to be responded to the t-th frame of image, and obtain the processed t-th frame of image. Wherein, the effect processing command to be responded to includes an effect map processing command, a stylization processing command, a brightness processing command, a lighting processing command and/or a tone processing command.

Optionally, the processing module 340 is further adapted to: determine the corresponding effect processing command to be responded to the t-th frame image according to the gesture recognition result of the specific object and the interaction information with the interactive object included in the t-th frame image, Get the image of frame t after processing.

The overlay module 350 is adapted to: cover the processed t-th frame of image with the t-th frame of image to obtain processed video data.

The display module 360 is adapted to: display the processed video data.

After the display module 360 obtains the processed video data, it can display it in real time, and the user can directly see the display effect of the processed video data.

The device may further include: an upload module 370, adapted to upload the processed video data to a cloud server.

The upload module 370 can directly upload the processed video data to the cloud server. Specifically, the upload module 370 can upload the processed video data to one or more cloud video platform servers, such as iQiyi, Youku, Kuai Video Wait for the cloud video platform server for the cloud video platform server to display video data on the cloud video platform. Or the upload module 370 can also upload the processed video data to the cloud live server. When a user at the live viewing end enters the cloud live server to watch, the cloud live server can push the video data to the viewing user client in real time. Or the upload module 370 can also upload the processed video data to the cloud public number server, and when a user pays attention to the public number, the cloud public number server pushes the video data to the public number attention client; further, the cloud public number server It is also possible to push video data that conforms to user habits to the official account follower client according to the viewing habits of users who follow the official account.

According to the video data real-time processing device based on adaptive tracking frame segmentation provided in this embodiment, for each group of frame images, based on the tracking frame corresponding to the t-1th frame image, the tracking frame corresponding to the t-th frame image is obtained, and using The tracking frame performs scene segmentation on the t-th frame image, and can quickly and accurately obtain the segmentation result corresponding to the t-th frame image, effectively improving the segmentation accuracy of the image scene segmentation. Compared with performing scene segmentation processing on all contents of the frame image in the prior art, the present invention only performs scene segmentation processing on a part of the frame image, which effectively reduces the amount of data processing for image scene segmentation and improves processing efficiency. The image scene segmentation processing method is optimized; and based on the obtained segmentation results, personalized special effects can be added to the frame image more accurately and quickly, and the video data display effect is beautified.

The present invention also provides a non-volatile computer storage medium, the computer storage medium stores at least one executable instruction, and the executable instruction can execute the video data real-time processing method based on adaptive tracking frame segmentation in any of the above method embodiments .

Fig. 4 shows a schematic structural diagram of a computing device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.

As shown in FIG. 4 , the computing device may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

in:

The processor 402 , the communication interface 404 , and the memory 406 communicate with each other through the communication bus 408 .

The communication interface 404 is used to communicate with network elements of other devices such as clients or other servers.

The processor 402 is configured to execute the program 410, and specifically, may execute relevant steps in the embodiment of the above-mentioned method for real-time processing of video data based on adaptive tracking frame segmentation.

Specifically, the program 410 may include program codes including computer operation instructions.

The processor 402 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be of the same type, such as one or more CPUs, or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 406 is used to store the program 410 . The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 410 may be specifically configured to enable the processor 402 to execute the video data real-time processing method based on adaptive tracking frame segmentation in any of the above method embodiments. For the specific implementation of each step in the program 410, refer to the corresponding description of the corresponding steps and units in the above-mentioned embodiment of the real-time processing of video data by adaptive tracking frame segmentation, and details are not repeated here. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the foregoing method embodiments, and details are not repeated here.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other device. Various generic systems can also be used with the teachings based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not specific to any particular programming language. It should be understood that various programming languages can be used to implement the content of the present invention described herein, and the above description of specific languages is for disclosing the best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components in the embodiments of the present invention. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

Claims (10)

1. it is a kind of based on adaptive tracing frame segmentation video data real-time processing method, the method be used for in video every The each group two field picture that n frames divide is handled, for one of which two field picture, the described method includes：

It obtains in the framing image and includes the t two field pictures of special object and tracking corresponding with t-1 two field pictures Frame, wherein t are more than 1；Tracking box corresponding with the 1st two field picture is according to determined by segmentation result corresponding with the 1st two field picture；

According to t two field pictures, a pair tracking box corresponding with t-1 two field pictures is adjusted processing, obtains corresponding with t two field pictures Tracking box；According to tracking box corresponding with t two field pictures, the subregion of the t two field pictures is carried out at scene cut Reason, obtains segmentation result corresponding with t two field pictures；

According to segmentation result corresponding with t two field pictures, the second foreground image of t two field pictures is determined；

According to second foreground image, personalized special efficacy is added, the t two field pictures that obtain that treated；

Treated the t two field pictures are covered into the t two field pictures video data that obtains that treated；

Video data after display processing.

2. according to the method described in claim 1, wherein, described according to second foreground image, the personalized special efficacy of addition obtains To treated, t two field pictures further comprise：

The key message of pending area is extracted from second foreground image；

According to the key message, effect textures are drawn；

The effect textures, second foreground image and default background image are subjected to fusion treatment, the t that obtains that treated Two field picture；It is alternatively, the effect textures, second foreground image is true with according to segmentation result corresponding with t two field pictures Fixed the second background image carries out fusion treatment, the t two field pictures that obtain that treated.

3. method according to claim 1 or 2, wherein, the key message is key point information；It is described according to the pass Key information is drawn effect textures and is further comprised：

Search basic effect textures corresponding with the key point information；Alternatively, obtain the basic effect textures that user specifies；

According to the key point information, the location information between at least two key points with symmetric relation is calculated；

According to the location information, the basic effect textures are handled, obtain effect textures.

4. according to claim 1-3 any one of them methods, wherein, it is described according to second foreground image, add individual character Change special efficacy, obtaining that treated, t two field pictures further comprise：

The key message in region to be identified is extracted from second foreground image；

According to the key message, the posture of the special object is identified, obtains the gesture recognition of the special object As a result；

According to the gesture recognition of the special object as a result, determining the corresponding effect process order to be responded to t two field pictures, The t two field pictures that obtain that treated.

5. according to claim 1-4 any one of them methods, wherein, the gesture recognition knot according to the special object Fruit determines the corresponding effect process order to be responded to t two field pictures, and obtaining that treated, t two field pictures further comprise：

According to the gesture recognition of the special object as a result, and the friendship with interactive object included in the t two field pictures Mutual information determines the corresponding effect process order to be responded to t two field pictures, the t two field pictures that obtain that treated.

6. according to claim 1-5 any one of them methods, wherein, the effect process order to be responded is pasted including effect Figure processing order, stylization processing order, brightness processed order, photo-irradiation treatment order and/or tone processing order.

7. according to claim 1-6 any one of them methods, wherein, the foundation t two field pictures, pair with t-1 two field pictures Corresponding tracking box is adjusted processing and further comprises：

Processing is identified to t two field pictures, determines to be directed to the first foreground image of special object in t two field pictures；

Tracking box corresponding with t-1 two field pictures is applied to t two field pictures；

The first foreground image in t two field pictures, a pair tracking box corresponding with t-1 two field pictures are adjusted processing.

8. it is a kind of based on adaptive tracing frame segmentation video data real-time processing device, described device be used for in video every The each group two field picture that n frames divide is handled, and described device includes：

Acquisition module, suitable for obtain include in a framing image special object t two field pictures and with t-1 frame figures As corresponding tracking box, wherein t is more than 1；Tracking box corresponding with the 1st two field picture is according to segmentation corresponding with the 1st two field picture As a result it is identified；

Split module, suitable for according to t two field pictures, a pair tracking box corresponding with t-1 two field pictures is adjusted processing, obtain and The corresponding tracking box of t two field pictures；According to tracking box corresponding with t two field pictures, to the subregions of the t two field pictures into The processing of row scene cut, obtains segmentation result corresponding with t two field pictures；

Determining module, suitable for according to segmentation result corresponding with t two field pictures, determining the second foreground image of t two field pictures；

Processing module, suitable for according to second foreground image, adding personalized special efficacy, the t two field pictures that obtain that treated；

Overlay module, suitable for treated the t two field pictures are covered the t two field pictures video counts that obtain that treated According to；

Display module, suitable for the video data after display processing.

9. a kind of computing device, including：Processor, memory, communication interface and communication bus, the processor, the storage Device and the communication interface complete mutual communication by the communication bus；

For the memory for storing an at least executable instruction, the executable instruction makes the processor perform right such as will Ask the corresponding operation of video data real-time processing method based on the segmentation of adaptive tracing frame any one of 1-7.

10. a kind of computer storage media, an at least executable instruction, the executable instruction are stored in the storage medium The video data based on the segmentation of adaptive tracing frame that processor is performed as any one of claim 1-7 is made to handle in real time The corresponding operation of method.