CN114445750A - Video object segmentation method, device, storage medium and program product - Google Patents

Abstract

The present application provides a video object segmentation method, device, storage medium and program product, wherein the method includes: determining a video to be processed and a target object to be segmented in the video; Process each frame of image to be processed in the video, and segment the target object from each frame of image; wherein, the target segmentation model includes a cyclic feature update model, which is used in the video processing process according to the currently processed image. The loop feature is updated. The application can realize the dynamic update of the loop feature through the model, which can not only make the loop feature fuse the information of the processed multi-frame images, but also effectively avoid the stacking of features. The size of the loop feature can not gradually increase with the progress of the video. It reduces the memory occupation, avoids the long video affecting the segmentation effect due to insufficient memory, and reduces the cost of the video target segmentation algorithm.

Description

Video object segmentation method, device, storage medium and program product

technical field

The present application relates to the technical field of video processing, and in particular, to a video object segmentation method, device, storage medium and program product.

Background technique

Video object segmentation can determine the region where the target object is located from each frame of the video, and plays an important role in object tracking, video matting and other scenes.

In the process of video target segmentation, the segmented image information can be stored to guide the segmentation of subsequent images. The disadvantage of this scheme is that, with the continuous advancement of video processing, the segmented image information will be continuously stacked in the memory, occupying a large amount of memory, and, for long videos, the limited memory may not be able to support infinite growth. Feature storage, which affects the segmentation effect.

SUMMARY OF THE INVENTION

The main purpose of the embodiments of the present application is to provide a video target segmentation method, device, storage medium and program product, so as to reduce the cost of video target segmentation and improve the segmentation effect.

In a first aspect, an embodiment of the present application provides a video target segmentation method, including:

Determine the video to be processed and the target object to be segmented in the video;

Through the pre-built target segmentation model, each frame of images to be processed in the video is sequentially processed based on the cyclic feature, and the target object is segmented from each frame of image; wherein, the target segmentation model includes a cyclic feature update model, which uses During the video processing, the loop feature is updated according to the currently processed image.

Optionally, the target segmentation model further includes: an encoder, a feature reading module, and a decoder;

Wherein, when processing any frame of image in the video, the encoder is used to extract the image feature corresponding to the image;

The feature reading module is used to calculate and obtain the associated information of the image feature and the current loop feature;

The decoder is configured to obtain a target segmentation result of the image according to the associated information.

Optionally, the number of the target object is at least one, and the method further includes:

According to the target segmentation result of the video, the target video of each target object is obtained; wherein, the target video of the target object is the video obtained after removing the background of the target object from the video to be processed, or, the The target video is composed of multiple frames of images where the target object is located in the video to be processed;

A target video of each target object and/or at least part of images in the target video is output.

Optionally, output the target video of each target object and/or at least part of the image in the target video, including:

displaying the target video and/or at least part of the images in the target video;

According to the user's editing operation on the target video and/or at least part of the images in the target video, a promotion video and/or promotion image of the target object is generated.

Optionally, the processing operations on the images in the video include:

Obtain the image to be processed in the video and the current loop feature, and extract the image feature corresponding to the image; wherein, the loop feature is a feature determined according to the processed image in the video;

Determine the target segmentation result of the image to be processed according to the image features and cycle features, and input the image features and cycle features into the cycle feature update model to obtain the updated cycle features, so as to obtain the updated cycle features according to the updated cycle features The feature processes the next frame in the video to be processed.

Optionally, determining the video to be processed and the target object to be segmented in the video, including:

Obtain the video to be processed, and determine at least one frame of reference image in the video and the target segmentation result corresponding to each reference image;

The method further includes: determining an initial loop feature according to the at least one frame of reference image and the corresponding target segmentation result;

Wherein, the target segmentation result of the reference image is determined by annotating the target object in the reference image by the user, or is determined by automatically labeling the reference image according to a preset image of the target object;

The images to be processed in the video are other images except the reference image; the initial cycle feature is used to segment the images to be processed in the first frame.

Optionally, the method also includes at least one of the following:

Displaying the reference image in the video, so that the user can mark the location of the target object on the reference image, and determine the target segmentation result of the reference image according to the user's label;

Semantically segment the reference image in the video to determine at least one object in the reference image; and compare the image area corresponding to the at least one object with the preset image of the target object to obtain the reference image The target segmentation result;

Semantic segmentation is performed on the reference image in the video to determine at least one object in the reference image; the image area corresponding to the at least one object is displayed, the target object selected by the user from the at least one object is acquired, and according to the The image area corresponding to the target object determines the target segmentation result.

Optionally, inputting the image features and cyclic features into a cyclic feature update model to obtain updated cyclic features, including:

Inputting the image features and the cyclic features into a refining module to obtain a refining result; wherein the refining module is used to fuse the image features and the cyclic features;

Inputting the refining result to an enhancement module to obtain an enhancement result; wherein, the enhancement module is used to perform a pooling operation on the refining result;

The refined result and the enhancement result are added and input to the compression module to obtain the updated loop feature of preset length.

Optionally, the refining module includes a downsampling layer, a convolution layer, a matrix computing unit, and a normalization layer; input the image features and cyclic features into the refining module to obtain a refining result, including:

The image features and cyclic features are down-sampled through the down-sampling layer, and the down-sampled image features and cyclic features are respectively input to the convolution layer to obtain a first feature matrix corresponding to the image features and a second feature matrix corresponding to the cyclic features. feature matrix;

A matrix correlation operation is performed on the first feature matrix and the second feature matrix by a matrix computing unit, and the operation result is input to the normalization layer to obtain a refined result.

In a second aspect, an embodiment of the present application also provides a video target segmentation method, including:

Determine the video to be processed and the target product to be segmented in the video;

Through a pre-built target segmentation model, each frame of images to be processed in the video is sequentially processed based on the cyclic feature, and the target product is segmented from each frame of image; wherein, the target segmentation model includes a cyclic feature update model, using In the video processing process, the loop feature is updated according to the currently processed image;

According to the target segmentation result of the video, a target video and/or a target image of the target product is generated.

Optionally, the target image is the image where the target object is located in the video to be processed, or an image obtained after removing the background in the image where the target object is located; the method further includes:

Obtain the filter conditions input by the user, and filter the target image according to the filter conditions;

The filtered images are updated to the material library, where the material library is used to make promotional videos and/or promotional images of the target product.

In a third aspect, an embodiment of the present application provides an electronic device, including:

at least one processor; and

a memory communicatively coupled to the at least one processor;

Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the electronic device to perform the method according to any one of the above aspects.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium, and when a processor executes the computer-executable instructions, any of the above-mentioned aspects is implemented. method.

In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any one of the foregoing aspects when the computer program is executed by a processor.

The video object segmentation method, device, storage medium and program product provided by the present application can determine the video to be processed and the target object to be segmented in the video, and then use a pre-built target segmentation model to sequentially analyze the video based on cycle features. Each frame of images to be processed in the video processing is processed, and the target object is segmented from each frame of images, wherein the target segmentation model includes a cyclic feature update model, which is used in the video processing process, according to the currently processed image. The feature is updated, so that the dynamic update of the loop feature can be realized through the model, which can not only make the loop feature integrate the information of the processed multi-frame images, but also effectively avoid the stacking of features, and the size of the loop feature can not gradually increase with the progress of the video. It reduces the memory occupation, avoids the long video affecting the segmentation effect due to insufficient memory, and reduces the cost of the video target segmentation algorithm.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.

1 is a schematic diagram of an application of target segmentation provided by an embodiment of the present application;

Fig. 2 is a kind of application scene diagram that this application provides;

3 is a schematic flowchart of a video target segmentation method provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a principle for determining a target segmentation result according to an embodiment of the present application;

FIG. 5 is a schematic diagram of updating a cycle feature according to a step size according to an embodiment of the present application;

FIG. 6 is an application flowchart of a target segmentation result provided by an embodiment of the present application;

FIG. 7 is a schematic schematic diagram of an update cycle feature provided by an embodiment of the present application;

8 is a schematic flowchart of another video object segmentation method provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Specific embodiments of the present application have been shown by the above-mentioned drawings, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the concepts of the present application in any way, but to illustrate the concepts of the present application to those skilled in the art by referring to specific embodiments.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this application.

First, the terms involved in this application are explained:

Semi-supervised Video Object Segmentation (VOS): For a video, the initial annotation of one or more objects to be segmented is provided in the first frame. The target provided by the frame is segmented. Semi-supervised video target segmentation only requires a small amount of interaction to complete the segmentation of the entire video, which can reduce manpower and improve efficiency.

Neural network model: a complex network system formed by interconnecting a large number of neurons, which is widely used in image processing and other fields.

The embodiments of the present application can be applied to any scene that needs to perform target segmentation on a video. For example, in video editing, post-production, and video conferencing, it is often necessary to perform precise, pixel-level segmentation of objects in the video. The user can improve the target segmentation result of the first frame by interactively clicking, and the algorithm can segment the target object in all subsequent frames according to the video and the segmentation result of the target object to be segmented in the first frame.

Optionally, in the commodity field, the video can be segmented by target, and the target commodity in the video can be segmented, which can be used for material production after removing the background.

FIG. 1 is an application schematic diagram of a target segmentation provided by an embodiment of the present application. As shown in Figure 1, through the target segmentation method, the target product in each frame of the video, such as a cup, can be segmented. After obtaining the target segmentation result, the background can be added to the target product and edited to obtain the processed video, such as the target product. It can quickly realize the production of various commodity videos.

Optionally, the embodiments of the present application can be particularly applied to target segmentation for live video. Exemplarily, in an avatar live broadcast scenario, the avatar can introduce the target commodity, form a video stream capable of displaying the target commodity, and transmit it to the user terminal for the convenience of the user to watch. During the live broadcast, target segmentation can be performed on the real-time video stream, or, after the live broadcast is over, the target segmentation can be performed on the video. According to the target segmentation result, the image frame of the target product and its position in each frame of image can be determined. Therefore, the video material and image material of the target product can be obtained by using the live video, which is convenient for subsequent use.

In the field of video conferencing, the participants can be photographed by the camera to obtain the video stream of the participants, and the participants in the obtained video stream can be segmented by the target segmentation method, and the background can be blurred, or Adding a virtual background to form a processed video stream to display to other participants in the conference can effectively protect user privacy and meet the individual needs of users.

In the field of short video, the target object in the video can be segmented as the video production material through the target segmentation method. For example, after a user records a dancing video indoors, he can segment the target person in the video and add scenes such as pastoral scenes, party scenes, etc., so that the user can select the appropriate scene according to the dance style and action, and improve the production effect of the short video. .

In the process of target segmentation of the video, in order to achieve a better segmentation effect, when segmenting each frame of image, the image and historical information can be used to obtain the segmentation result corresponding to the image, where the historical information can be Information corresponding to the segmented images in the video. As the video progresses, historical information will continue to increase, affecting the effect of video segmentation.

In view of this, the present application provides a video target segmentation method, which can set a cyclic feature to assist in achieving target segmentation, and in the progress of video processing, utilize the cyclic feature to process the current image to obtain the target segmentation result of the current image, and The model is used to fuse the cyclic features with the current image features to obtain the updated cyclic features for subsequent image processing.

FIG. 2 is an application scenario diagram provided by the present application. As shown in FIG. 2 , the user terminal can upload the captured video containing the target object to the server, and the server can perform target segmentation on the video based on the target segmentation model. The target segmentation model can include a cyclic feature update model and a segmentation module. When processing each frame of image, the current cycle feature can be extracted from the storage unit, the image feature and the current cycle feature can be input into the segmentation module, the target segmentation result of the image can be obtained, and the image feature and cycle feature can be input into the cycle The feature updates the model, obtains the updated cyclic feature, and updates the cyclic feature in the storage unit. When processing the next frame of image, the updated cyclic feature can be used. After processing each frame of images in the video, the server may feed back the target segmentation result of the video to the user terminal.

Through the cyclic feature update model, the currently processed image features and the cyclic features are continuously fused to obtain the updated cyclic features, which can realize the cyclic dynamic change of the cyclic features. It can also effectively avoid the stacking of features. The size of the loop feature is related to the output dimension of the model, and it can gradually increase without the advancement of the video, which reduces the memory usage, prevents the long video from affecting the segmentation effect due to insufficient memory, and reduces the video target. The cost of the segmentation algorithm improves the target segmentation effect.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments may be combined with each other without conflict between the embodiments. In addition, the sequence of steps in the following method embodiments is only an example, and is not strictly limited.

FIG. 3 is a schematic flowchart of a video object segmentation method provided by an embodiment of the present application. The execution body of the method in this embodiment may be any device having an image processing function, such as a server, a user terminal, and the like. As shown in Figure 3, the method may include:

Step 301: Determine the video to be processed and the target object to be segmented in the video.

The video may be any type of video, such as a long video or a short video. The video may contain at least one object, and the embodiment of the present application may be used to segment the target object from the video.

Optionally, the target object may be set according to actual needs, for example, it may be a target commodity, or a person. The number of the target objects may be one or more. The target object may be determined by the user or automatically determined by the device.

Step 302: Process each frame of image to be processed in the video in turn based on the cyclic feature through a pre-built target segmentation model, and segment the target object from each frame of image; wherein, the target segmentation model includes cyclic feature update. The model is used to update the loop feature according to the currently processed image during the video processing.

Wherein, the video may include multiple frames of images, and when the video is segmented, each frame of images in the video may be processed in sequence according to the cycle feature.

Optionally, the loop feature saves the feature of the processed image in the video, and can be used as a historical feature when dividing each frame of image to assist in the target segmentation of the current image, and the loop feature is used in the video multi-frame image. It is continuously updated in the process of processing. The cycle signature may be stored in memory, or in other storage devices.

Optionally, the recurring feature may specifically be a recurring dynamic embedding feature (Recurrent Dynamic Embedding, RDE), which is an embedded feature that is continuously updated and dynamically changed in the process of processing multiple frames of video images. The cyclic features can be obtained by updating the model with cyclic features. The recurrent feature update model may be a machine learning model, such as a neural network model.

When processing any frame of image, the current cycle feature can be obtained, and the image can be processed by using the current cycle feature to obtain the target segmentation result. segment the target object.

In an example, the target segmentation result may include a label of each pixel in the image, and the label is used to indicate whether the pixel belongs to the target object. For example, the number of the label can take a value between 0 and 1. If the label of any pixel in the image is 0, it means that the pixel does not belong to the target object. If the label of the pixel is 1, it means that the pixel belongs to the target object. target. By labeling each pixel in the image with 0 and 1, the pixels contained in the target object can be determined, thereby realizing pixel-level segmentation of the target object.

In another example, the target segmentation result may be used to determine multiple target objects in the image, and different numbers may be used to represent different target objects. For example, when annotating pixel points, numbers 1, 2, and 3 represent target object 1, target object 2, and target object 3, respectively, and number 0 represents an uninteresting background area.

In this embodiment, after reading the loop feature from the storage space, updating the loop feature and determining the target segmentation result may be performed simultaneously or sequentially, and the execution order of the two is not limited in this embodiment. It should be noted that regardless of the order, the cyclic features before the update can be used to determine the target segmentation result.

To sum up, the video object segmentation method provided in this embodiment can determine the video to be processed and the target object to be segmented in the video, and through the pre-built target segmentation model, based on the cycle feature, the to-be-processed objects in the video are sequentially analyzed. Each frame of image is processed, and the target object is segmented from each frame of image, wherein the target segmentation model includes a cyclic feature update model, which is used to update the cyclic feature according to the currently processed image during the video processing process, In this way, the dynamic update of the loop feature can be realized through the model, which not only enables the loop feature to fuse the information of the processed multi-frame images, but also effectively avoids the stacking of features. The size of the loop feature can not gradually increase with the progress of the video. Memory occupation, avoiding long videos affecting the segmentation effect due to insufficient memory, and reducing the cost of video target segmentation algorithms.

In one or more embodiments of the present application, optionally, the processing operation on the image in the video may include: acquiring the image to be processed and the current loop feature in the video, and extracting the image feature corresponding to the image ; wherein, the cycle feature is a feature determined according to the processed image in the video; according to the image feature and the cycle feature, determine the target segmentation result of the image to be processed, and combine the image feature and the cycle feature. The loop feature is input to the loop feature update model to obtain the updated loop feature, so as to process the next frame of the image to be processed in the video according to the updated loop feature.

Specifically, when any frame of image is processed, corresponding image features can be obtained according to the image. Optionally, the image features can be extracted by an encoder. After the image features are obtained, the image features and cyclic features can be input into the deep learning model to obtain the corresponding target segmentation results. Alternatively, operations such as matrix operations can be performed on image features and cyclic features to obtain corresponding target segmentation results. Alternatively, the deep learning model can be combined with operations such as matrix operations to comprehensively determine the target segmentation result.

In addition to determining the target segmentation result, the image features can also be used to update the cyclic features. Optionally, the image features and the current cyclic features can be input into the cyclic feature updating model to obtain the updated cyclic features.

In practical applications, the images in the video can be segmented in turn. When processing each frame of image, the target segmentation result of the current image can be determined according to the characteristics and cycle characteristics of the current image, and the cycle characteristics can be updated. When framing the image, use the updated loop feature.

In other optional implementation manners, the above target segmentation algorithm may also be used only for some images in the video. For example, only part of the images in the video can be processed by the target segmentation algorithm to obtain the target segmentation result, and the target segmentation results of the remaining images can be input by the user or obtained by other methods, or the remaining images may not be subjected to target segmentation.

To sum up, the video target segmentation method provided in this embodiment can acquire the image to be processed and the current loop feature in the video, and extract the image feature corresponding to the image, wherein the loop feature is based on the Processed image. Determined features, according to the image features and cycle features, determine the target segmentation result of the image to be processed, and input the image features and cycle features into the cycle feature update model to obtain the updated cycle feature to process the image to be processed in the next frame of the video according to the updated loop feature, so that in the process of video processing, the loop feature can be continuously updated by using the existing image feature through the model, so that the loop The feature can fuse the information of the processed image, realize the update of the cyclic feature quickly and accurately, and improve the efficiency and accuracy of the cyclic feature update.

In other optional implementations, the image and cyclic features may also be input into the cyclic feature updating model to obtain updated cyclic features.

In one or more embodiments of the present application, optionally, the target segmentation model further includes: an encoder (Encoder), a feature reading module (Read), and a decoder (Decoder); wherein, in processing the video When any frame of image is obtained, the encoder is used to extract the image feature corresponding to the image; the feature reading module is used to calculate the correlation information between the image feature and the current loop feature; the decoder is used to From the association information, the target segmentation result of the image is obtained.

Correspondingly, determining the target segmentation result of the image frame to be processed according to the image features and cycle features may include: inputting the image features and cycle features into a feature reading module to obtain the image features and cycle features. The associated information of the feature; the associated information is input to the decoder to obtain the target segmentation result of the image to be processed.

FIG. 4 is a schematic diagram of a principle for determining a target segmentation result according to an embodiment of the present application. As shown in Figure 4, the image of each frame is input to the encoder to obtain the corresponding image features. The image features and cyclic features are input to the feature reading module for calculation, and the calculation results are input to the decoder to obtain the target segmentation result.

Optionally, the input and output sizes of the image and each model can be set according to actual needs. Exemplarily, the image to be processed can be an RGB image, including data of H*W*3, and the image features of h*w*c are obtained after inputting to the encoder, and the size of the loop feature can be consistent with the image feature. To the feature reading module, after calculation, it is input to the decoder to obtain the H*W target segmentation result.

Among them, the feature reading module can be used to fuse image features and cyclic features to obtain associated information. Optionally, both the image feature and the cyclic feature can be expressed in the form of a matrix, and the feature reading module can perform operations such as matrix multiplication and/or matrix addition on the image feature and the cyclic feature, so as to realize the fusion of the two.

Based on the scheme shown in Figure 4, the historical information is maintained by a cyclic feature. The historical image frame obtains the image feature through the encoder, and together with the original cyclic feature as the input, the model is updated through a cyclic feature to obtain a new cyclic feature. When the current frame image is to be segmented, the features are obtained by encoding, and input to a feature reading module together with the cyclic features, and finally the segmentation results are obtained through the decoder.

In summary, through the encoder, the feature reading module and the decoder, the image features can be extracted, the image features and the cyclic features can be fused, and the fused features can be decoded to obtain the target segmentation result, which can effectively use the cyclic features to guide the image segmentation. Segmentation to improve the efficiency and accuracy of image segmentation.

In an optional implementation manner, a certain step size can be set, and the cyclic feature is updated after the step size requirement is met. Optionally, inputting the image feature and the cyclic feature into the cyclic feature updating model to obtain the updated cyclic feature may include: judging whether the current counter is equal to the step size; if it is equal, inputting the image feature and the cyclic feature into the model. Update the model to the cyclic feature, get the updated cyclic feature, and clear the counter to 0. If not equal, the counter can be incremented by 1 and the update loop feature is skipped.

FIG. 5 is a schematic diagram of updating a cycle feature according to a step size according to an embodiment of the present application. As shown in Figure 5, the step size can be set to θ. The cyclic feature update model may specifically be a SAM (Spatio-temporal Aggregation Module, spatio-temporal aggregation module), wherein the spatio-temporal aggregation module can aggregate image features and cyclic features in the spatio-temporal domain.

Optionally, the image feature of the t-nθth frame and the cyclic feature corresponding to the image of the t-nθth frame are input into the SAM, and the updated cyclic feature can be obtained. With the gradual advancement of video processing, the image features of t-θ and the cycle features corresponding to the images of the t-θth frame are input into the SAM, and the cycle features corresponding to the t-th frame can be obtained. Wherein, the cycle feature corresponding to the t-th frame may be the cycle feature used when determining the target segmentation result of the t-th frame image.

Exemplarily, the initial cycle feature may be determined according to the image feature of the first frame of image and the target segmentation result marked by the user. According to the image feature of the second frame and the initial cycle feature, the target segmentation result of the second frame image is determined.

Assuming that the step size is 3, since the second frame does not meet the step size requirement, the cyclic features are not updated. The third frame image uses the current cycle feature, that is, the initial cycle feature, and obtains the target segmentation result of the third frame image together with the image feature of the third frame, and the cycle feature is not updated.

When processing the fourth frame image, since the step size requirement is met, the target segmentation result of the fourth frame image can be determined according to the initial cycle feature and the image feature of the fourth frame, and the cycle feature can be updated according to the image feature of the fourth frame . The updated loop feature can be used to process the image of the fifth frame, and the target segmentation result of the image of the fifth frame can be obtained together with the image feature of the fifth frame.

In summary, the update frequency of the cyclic features can be adjusted through the step size, so that the target segmentation algorithm takes into account both efficiency and accuracy. The step size can also be input by the user. In the case of high real-time requirements, the user can set a larger step size, otherwise, a smaller step size can be set, so that the user can adjust the update frequency of the loop feature according to the scene. , to meet the needs of different scenarios and improve the flexibility of the algorithm.

FIG. 6 is an application flowchart of a target segmentation result provided by an embodiment of the present application. As shown in Figure 6, after obtaining the target segmentation result, the following operations can also be performed.

Step 601, according to the target segmentation result of the video, obtain the target video of each target object; wherein, the target video of the target object is the video obtained after removing the background of the target object in the video to be processed, Or, the target video is composed of multiple frames of images where the target object is located in the video to be processed.

Wherein, the number of target objects to be segmented may be at least one. Each target object may have a corresponding target segmentation result, and according to the target segmentation result, a target video may be extracted, and the target video may include multiple frames of images.

In an example, the extracted multi-frame images may be images where the target object is located in the original video. For example, if the original video has 1000 frames in total, of which 600 frames have the target object, the 600 frames of images can be extracted to form the target video. If the multiple frames of images in which the target object appears are continuous, one target video can be formed; if they are discontinuous, one or more target videos can be formed. For example, if the target object appears in the 1st to 200th frames, a target video can be formed. After 200 frames, the target object disappears, and the target object reappears in the 301st to 700th frames, and another target video can be formed.

By extracting the image where each target object is located, the target video of each target object can be quickly and accurately extracted from the original video, and the object-based video segmentation operation can be realized.

In another example, according to the target segmentation result, the original video can be cut out to obtain the target video after removing the background. Only the target object appears in the target video, and no other objects appear, which can quickly realize the video cutout operation for the target object. .

Step 602: Output the target video of each target object and/or at least part of the images in the target video.

Optionally, the target video and/or at least part of the images in the target video may be displayed to the user for viewing or editing by the user, and functions such as downloading the target video and/or at least part of the image in the target video may also be provided for the user. . Alternatively, the target video and/or at least part of the images in the target video may be uploaded to the cloud for storage.

In an optional implementation manner, outputting the target video of each target object and/or at least part of the image in the target video may include: displaying the target video and/or at least part of the image in the target video; The user's editing operation on the target video and/or at least part of the images in the target video generates a promotion video and/or promotion image of the target object.

Optionally, the target video can be played directly to the user, or each frame of the target video can be statically displayed to the user, and the user can edit, resize, beautify, add text and other operations on the target video or image to generate the target Promotional video or image of the object.

Exemplarily, in the commodity field, after determining the image frame where the target commodity is located in the original video through the target segmentation algorithm, the image frame can be extracted to form the target video, or the original video can be cut out to obtain the target video. The target video can be used as a promotion video or image of the target product, such as a promotional video, an advertising video, a poster, etc., to be published on an e-commerce platform or other platforms, or the target video can be in the form of a short video and published to a short video. platform. In addition, the user can edit the target video or image, and publish the edited video.

By displaying the target video or at least part of the image in the target video, and according to the user's editing operation on the target video or the at least part of the image, the promotion video or promotion image of the target object is generated, which can rely on The target segmentation results provided by the algorithm assist users in the production of target object promotion materials, improve the efficiency of material production, and improve user experience.

In another optional implementation manner, after the target video is obtained, the target video or at least some images therein may be added to the material library, or added to the material library after being filtered or edited by the user.

Optionally, the filter conditions input by the user can be obtained, the target video and/or the target image can be screened according to the filter conditions, and the screened video or image can be updated to the material library, and the material library can be used to make the target Promotional video and/or promotional image of the object. The target image may refer to an image in the target video.

Optionally, the screening conditions may be set according to actual needs, for example, may include but are not limited to: the image displayed by the target object in its entirety, the image displayed on the front of the target object, the image where the target object in a special form is located, the target object A video clip in some changing state, etc.

Exemplarily, in the field of commodities, the filtering conditions may be: an image of the introduction of the commodity, an image of the complete display of the commodity, an image of the complete packaging of the commodity, an image of which the packaging has been removed, an image of which part of the packaging has been removed, and a video clip corresponding to the use of the commodity. , product unpacking fragment, etc. The target video or target image can be selected according to the filter conditions input by the user, and the images that meet the filter conditions can be determined by simple comparison or machine learning model during selection.

Alternatively, after obtaining the target video or target image, the user may edit the target video or target image, and store the edited video or image in the material library.

Optionally, the identifier and attribute information of the target object may be stored in the material library together with the edited or filtered video and image. In practical applications, users can input query conditions, such as the identification of the target object, so as to quickly find the corresponding video or image, which can be used to make short videos, posters, promotional materials, advertising videos, etc. of the target object.

To sum up, by performing matting or image extraction operations on the original video according to the target segmentation result of the video, the target video of each target object is obtained, and the target video of each target object or at least part of the image in the target video can be output. Improve the production efficiency and accuracy of the video or image of the target object to meet the needs of use in different scenarios.

In one or more embodiments of the present application, optionally, determining the video to be processed and the target object to be segmented in the video includes: acquiring the video to be processed, and determining at least one frame in the video The reference image and the target segmentation result corresponding to each reference image.

Correspondingly, the initial cycle feature may also be determined according to the at least one frame of reference image and the corresponding target segmentation result.

Optionally, the video can be obtained through various methods, for example, shooting by a user, reading a local video, obtaining from a server, and so on. The reference image is used to determine the initial cycle feature, and the reference image may be any one frame or multiple frames in the video, for example, may be the first frame in the video.

The target segmentation result of the reference image is determined by marking the target object in the reference image by the user, or automatically marking the reference image according to a preset image of the target object. The to-be-processed image is an image other than the reference image in the video, and the corresponding target segmentation result is used to segment the target object from the image; the initial loop feature is used for the first frame to be processed image for segmentation.

In an optional implementation manner, a semi-supervised video object segmentation method can be used, and after the video is acquired, the user can mark the reference image to determine the location of the target object.

In another optional implementation manner, the reference image may be automatically annotated according to a preset image of the target object. Wherein, the preset image of the target object may be other images than the video, in particular, it may be a typical image that can accurately display the target object. For example, for a target product, the preset image may be the main image of the target product on the product details page.

After the target segmentation result of the reference image is determined, the initial cycle feature can be determined according to the reference image and the target segmentation result, and the segmentation of other images can be guided by the cycle feature.

In practical applications, the first frame image in the video can be marked by the user or according to the preset image, the target segmentation result of the first frame image can be determined, and the initial cycle feature can be generated according to the first frame image and the target segmentation result. , and then process the image of the second frame, determine the target segmentation result of the second frame according to the method provided in the foregoing embodiment, update the cyclic feature, and then process the image of the third frame according to the updated cyclic feature, and so on, until All images in the video are processed.

Alternatively, the last frame of images in the video can be marked by the user or according to a preset image, and the algorithm can segment the remaining images in sequence from the back to the front. Alternatively, the intermediate frame images in the video can be marked, and the algorithm sequentially divides the images from the intermediate frame forward and backward directions.

To sum up, by first marking at least one frame of reference image in the video by the user or according to a preset image, the target segmentation result of the reference image is determined, and according to the at least one frame of reference image and the target segmentation result corresponding to each reference image, the target segmentation result is determined. The initial loop feature can make the initial loop feature refer to a more accurate target segmentation result, guide the prediction of the target segmentation result of the subsequent image, and improve the segmentation accuracy of the subsequent image.

In one or more embodiments of the present application, optionally, the following labeling methods may be used to realize labeling of the reference image.

In the first optional labeling method, the reference image in the video can be displayed, so that the user can label the location of the target object on the reference image, and determine the target segmentation result of the reference image according to the user's labeling .

In an example, after the reference image is presented to the user, the user can draw the outline of the target object in the reference image, so that the precise position of the target object in the reference image can be determined.

In another example, the user can use image editing software to mark the target object in the reference image, and the algorithm can obtain the position of the target object from the target editing software.

In this implementation manner, the user annotates the reference image, which can improve the accuracy of the annotation, thereby improving the accuracy of video target segmentation.

In the second optional labeling method, semantic segmentation can be performed on the reference image in the video to determine at least one object in the reference image; the image area corresponding to the at least one object is respectively associated with the target object The preset images are compared to obtain the target segmentation result of the reference image.

Exemplarily, the similarity between the image area corresponding to each object and the preset image may be determined, wherein the object with the highest similarity may be determined as the target object in the reference image.

Optionally, after the object with the highest similarity is determined, the region where the object is located can be highlighted and displayed to the user, the user confirms whether the object belongs to the target object, and after the user confirms, the target segmentation result of the reference image is generated.

In this implementation manner, the reference image is marked by the preset image of the target object, which can reduce the workload of the user and improve the user experience.

In a third optional labeling method, the reference image in the video can be semantically segmented to determine at least one object in the reference image; the image area corresponding to the at least one object is displayed, and the user's The target object selected from the at least one object is described, and the target segmentation result is determined according to the image area corresponding to the target object.

Exemplarily, the image areas where different objects are located can be set to different colors, or the outline of each object can be drawn through lines, the user can directly click on the image area where an object is located, and the object can be used as the target object. , the user can also adjust the outline of the target object.

In this implementation manner, the user can directly mark the reference image through a click operation, without requiring the user to draw the outline of the reference image, thereby improving the labeling efficiency of the reference image, and taking into account the labeling accuracy and user experience.

The above several labeling methods can be used alone or in combination. For example, the first method is used for some reference images, and the second method is used for the rest of the reference images, or the same reference image can be marked with the second method and the third method respectively, and the two methods The results are fused to determine the final target segmentation result.

In one or more embodiments of the present application, optionally, for each image in the video, the following operations may be performed: semantically segment the image, determine at least one object in the image, and classify each object The corresponding image area is compared with the preset image of the target object, the similarity between each object and the target object is determined, and the highest similarity among them is taken as the similarity of the image.

From each frame of images in the video, one or more images with the highest similarity are selected as the reference image, and the target segmentation result is determined according to the object with the highest similarity with the target object in the reference image.

Exemplarily, the first frame of image contains 3 objects, and the similarity with the target object is 10%, 1%, and 80%, respectively, and the highest similarity among them, that is, 80%, can be taken as the first frame of image and the target object. similarity. Similarly, the similarity between the other frame images and the target object can be determined, and if the similarity of a certain frame image in the middle is at most 90%, this image can be used as the reference image.

In summary, each frame image in the video can be compared with the target object, and the one with the highest similarity can be determined as the reference image, which can improve the accuracy of the initial annotation without relying on the user.

In one or more embodiments of the present application, optionally, in a scenario that relies on the user to perform initial annotation, the similarity between each frame of image in the video and the previous frame of image may be calculated. If the similarity is lower than the preset similarity threshold, the next frame of the image in the two adjacent frames may be used as a reference image, and displayed to the user for marking. Alternatively, one or more groups of adjacent images with the lowest similarity may be selected from the video, and the next frame of images in the adjacent images may be used as a reference image, and displayed to the user for annotation.

To sum up, when the user annotates the initial image, the image with low similarity to the previous frame image can be selected from the video as the reference image. The effect of segmenting the reference image by the algorithm is slightly worse than that of other images segmented by the algorithm. The user annotates the reference image, which can reduce the situation that the target segmentation effect is poor due to the sudden change of the video scene, and further improve the video target segmentation. accuracy.

In one or more embodiments of the present application, optionally, determining the initial cycle feature according to the at least one frame of reference image and the corresponding target segmentation result includes: dividing the at least one frame of the reference image and the corresponding The target segmentation results are spliced, and the splicing results are input into the encoder to obtain the initial cyclic features.

Exemplarily, when there is only one frame of reference image, the reference image and the corresponding target segmentation result can be spliced and input into the encoder to obtain the initial cyclic feature. When there are multiple frames of reference images, the reference image of each frame and the target segmentation result can be spliced in turn, and input into the encoder to obtain the initial loop feature.

Optionally, the encoder may be an encoder with a fixed input dimension, or an encoder capable of accepting different input dimensions, so that an initial loop feature of a fixed size can be determined according to a non-fixed number of reference images.

The encoder can be implemented by a convolutional neural network or the like, and the encoder can be trained with a recurrent feature update model, segmentation module, and the like.

Wherein, the encoder for extracting image features and the encoder for determining initial cyclic features may be different. Optionally, the encoder that extracts image features may be denoted as the first encoder, and the encoder that determines the initial cyclic features may be denoted as the second encoder. The role of the first encoder and the second encoder are different, but both can be trained together with other modules.

To sum up, by splicing the at least one frame of the reference image and the corresponding target segmentation result, and inputting the splicing result into the encoder, the features of the reference image and the features of the target segmentation result can be extracted, and determined according to the extracted features. The initial loop feature enables the loop feature to retain the key information of the reference image, improves the accuracy of the initial loop feature, and improves the overall segmentation effect of the video.

In other optional implementation manners, when there are multiple frames of the reference image, one of the frames can be selected to determine the initial cycle feature, and the rest of the reference images may not be used to calculate the initial cycle feature, but directly use the target segmentation result of the reference image , in the process of video target segmentation through the algorithm, the processing of these reference images is skipped, and the efficiency of video target segmentation is improved.

FIG. 7 is a schematic schematic diagram of an update cycle feature provided by an embodiment of the present application. As shown in FIG. 7 , the recurrent feature update module may include: an extraction (Extracting) module, an enhancement (Enhancing) module, and a compression (Squeezing) module. Inputting the image features and the cyclic features into the cyclic feature updating model to obtain the updated cyclic features may include: inputting the image features and the cyclic features into a refining module to obtain a refining result; wherein the refining module is used for Integrate the image feature and the cyclic feature; input the refining result into an enhancement module to obtain an enhancement result; wherein, the enhancement module is used to perform a pooling operation on the refining result; compare the refining result with the enhancement result. After adding, it is input to the compression module to obtain the updated cyclic feature of preset length.

Optionally, the loop feature may be a feature of a preset length, and the preset length is maintained during the video object segmentation process. Specifically, when updating the cyclic features, the image features and the current cyclic features can be spliced and input into the extraction module. It is added to the enhanced features, and compressed to a preset length by the compression module to obtain the updated cyclic features.

To sum up, in the video target segmentation, through the refining module, the enhancement module and the compression module, a lightweight and constant size loop feature can be generated as historical information to store and guide the target segmentation. No matter how many frames are processed, the size of the loop feature does not vary. change to meet lightweight requirements.

In one or more embodiments of the present application, optionally, the extraction module may be used to perform operations such as multiplying or adding features of the image and features of the cycle to realize the fusion of the features of the image and the cycle.

Optionally, the refining module may include a downsampling layer, a convolution layer, a matrix computing unit, and a normalization layer. Inputting the image features and cyclic features into the refining module to obtain a refining result, including: down-sampling the image features and cyclic features through a downsampling layer, and inputting the down-sampled image features and cyclic features into the volume Layer layers to obtain the first feature matrix corresponding to the image feature and the second feature matrix corresponding to the cyclic feature; perform a matrix correlation operation on the first feature matrix and the second feature matrix through the matrix calculation unit, and input the operation result into the normalization unit. One layer, the refined result is obtained.

Specifically, after down-sampling the image features and the cyclic features, they are input to the convolutional layer respectively. The convolutional layer can extract the matrix corresponding to the image features, and can also extract the matrix corresponding to the cyclic features. These two matrices represent the image features and For the deeper features of cyclic features, performing matrix correlation operations on two matrices such as matrix addition and/or matrix multiplication, etc., can realize the fusion of image features and cyclic features, and then perform a normalization operation to obtain a refined result.

In an implementation manner, the enhancement module may specifically be used to perform pyramid pooling on the refining result. Optionally, the enhancement module may include a pyramid pooling layer, such as ASPP (atrous spatial pyramid pooling, atrous spatial pyramid pooling), which can better perform pyramid pooling on the extraction result to obtain an enhancement result of a fixed size.

In summary, the extraction module can enhance the feature correlation in the spatiotemporal domain through matrix correlation calculation, and better realize the fusion of image features and cyclic features. The enhancement module can obtain different scales of receptive fields through ASPP, which are different in space and time. Feature screening capability to further improve the target segmentation effect of video.

In one or more embodiments of the present application, optionally, the model can be trained by using training samples first, and the trained model can be used to perform target segmentation on the video to be processed.

In an optional implementation manner, training samples may be obtained, and the training samples include training videos and target segmentation results corresponding to each image in the training videos. During the training process, at least one frame of reference image is selected from the training video, and other images except the at least one frame of reference image in the training video are images to be processed; The target segmentation result is perturbed to obtain a noisy target segmentation result, and an initial loop feature is determined according to the noisy target segmentation result. For any to-be-processed image in the training video, perform the following operations: predict the target segmentation result of the image based on the current cycle feature, so as to update the cycle feature update model according to the prediction result and the target segmentation result in the training sample perturb the image features corresponding to the image to obtain image features with noise, and update the cyclic features according to the image features with noise, so as to process the middle and lower parts of the training video according to the updated cyclic features An image to be processed.

Optionally, in the training process, the loss value may be determined according to the prediction result corresponding to the image to be processed and the actual result, and the model parameters are updated according to the loss value so that the prediction result is close to the actual result. The prediction result refers to the target segmentation result predicted by the algorithm, and the actual result is the target segmentation result corresponding to the image in the training sample.

In order to improve the anti-noise ability of the algorithm, perturbation can be added to update the recurrent features during training. Taking the first frame image as the reference image as an example, the target segmentation result of the first frame image can be perturbed to obtain the target segmentation result with noise and construct the initial loop feature. When processing subsequent images, the image features can be perturbed and the recurrent features can be updated based on the noisy image features.

Optionally, the step size can be set to 2, then the disturbance is not added when processing the 2nd, 4th, and 6th frames, and the disturbance can be added and the cyclic feature can be updated when processing the 3rd, 5th, and 7th frames.

When the model is used, the video to be processed is continuously segmented. Since the historical segmentation results are not necessarily correct, a lot of noise may be introduced, resulting in poor video segmentation results. Therefore, during training, by constructing interference input, you can Ensure that the model learns to overcome the noise problem in feature storage, so that the model has better anti-noise ability, overcomes the noise caused by video sequences in practical applications, and improves the effect and performance of target segmentation.

Optionally, each module used by the target segmentation algorithm, such as the first encoder, the second encoder, the decoder, the cyclic feature update model, etc., may be trained together.

In other optional implementation manners, the structure of the above model may also be adjusted according to actual needs, and some modules may be added, reduced or replaced. For example, the feature reading module and the decoder can be combined into one module, the recurrent feature update model can be a convolutional neural network model of other structures, and so on.

FIG. 8 is a schematic flowchart of another video object segmentation method provided by an embodiment of the present application. As shown in Figure 8, the method may include:

Step 801: Determine the video to be processed and the target commodity to be segmented in the video.

Optionally, the video to be processed may be a video corresponding to the target product. The video corresponding to the target product may be shot by the user, or may be a video corresponding to the target product obtained from other devices, for example, a video of the target product on the product details page obtained from the server.

The product details page may be a page provided by an e-commerce platform for displaying products, and the page may display the name, price, specifications, comments, details, etc. of the product, and may also display the video corresponding to the product.

Step 802: Process each frame of image to be processed in the video in turn based on the cyclic feature through a pre-built target segmentation model, and segment the target product from each frame of image; wherein, the target segmentation model includes cyclic feature update. The model is used to update the loop feature according to the currently processed image during the video processing.

Optionally, each image to be processed in the video may be sequentially processed as follows: according to the image feature and cycle feature corresponding to the current image to be processed, determine the target segmentation result of the image, and compare the image feature with the cyclic feature. The loop feature is input to the loop feature update model, and the updated loop feature is obtained, so as to process the next image to be processed in the video according to the updated loop feature; wherein, the target segmentation result is used to represent the target product in the position in the image.

Optionally, by referring to any of the foregoing embodiments, the target object therein may be replaced with a target commodity, and the images in the video may be processed to obtain target segmentation results of each image.

Step 803: Generate a target video and/or a target image of the target product according to the target segmentation result of the video.

Optionally, the target image may be the image where the target object is located in the video to be processed, or an image obtained after removing the background from the image where the target object is located; the target video may be an image obtained by A video composed of target images.

Optionally, each image can be cut out according to the target segmentation result, and the target product in each image can be extracted to form a target image or target video. The image or video does not contain the background in the original video and can be provided as a video production material. Editing for merchants, or extracting images or video clips in which the target product appears from the original video, as the target image or target video.

Optionally, the filter conditions input by the user can also be obtained, and the target images are screened according to the filter conditions; the images obtained by the screening are updated to the material library, and the material library is used to make the promotion video and/or the target product. or promotional images.

By screening the generated target images based on the needs of users, and using the screened images as materials for product promotion, the product materials can be accurately extracted from the original video to meet the personalized needs of users.

In an optional implementation manner, the reference image in the video can also be displayed, so that the user can mark the location of the target product on the reference image, and determine the target segmentation result of the reference image according to the user's label; And/or, obtain the main image corresponding to the target product on the product detail page, perform semantic segmentation on the reference image in the video, determine at least one object in the reference image, and divide the image area corresponding to the at least one object. Comparing with the main image of the target product respectively, the target segmentation result of the reference image is obtained; wherein, the reference image and the corresponding target segmentation result are used to determine the initial cycle feature.

Exemplarily, the first frame image can be used as the reference image, and the user can mark the position of the target product in the first frame image, or the main image of the target product can be obtained from the product details page, and the main image and the first frame image can be compared. Compare each object in the target product to determine the target product, so as to obtain the target segmentation result, which is used to construct the initial cycle feature.

To sum up, the video target segmentation method provided in this embodiment can determine the video to be processed and the target commodity to be segmented in the video, and through the pre-built target segmentation model, based on the cycle feature, the to-be-processed objects in the video are sequentially analyzed. Each frame of image is processed, and the target product is segmented from each frame of image, wherein the target segmentation model includes a cyclic feature update model, which is used to update the cyclic feature according to the currently processed image during the video processing process, According to the target segmentation result of the video, the target video and/or target image of the target product is generated. Since the loop feature may not gradually increase with the progress of the video, the memory occupation is effectively reduced, and the segmentation of long videos due to insufficient memory is avoided. It reduces the cost of the video target segmentation algorithm, improves the target segmentation effect of the product video, and improves the production efficiency and accuracy of the product video or image.

In the field of video conferencing, an embodiment of the present application also provides a video target segmentation method, including:

Determine pending meeting videos and target participants;

Through the pre-built target segmentation model, each frame of images to be processed in the conference video is sequentially processed based on the cyclic features, and the target participants are segmented from each frame of images; wherein, the target segmentation model includes cyclic feature updates A model for updating the loop feature according to the currently processed image during the video processing;

According to the target segmentation result, the background of the conference video is blurred or a virtual background is added.

Exemplarily, the device or server used by the target participant of the video conference can send the video after the background blur or the virtual background is added to the device used by other participants to play the video to other participants. Effectively hide the user's environment, improve meeting efficiency, and protect user privacy. Optionally, the virtual background can be set by the user to meet the individual needs of the user.

In the field of short videos, an embodiment of the present application also provides a video target segmentation method, including:

Determine the long video to be processed and the target person to be segmented in the video;

Through the pre-built target segmentation model, each frame of images to be processed in the long video is sequentially processed based on the cyclic feature, and the target person is segmented from each frame of image; wherein, the target segmentation model includes a cyclic feature update model, for updating the loop feature according to the currently processed image during the video processing;

According to the target segmentation result, a short video corresponding to the target person is generated.

The target character may be a character designated by the user, or may be a character that meets certain requirements, for example, a character who performs certain actions in the video.

Exemplarily, in a dance video, dancing characters can be segmented from the video and used as materials to produce a short video or other types of videos, which can effectively improve the production effect of the short video.

In each embodiment of the present application, the execution body of the method may be set according to actual needs. Exemplarily, it may be performed by a terminal device, or may be performed by a server, or some steps may be performed by a terminal device, and some steps may be performed by a server.

Corresponding to the above video object segmentation method, the embodiment of the present application further provides a video object segmentation device, and the device may include:

A determination module, used to determine the video to be processed and the target object to be segmented in the video;

The processing module is used for sequentially processing each frame of images to be processed in the video based on the cycle feature through a pre-built target segmentation model, and segmenting the target object from each frame image; wherein, the target segmentation model includes a cycle The feature update model is used to update the loop feature according to the currently processed image during the video processing.

In one or more embodiments of the present application, optionally, the target segmentation model further includes: an encoder, a feature reading module, and a decoder;

Wherein, when processing any frame of image in the video, the encoder is used to extract the image feature corresponding to the image;

The feature reading module is used to calculate the associated information of the image feature and the current loop feature;

The decoder is configured to obtain a target segmentation result of the image according to the associated information.

In one or more embodiments of the present application, optionally, the number of the target object is at least one, and the processing module is further configured to:

According to the target segmentation result of the video, the target video of each target object is obtained; wherein, the target video of the target object is the video obtained after removing the background of the target object from the video to be processed, or, the The target video is composed of multiple frames of images where the target object is located in the video to be processed;

A target video of each target object and/or at least part of images in the target video is output.

In one or more embodiments of the present application, optionally, when outputting the target video of each target object and/or at least part of the image in the target video, the processing module is specifically configured to:

displaying the target video and/or at least part of the images in the target video;

According to the user's editing operation on the target video and/or at least part of the images in the target video, a promotion video and/or promotion image of the target object is generated.

In one or more embodiments of the present application, optionally, the processing operations performed by the processing module on the images in the video include:

Obtain the image to be processed in the video and the current loop feature, and extract the image feature corresponding to the image; wherein, the loop feature is a feature determined according to the processed image in the video;

Determine the target segmentation result of the image to be processed according to the image features and cycle features, and input the image features and cycle features into the cycle feature update model to obtain the updated cycle features, so as to obtain the updated cycle features according to the updated cycle features The feature processes the next frame in the video to be processed.

In one or more embodiments of the present application, optionally, the determining module is specifically configured to:

Obtain the video to be processed, and determine at least one frame of reference image in the video and the target segmentation result corresponding to each reference image;

The method further includes: determining an initial loop feature according to the at least one frame of reference image and the corresponding target segmentation result;

Wherein, the target segmentation result of the reference image is determined by annotating the target object in the reference image by the user, or is determined by automatically labeling the reference image according to a preset image of the target object;

The images to be processed in the video are other images except the reference image; the initial cycle feature is used to segment the images to be processed in the first frame.

In one or more embodiments of the present application, optionally, the determining module is further configured to perform at least one of the following:

Displaying the reference image in the video, so that the user can mark the location of the target object on the reference image, and determine the target segmentation result of the reference image according to the user's label;

Semantically segment the reference image in the video to determine at least one object in the reference image; and compare the image area corresponding to the at least one object with the preset image of the target object to obtain the reference image The target segmentation result;

Semantically segment the reference image in the video to determine at least one object in the reference image; display the image area corresponding to the at least one object, obtain the target object selected by the user from the at least one object, and perform The image area corresponding to the target object determines the target segmentation result.

In one or more embodiments of the present application, optionally, when the processing module inputs the image feature and the cyclic feature into the cyclic feature update model to obtain the updated cyclic feature, the processing module is specifically used for:

Inputting the image features and the cyclic features into a refining module to obtain a refining result; wherein the refining module is used to fuse the image features and the cyclic features;

Inputting the refining result to an enhancement module to obtain an enhancement result; wherein, the enhancement module is used to perform a pooling operation on the refining result;

The refined result and the enhancement result are added and input to the compression module to obtain the updated cyclic feature of the preset length.

In one or more embodiments of the present application, optionally, the refining module includes a downsampling layer, a convolution layer, a matrix computing unit, and a normalization layer; input the image features and cyclic features into the refining module , get the refined results, including:

The image features and cyclic features are down-sampled through the down-sampling layer, and the down-sampled image features and cyclic features are respectively input to the convolution layer to obtain a first feature matrix corresponding to the image features and a second feature matrix corresponding to the cyclic features. feature matrix;

A matrix correlation operation is performed on the first feature matrix and the second feature matrix by a matrix computing unit, and the operation result is input to the normalization layer to obtain a refined result. The video object segmentation apparatus provided in the embodiments of the present application can be used to implement the technical solutions of the embodiments shown in FIG. 1 to FIG. 7 , and the implementation principles and technical effects thereof are similar, and details are not described herein again in this embodiment.

The embodiment of the present application further provides another video object segmentation apparatus, and the apparatus may include:

a target commodity determination module, used to determine the video to be processed and the target commodity to be segmented in the video;

The commodity video processing module is used to sequentially process each frame of images to be processed in the video based on the cycle feature through a pre-built target segmentation model, and segment the target commodity from each frame of image; wherein, the target segmentation model Including a cyclic feature update model, used for updating the cyclic feature according to the currently processed image in the video processing process;

A generating module, configured to generate a target video and/or a target image of the target product according to the target segmentation result of the video.

In one or more embodiments of the present application, optionally, the target image is the image where the target object is located in the video to be processed, or, after removing the background from the image where the target object is located the resulting image; the generation module is also used to:

Obtain the filter conditions input by the user, and filter the target image according to the filter conditions;

The filtered images are updated to the material library, where the material library is used to make promotional videos and/or promotional images of the target product. The video object segmentation apparatus provided in the embodiment of the present application can be used to implement the technical solution of the embodiment shown in FIG. 8 , and the implementation principle and technical effect thereof are similar, and are not repeated here in this embodiment.

FIG. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 9 , the electronic device of this embodiment may include:

at least one processor 901; and

a memory 902 in communication with the at least one processor;

Wherein, the memory 902 stores instructions that can be executed by the at least one processor 901, and the instructions are executed by the at least one processor 901, so that the electronic device executes any of the foregoing embodiments. method.

Optionally, the memory 902 may be independent or integrated with the processor 901 .

For the implementation principle and technical effect of the electronic device provided in this embodiment, reference may be made to the foregoing embodiments, and details are not repeated here.

Embodiments of the present application further provide a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium, and when a processor executes the computer-executable instructions, the method described in any of the foregoing embodiments is implemented.

Embodiments of the present application further provide a computer program product, including a computer program, which implements the method described in any of the foregoing embodiments when the computer program is executed by a processor.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated. to another system, or some features can be ignored, or not implemented.

The above-mentioned integrated modules implemented in the form of software functional modules may be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, and include several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in the various embodiments of the present application.

It should be understood that the above-mentioned processor may be a central processing unit (Central Processing Unit, CPU for short), and may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP for short), application specific integrated circuit (Application Specific Integrated Circuit, Referred to as ASIC) and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the application can be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one magnetic disk memory, and may also be a U disk, a removable hard disk, a read-only memory, a magnetic disk or an optical disk, and the like.

The above-mentioned storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Except programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. A storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

An exemplary storage medium is coupled to the processor, such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium may be located in application specific integrated circuits (Application Specific Integrated Circuits, ASIC for short). Of course, the processor and the storage medium may also exist in the electronic device or the host device as discrete components.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

From the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course hardware can also be used, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of this application.

The above are only the preferred embodiments of the present application, and are not intended to limit the scope of the patent of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present application, or directly or indirectly applied in other related technical fields , are similarly included within the scope of patent protection of this application.

Claims (14)

1. A method for segmenting video objects, comprising:

determining a video to be processed and a target object to be segmented in the video;

sequentially processing each frame image to be processed in the video based on the cycle characteristics through a pre-constructed target segmentation model, and segmenting a target object from each frame image; the target segmentation model comprises a cyclic feature updating model used for updating the cyclic features according to the currently processed image in the video processing process.

2. The method of claim 1, wherein the object segmentation model further comprises: the device comprises an encoder, a characteristic reading module and a decoder;

when any frame of image in a video is processed, the encoder is used for extracting image characteristics corresponding to the image;

the characteristic reading module is used for calculating and obtaining the associated information of the image characteristic and the current cycle characteristic;

and the decoder is used for obtaining a target segmentation result of the image according to the associated information.

3. The method of claim 1, wherein the number of target objects is at least one, the method further comprising:

obtaining target videos of all target objects according to the target segmentation result of the video; the target video of the target object is a video obtained after the background of the target object is removed from the video to be processed, or the target video is composed of a plurality of frames of images where the target object is located in the video to be processed;

and outputting the target video of each target object and/or at least partial images in the target video.

4. The method according to claim 3, wherein outputting the target video of each target object and/or at least a partial image of the target video comprises:

displaying the target video and/or at least part of images in the target video;

and generating a promotion video and/or a promotion image of the target object according to the editing operation of the target video and/or at least part of images in the target video by the user.

5. The method according to any of claims 1-4, wherein the processing operation on the images in the video comprises:

acquiring an image to be processed and a current cycle characteristic in a video, and extracting an image characteristic corresponding to the image; wherein the cyclic features are features determined from processed images in the video;

and determining a target segmentation result of the image to be processed according to the image characteristic and the cycle characteristic, inputting the image characteristic and the cycle characteristic into a cycle characteristic updating model to obtain an updated cycle characteristic, and processing the image to be processed of the next frame in the video according to the updated cycle characteristic.

6. The method of claim 5, wherein determining a video to be processed and a target object to be segmented in the video comprises:

acquiring a video to be processed, and determining at least one frame of reference image in the video and a target segmentation result corresponding to each reference image;

the method further comprises the following steps: determining an initial cycle characteristic according to the at least one frame of reference image and the corresponding target segmentation result;

the target segmentation result of the reference image is determined by labeling a target object in the reference image by a user, or is determined by automatically labeling the reference image according to a preset image of the target object;

the image to be processed in the video is other images except the reference image; the initial cyclic feature is used to segment the first frame of the image to be processed.

7. The method of claim 6, further comprising at least one of:

displaying a reference image in the video to enable a user to mark the position of a target object on the reference image, and determining a target segmentation result of the reference image according to the mark of the user;

performing semantic segmentation on a reference image in the video, and determining at least one object in the reference image; comparing the image area corresponding to the at least one object with a preset image of a target object respectively to obtain a target segmentation result of the reference image;

performing semantic segmentation on a reference image in the video, and determining at least one object in the reference image; and displaying an image area corresponding to the at least one object, acquiring a target object selected by a user from the at least one object, and determining a target segmentation result according to the image area corresponding to the target object.

8. The method of claim 5, wherein inputting the image features and the cyclic features into a cyclic feature update model, resulting in updated cyclic features, comprises:

inputting the image features and the cycle features into a refining module to obtain a refining result; the refining module is used for fusing the image features and the cycle features;

inputting the refined result into an enhancement module to obtain an enhancement result; the enhancement module is used for performing pooling operation on the extraction result;

and adding the refined result and the enhanced result, and inputting the result into a compression module to obtain the updated cycle characteristic with the preset length.

9. The method of claim 8, wherein the refinement module comprises a downsampling layer, a convolution layer, a matrix computation unit, a normalization layer; inputting the image features and the cycle features into a refining module to obtain a refining result, wherein the refining result comprises:

the image features and the cyclic features are downsampled through a downsampling layer, and the downsampled image features and the downsampled cyclic features are input into a convolutional layer respectively to obtain a first feature matrix corresponding to the image features and a second feature matrix corresponding to the cyclic features;

and performing matrix correlation operation on the first feature matrix and the second feature matrix through a matrix calculation unit, and inputting an operation result into a normalization layer to obtain an extraction result.

10. A method for segmenting video objects, comprising:

determining a video to be processed and a target commodity to be segmented in the video;

sequentially processing each frame image to be processed in the video based on the cycle characteristics through a pre-constructed target segmentation model, and segmenting a target commodity from each frame image; the target segmentation model comprises a cyclic feature updating model used for updating the cyclic features according to the currently processed image in the video processing process;

and generating a target video and/or a target image of the target commodity according to the target segmentation result of the video.

11. The method according to claim 10, wherein the target image is an image of the target commodity in the video to be processed, or an image obtained by removing a background from the image of the target commodity; the method further comprises the following steps:

obtaining a screening condition input by a user, and screening the target image according to the screening condition;

and updating the image obtained by screening to a material library, wherein the material library is used for manufacturing a promotion video and/or a promotion image of the target commodity.

12. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor;

wherein the memory stores instructions executable by the at least one processor to cause the electronic device to perform the method of any of claims 1-11.

13. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a processor, implement the method of any one of claims 1-11.

14. A computer program product comprising a computer program, characterized in that the computer program realizes the method according to any of claims 1-11 when executed by a processor.

Images