CN117201837A - Video generation method, device, electronic equipment and storage medium - Google Patents

Abstract

Embodiments of the present application disclose a video generation method, device, electronic device and storage medium. The method includes: obtaining a target detection result corresponding to the target multimedia data, and the target detection result is used to indicate the target object included in the target multimedia data; according to the target According to the detection results, the target template video is determined; the target template video corresponds to the target style; based on the target template video, the target video generation model is determined; the target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video; through the target video The generation model processes the target template video according to the target detection results to generate a target video, the video frames of the target video correspond to the target style, and the video frames of the target video include the target object. Implementing the embodiments of the present application can improve the efficiency of generating style videos.

Description

Video generation method, device, electronic equipment and storage medium

Technical field

This application relates to the field of multimedia technology, and specifically to a video generation method, device, electronic equipment and storage medium.

Background technique

With the popularity of the Internet and social media, video has become the main way for people to share information, stories and experiences. People are keen to produce videos of various styles to meet different expression needs. These videos usually have obvious style characteristics in terms of vision, hearing, aesthetics, etc. Currently, users generally refer to ready-made video templates and manually create videos of various styles. However, the threshold for manually producing videos is higher and takes longer. Therefore, how to improve the efficiency of generating style videos has become an urgent technical problem that needs to be solved.

Contents of the invention

Embodiments of the present application disclose a video generation method, device, electronic device and storage medium, which can improve the video quality of the generated style video and improve the efficiency of generating the style video.

An embodiment of the present application discloses a video generation method, which method includes:

Obtaining a target detection result corresponding to the target multimedia data, where the target detection result is used to indicate a target object included in the target multimedia data;

According to the target detection result, a target template video is determined; the target template video corresponds to the target style;

Determine a target video generation model according to the target template video; the target video generation model is obtained by adjusting a pre-trained video generation model according to the target template video;

The target template video is processed according to the target detection result by the target video generation model to generate a target video, the video frames of the target video correspond to the target style, and the video frames of the target video include The target object.

An embodiment of the present application discloses a video generation method, which is applied to a terminal device. The method includes:

In response to the selection operation, determining the selected target multimedia data;

Perform target detection on the target multimedia data through a target detection model to obtain a target detection result; the target detection result is used to indicate the target object included in the target multimedia data;

Obtain a target video generated based on the target detection result, the video frames of the target video correspond to the target style, and the video frames of the target video include the target object;

Wherein, the target video is generated by processing a target template video according to the target detection result through a target video generation model; the target template video corresponds to the target style, and the target template video is determined according to the target detection result. ; The target video generation model is obtained by adjusting a pre-trained video generation model based on the target template video.

An embodiment of the present application discloses a video generation device, which includes:

An acquisition module, configured to acquire a target detection result corresponding to the target multimedia data, where the target detection result is used to indicate a target object included in the target multimedia data;

A template determination module, configured to determine a target template video according to the target detection result; the target template video corresponds to the target style;

A model determination module, configured to determine a target video generation model based on the target template video; the target video generation model is obtained by adjusting a pre-trained video generation model based on the target template video;

A video generation module, configured to process the target template video according to the target detection result through the target video generation model to generate a target video, the video frames of the target video correspond to the target style, and the target The target object is included in the video frame of the video.

An embodiment of the present application discloses a video generation device, which is applied to terminal equipment. The device includes:

a response module, configured to determine the selected target multimedia data in response to the selection operation;

A detection module, configured to perform target detection on the target multimedia data through a target detection model to obtain a target detection result; the target detection result is used to indicate a target object included in the target multimedia data;

An acquisition module, configured to acquire a target video generated based on the target detection result, the video frame of the target video corresponds to the target style, and the video frame of the target video includes the target object; wherein, the target video is The target template video is processed and generated by the target video generation model according to the target detection result; the target template video corresponds to the target style, and the target template video is determined according to the target detection result; the target video generation model It is obtained by adjusting the pre-trained video generation model according to the target template video.

An embodiment of the present application discloses an electronic device, including a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor can implement any of the methods disclosed in the embodiment of the present application. A method in an embodiment.

An embodiment of the present application discloses a computer-readable storage medium that stores a computer program, wherein the computer program causes the computer to execute the method in any embodiment disclosed in the embodiment of the present application.

Compared with related technologies, embodiments of the present application disclose a video generation method, device, electronic device and storage medium, which have the following beneficial effects:

Obtain the target detection result corresponding to the target multimedia data, and the target detection result is used to indicate the target object included in the target multimedia data; according to the target detection result, determine the target template video corresponding to the target style; according to the target template video, determine the target video generation model, The target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video; the target video generation model processes the target template video according to the target detection results to generate the target video, and the video frames of the target video correspond to the target style. , and the target object is included in the video frame of the target video. In the embodiment of the present application, the corresponding target template video can be matched based on the target detection result corresponding to the target multimedia data, so that the target template video is processed according to the target detection result through the target video generation model corresponding to the target template video, and the target video is generated. By using the target video generation model adjusted by the target template video to generate the target video, the generated style video is more targeted, the video quality of the generated style video can be improved, and the efficiency of generating the style video can be improved.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. Those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of an application scenario of a video generation method in an embodiment;

Figure 2 is a schematic flowchart of a video generation method in one embodiment;

Figure 3 is a schematic flow chart of a video generation method in another embodiment;

Figure 4 is a schematic flow chart of a video generation method in another embodiment;

Figure 5 is a timing diagram of a video generation method in an embodiment;

Figure 6 is a timing diagram of a video generation method in another embodiment;

Figure 7 is a timing diagram of a video generation method in another embodiment;

Figure 8 is a schematic structural diagram of a video generation device in one embodiment;

Figure 9 is a schematic structural diagram of a video generation device in another embodiment;

Figure 10 is a schematic structural diagram of an electronic device in an embodiment.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

It should be noted that the terms "including" and "having" and any variations thereof in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes Other steps or units inherent to such processes, methods, products or devices.

Embodiments of the present application disclose a video generation method, device, electronic device and storage medium, which can improve the efficiency of generating style videos. Each is explained in detail below.

Please refer to Figure 1, which is a schematic diagram of an application scenario of a video generation method in an embodiment. As shown in FIG. 1 , the server 10 can communicate with one or more terminal devices 20 .

The server 10 may be a computer that provides computing, storage, network and other resource services. Alternatively, the server 10 may be a cloud server. In some embodiments, the terminal device 20 may communicate with the cloud server through Hyper Text Transfer Protocol (HTTP)/Hypertext Transfer Protocol Secure (HTTPS), or the terminal device 20 may communicate through a representative The Representational State Transfer Application Programming Interface (RESTAPI) communicates with the cloud server, so that the terminal device 20 can communicate with the cloud server through web browsing and applications, thereby obtaining data from the cloud server or uploading data to the server. .

The terminal device 20 may include but is not limited to mobile phones, wearable devices, tablet computers, vehicle-mounted terminals and other electronic devices, which are not limited in the embodiments of this application.

In some embodiments, the terminal device 20 can determine the selected target multimedia data in response to the selection operation; perform target detection on the target multimedia data through the target detection model to obtain a target detection result; the target detection result is used to indicate that the target multimedia data includes The target object; obtain the target video generated based on the target detection result, the video frame of the target video corresponds to the target style, and the video frame of the target video includes the target object; among them, the target video is generated based on the target detection result through the target video generation model. The template video is processed and generated; the target template video corresponds to the target style, and the target template video is determined based on the target detection results; the target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video.

It should be noted that the method in the embodiment of the present application can be applied to electronic equipment, and the electronic equipment can be a server or a terminal device, and is not specifically limited.

In some embodiments, the terminal device 20 may send the target detection result to the server 10 , so that the server 10 processes the target detection result and generates a target video. The server 10 can receive the target detection result sent by the terminal device 20. The server 10 determines the target template video according to the target detection result, and the target template video corresponds to the target style; the server 10 determines the target video generation model according to the target template video, and the target video generation model is The pre-trained video generation model is adjusted according to the target template video; the server 10 processes the target template video according to the target detection results through the target video generation model to generate the target video, the video frames of the target video correspond to the target style, and the target The video frame includes the target object. The server 10 may send the target video to the terminal device 20, and the terminal device 20 may display the received target video.

In other embodiments, the terminal device 20 can directly process the target detection results and generate the target video. After obtaining the target detection result corresponding to the target multimedia data, the terminal device 20 can determine the target template video according to the target detection result, and the target template video corresponds to the target style; the terminal device 20 determines the target video generation model according to the target template video, and the target video The generation model is obtained by adjusting the pre-trained video generation model based on the target template video; the terminal device 20 processes the target template video according to the target detection results through the target video generation model to generate the target video, and the video frames of the target video correspond to the target. style, and the target object is included in the video frames of the target video. The terminal device 20 can display the generated target video.

Please further refer to Figure 2, which is a schematic flow chart of a video generation method in one embodiment; the video generation method can be applied to an electronic device, and the electronic device can be the above-mentioned terminal device or the above-mentioned server. As shown in Figure 2, the video generation method may include the following steps:

201. Obtain the target detection result corresponding to the target multimedia data. The target detection result is used to indicate the target object included in the target multimedia data.

In some embodiments, the terminal device may respond to the selection operation, determine the selected target multimedia data, and perform target detection on the target multimedia data through a target detection model to obtain a target detection result. The terminal device can send the target detection results to the server, so that the target detection results are processed by the server and generate a target video; or, the terminal device can directly process the target detection results and generate a target video.

Optionally, the terminal device 20 can display one or more multimedia data, and the selection operation can be a user operation that triggers a virtual control corresponding to the multimedia data, such as clicking or double-clicking the virtual control corresponding to the multimedia data, thereby determining the selected target. Multimedia data.

In some embodiments, target detection models may include, but are not limited to, Region-based Convolutional Neural Networks (RNN) models, Single Shot Detector (SSD) models, YOLO (You Only Look Once ) model, etc. The terminal device 20 performs target detection on the target multimedia data through the target detection model, identifies the target object in the target multimedia data, and generates a bounding box for the target object. The bounding box can indicate the position of the target object in the target multimedia data, and can Generate a category label corresponding to the target object. The category label can indicate which object category the target object belongs to. The object category of the target object can include people, vehicles, animals, etc. Further, the animal category can also be divided into dogs, cats, rabbits, etc.

The target detection result may indicate the target object included in the target multimedia data. Optionally, the target detection result may refer to the content corresponding to the target object in the target multimedia data or the corresponding content in the bounding box.

In some embodiments, the target detection results may also include but are not limited to the object category corresponding to the target object, the position of the target object in the target multimedia data, etc.; or, the target detection results may include the object image corresponding to the target object, and the object image is The image area of the target object extracted from the target multimedia data.

The target multimedia data may include but is not limited to image data, video data, etc. For example, if the target multimedia data selected by the user is a pet image, the terminal device can perform target detection on the pet image to obtain a target detection result, and the target detection result can indicate the pet included in the target multimedia data; if the target multimedia data selected by the user For pet videos, the terminal device can perform frame-by-frame target detection on the pet video to obtain the target detection results corresponding to each video frame in the pet video. Since the pet in the pet video is dynamic and the angle of the pet also changes, each frame in the pet video The target detection results corresponding to the video frames can indicate multiple angles of the pet included in the target multimedia data, such as the front, back, side, etc. of the pet. Therefore, the terminal device performs frame-by-frame target detection on video data, and can also detect and judge the target object, and can enrich various angle information of the target object to a certain extent, so that the target video generation model can include The angle information of the target object is used to generate a style video that is more accurate and closer to the real target object.

202. According to the target detection results, determine the target template video, and the target template video corresponds to the target style.

The target style may refer to the video style corresponding to the target template video. Optionally, the target style may include but is not limited to the color tone style, filter special effects style, animation special effects style, action style, etc. of the video, and is not specifically limited.

In some embodiments, the target detection result includes an object category corresponding to the target object. The electronic device can determine the target template video corresponding to the object category according to the object category corresponding to the target object. The electronic device can pre-store correspondences between multiple object categories and multiple template videos, and store template videos corresponding to the multiple object categories. The electronic device determines the target template video corresponding to the object category from the plurality of template videos according to the object category corresponding to the target object. For example, if the object category is cat, the corresponding template video can be a cat dancing video, and the corresponding style can be dancing movements; if the object category is dog, the corresponding template video can be a dog singing video, and the corresponding style can be singing movements.

203. Determine the target video generation model based on the target template video. The target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video.

The target video generation model may refer to a trained video generation model obtained by adjusting a pre-trained video generation model based on the target template video.

In some embodiments, the video generation model can be trained using a large number of video data sets in the pre-training stage. The pre-trained video generation model has the ability to generate videos. In order to further improve the pertinence and video quality of generated style videos, multiple different template videos can be used to adjust the model parameters of the pre-trained video generation model, thereby differentiating multiple different video generation models. Each video generation model is adjusted based on its corresponding template video. The adjusted video generation model has the ability to generate videos of the same style as its corresponding template video.

The target template video is any template video among multiple different template videos, and the target video generation model is a video generation model obtained by adjusting the pre-trained video generation model using the target template video. The adjusted target video generation model has the ability to generate videos of the same style as the target template video.

The electronic device can pre-store correspondences between multiple template videos and multiple video generation models. For example, video generation model A is adjusted based on template video A, and video generation model B is adjusted based on template video B. Therefore, after the target template video is determined, the target video generation model corresponding to the target template video can be determined based on the correspondence between the multiple template videos and the multiple video generation models.

In some embodiments, the electronic device uses multiple video frames contained in the target template video to adjust model parameters of the pre-trained video generation model. Specifically, the predicted video frame generated by the pre-trained video generation model and the target template video can be calculated. The training loss between multiple video frames is included, thereby adjusting the parameters of the pre-trained video generation model according to the training loss to obtain the target video generation model.

In some embodiments, the target video generation model may be an artificial intelligence model, which may include but is not limited to a Generative Adversarial Networks (GAN) model, a Variational Auto-Encoder (VAE) model, a diffusion model ( Diffusion Models, DM), etc.

Through the above embodiments, the template video is used to adjust the pre-trained video generation model to obtain a video generation model corresponding to the template video, so that the generated style video is more targeted, the video quality of the generated style video can be improved, and the generation of the style video can be improved. Style video efficiency.

204. Use the target video generation model to process the target template video according to the target detection result to generate the target video.

The electronic device uses the target detection results and the target template video as inputs to the target video generation model, processes the target template video according to the target detection results through the target video generation model, and outputs the corresponding target video. The video frames of the target video correspond to the target style, and The target object is included in the video frames of the target video.

In some embodiments, the target style may include a target tone, the tone of the video frame of the target video is consistent with the target tone, and the video frame of the target video includes the target object.

In some embodiments, the target style may include a target action, and the video frames of the target video include a target object performing the target action. For example, if the target detection result indicates that the target object included in the target multimedia data is a dog and the target action is dancing, then the video frame of the target video output by the target video generation model will contain a dancing cat.

In some embodiments, the electronic device can perform feature extraction on the target template video to obtain video feature data corresponding to the target template video; input the video feature data into the control model, and use the control model to generate video generation control information based on the video feature data, and The video generation control information is input to the target video generation model; the target video generation model processes the target template video according to the target detection results and the video generation control information to generate the target video.

In the embodiment of the present application, the corresponding target template video can be matched based on the target detection result corresponding to the target multimedia data, so that the target template video is processed according to the target detection result through the target video generation model corresponding to the target template video, and the target video is generated. By using the target video generation model adjusted by the target template video to generate the target video, the generated style video is more targeted, the video quality of the generated style video can be improved, and the efficiency of generating the style video can be improved.

Please further refer to Figure 3. Figure 3 is a schematic flow chart of a video generation method in another embodiment. As shown in Figure 3, the video generation method may include the following steps:

301. Obtain the target detection result corresponding to the target multimedia data. The target detection result is used to indicate the target object included in the target multimedia data.

302. According to the target detection results, determine the target template video, and the target template video corresponds to the target style.

303. Determine the target video generation model based on the target template video. The target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video.

304. Extract features from the target template video to obtain video feature data corresponding to the target template video.

In some embodiments, the electronic device can perform feature extraction on each video frame included in the target template video to obtain video feature data corresponding to each video frame. Feature extraction methods may include, but are not limited to, feature extraction through deep learning models such as convolutional neural networks (CNN) and recurrent neural networks (RNN). In some embodiments, the video feature data corresponding to each video frame may include but is not limited to edge information, depth information, semantic segmentation information, etc. of each video frame, thereby extracting key information in each video frame and reducing redundant information. interference.

The control model may be a neural network model that controls the video generated by the target video generation model. The control model can introduce various custom conditions to control the target video generation model to generate videos that meet the custom conditions. Specifically, the control model may generate video generation control information according to the video feature data.

The video generation control information is control information used to control the target video generation model to generate the target style. The video generation control information may include key information corresponding to the target style. For example, if the target style includes a target action, the video generation control information may include gesture information corresponding to the target action; for example, if the target style includes a target hue, the video generation control information may include hue information corresponding to the target hue.

In some embodiments, the control model may include, but is not limited to, an edge extraction module, a depth estimation module, a gesture recognition module, etc. The control model may use these modules to extract key information corresponding to the target style from the video feature data, and use these modules to extract key information corresponding to the target style. The corresponding key information generates video generation control information, so that the target video generation model can process the target template video according to the target detection results and video generation control information to generate the target video, and the video frames of the target video correspond to the target style, and the target video The video frame includes the target object.

Exemplarily, the control model includes a posture recognition module. The posture recognition module can identify the posture information corresponding to the target action contained in each video frame in the target template video from the video feature data corresponding to each video frame through a posture recognition algorithm. The posture estimation algorithm It can include but is not limited to Openpose algorithm, DeepCut algorithm, etc. The control model generates video generation control information based on the posture information corresponding to each video frame in the target template video, thereby controlling the target video generation model to process the target template video based on the target detection results and video generation control information to generate video frames containing execution Target video of the target object of the target action.

It can be seen that the control model generates video generation control information according to the video feature data, and inputs the video generation control information to the target video generation model, so that the target video generation model can generate a target video that conforms to the target style based on the video generation control information, which is helpful. To improve the controllability and accuracy of generated videos.

305. Input the video feature data into the control model, generate video generation control information according to the video feature data through the control model, and input the video generation control information into the target video generation model.

For the implementation of steps 301 to 305, reference may be made to the above embodiments, and details will not be described again.

306. Obtain text description information corresponding to the target detection result and the target template video.

The text description information corresponding to the target template video may refer to the text description information corresponding to the target style of the target template video, and the text description information may be used to describe the target object and the target style. For example, if the target style is a target action, the corresponding text description information can describe the action process of the target object in the target template video, such as "a dog is wearing headphones, shaking his head and listening to music"; if the target style is Target tone, the corresponding text description information can be the tone style in the target template video, such as "bright and vivid".

In some embodiments, the text description information may be a prompt word (prompt). The prompt is text input information used to guide the artificial intelligence model to complete the task. The prompt can be used to prompt the artificial intelligence model for the input context and input parameter information to help The artificial intelligence model better understands the intent of the input and responds accordingly, helping the artificial intelligence model better understand and complete tasks.

As an optional implementation, the text description information may be a prompt template, and the prompt template may include keywords, for example, "a xxx is wearing headphones, shaking his head and listening to music", where "xxx " is the keyword. After determining the object category of the target object based on the target detection results, the keyword "xxx" in the prompt template can be replaced with the specific object category. For example, if the object category is a dog, then The prompt template is replaced with "a dog wearing headphones, shaking his head and listening to music."

The target detection result includes an object image corresponding to the target object. The object image may include image content corresponding to the target object, and may be content intercepted from the target multimedia data.

307. Analyze the object image and text description information respectively through the target video generation model to obtain the analysis results, and process the target template video according to the analysis results and video generation control information to generate the target video. The video frames of the target video correspond to the target style, and the target object is included in the video frames of the target video.

The electronic device analyzes the object image and text description information respectively through the target video generation model to obtain an analysis result. Optionally, the analysis result may include the object category corresponding to the object image and the model task indicated by the text description information.

For example, the target object in the object image is a dog, the target template video is a video describing "a xxx is wearing headphones, shaking his head and listening to music", the target style is the target action, and the target action is "wearing headphones, Shaking his head and listening to the song", the video generation control information may include the posture information corresponding to the target action, the analysis result may indicate that the object category corresponding to the object image is a dog, and the model task indicated by the text description information is "generate a paragraph of a dog. A video of a dog wearing headphones and shaking his head while listening to music." Therefore, the target video can be generated by processing the target template video based on the above analysis results and video generation control information.

It can be seen that inputting text description information into the target video generation model can help guide the model to generate responses related to user needs, clarify the tasks of the model, ensure that the video generated by the model conforms to the target style, improve the accuracy of the generated video, and can Timely keyword replacement of text description information can adapt to the needs of different application scenarios, and also improves the flexibility of video generation through artificial intelligence models.

Moreover, the control model generates video generation control information according to the video feature data, and the video generation control information is input to the target video generation model, so that the target video generation model can generate a target video that conforms to the target style based on the video generation control information, which is helpful. To improve the controllability and accuracy of generated videos.

Taking the target video generation model as a diffusion model as an example, the control model may be a ControlNet model, in which the diffusion model may be a stable diffusion (Stable Diffusion, SD) model, which is not specifically limited. The electronic device can train the pre-trained diffusion model based on the target template video and the text description information corresponding to the target template video to obtain a diffusion model corresponding to the target style; the video feature data obtained by feature extraction from the target template video can include edges information, depth information, and semantic segmentation information. The video feature data is input into the ControlNet model. The ControlNet model can generate video generation control information based on the video feature data, and input the video generation control information into the diffusion model.

The ControlNet model defines a set of input conditions that can be used to control the output of the diffusion model. These conditions may include color schemes, object categories, or other task-specific parameters. The ControlNet model can pass the input conditions as additional input information to the diffusion model, so that the diffusion model can adjust the output results according to these input conditions and better adapt to specific tasks. Task. Optionally, ControlNet can provide a variety of input conditions including Canny edges, semantic segmentation maps, key points, graffiti, etc., which expands the capabilities of the diffusion model and greatly improves the controllability of the videos generated by the diffusion model.

The ControlNet model creates two copies of the diffusion model, one of which is "locked" and cannot be modified, while the other is "trainable" and can be fine-tuned on specific tasks. The ControlNet model uses "weight sharing" technology. This technique copies the weights of the diffusion model into two different neural networks, so that when fine-tuning the diffusion model, the locked copies still retain common knowledge and can provide better initial states.

In the embodiment of the present application, the corresponding target template video can be matched based on the target detection result corresponding to the target multimedia data, and feature extraction is performed on the target template video to obtain the video feature data input to the control model, so as to use the control model according to the video The feature data generates video generation control information, which improves the controllability and accuracy of the generated video, and then uses the target video generation model to process the target template video based on the target detection results, the text description information corresponding to the target template video, and the video generation control information. , generate a target video. The video frames of the target video correspond to the target style and contain the target object in the target multimedia data. The target video is generated by using the target video generation model adjusted by the target template video, making the generated style video more targeted and able to Improve the video quality of generated style videos and improve the efficiency of generating style videos.

Please further refer to Figure 4. Figure 4 is a schematic flow chart of a video generation method in another embodiment. As shown in Figure 4, the video generation method may include the following steps:

401. Obtain the target detection result corresponding to the target multimedia data. The target detection result is used to indicate the target object included in the target multimedia data.

In some embodiments, the target multimedia data satisfies video generation conditions, and the video generation conditions include one or more of the following: the object category corresponding to the target object in the target multimedia data meets the preset object category; the target in the target multimedia data The number of objects is less than or equal to the preset number; the image size corresponding to the target object in the target multimedia data is greater than or equal to the preset size.

The terminal device determines the selected target multimedia data in response to the selection operation. For example, the user can select any photo or video as the target multimedia data in the photo album application, or can open the camera application, shoot, and obtain the target multimedia data.

The terminal device can determine whether the target multimedia data meets the video generation conditions. Optionally, the terminal device can output prompt information prompting the user to reselect the target multimedia data when none of the above video generation conditions are met, or one of them is not met.

For example, if the preset object category is animal, the video generation conditions may include: the object category of the target object is animal, the detected target object is unique, and the short side of the bounding box generated by target detection is greater than 512 pixels. It should be noted that the object category of the target object is dog, cat, rabbit, etc., which can all conform to the preset object category. If any condition is not met, the user is prompted to reselect the target multimedia data.

By limiting the video generation conditions, it is helpful to obtain targeted multimedia data that is adapted to the video generation model, thereby improving the effectiveness and efficiency of the generated style video, and obtaining high-quality, effective Target multimedia data, thereby improving the video quality of the generated style video.

402. According to the target detection results, determine the target template video, and the target template video corresponds to the target style.

403. Determine the target video generation model based on the target template video. The target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video.

404. Perform feature extraction on the target template video to obtain video feature data corresponding to the target template video.

405. Input the video feature data into the control model, use the control model to generate video generation control information corresponding to the current video frame according to the video feature data corresponding to the current video frame, and input the video generation control information corresponding to the current video frame into the target video. Generate models.

In some embodiments, the electronic device performs feature extraction on each video frame of the target template video to obtain video feature data corresponding to each video frame in the target template video; the video feature data corresponding to each video frame is input to the control model, and The control model generates video generation control information corresponding to the current video frame according to the video feature data corresponding to the current video frame, and inputs the video generation control information corresponding to the current video frame into the target video generation model.

The current video frame can be the video frame currently being processed by the control model among each video frame.

406. Obtain text description information corresponding to the target detection result and the target template video.

407. Perform image segmentation on each video frame of the target template video to obtain segmentation data corresponding to each video frame.

The electronic device can perform frame-by-frame subject detection and segmentation on the target template video to segment the foreground (subject) image area and background image area in each video frame. Optionally, the method of image segmentation for each video frame may include: Not limited to Mask R-CNN algorithm, GrabCut algorithm, fully convolutional network (FCN) and other methods.

It should be noted that the depth of field of the foreground image area and the background image area of each video frame is different. Therefore, the foreground image area and the background image area contained in each video frame can be segmented based on the depth information of the pixels in each video frame. The segmentation data corresponding to each video frame may include foreground data and background data corresponding to each video frame.

It can be seen that frame-by-frame subject detection and segmentation of the target template video helps to accurately determine the area used to generate the image content of the target object in the video frame of the target template video, and helps to improve the efficiency and accuracy of style video generation. sex.

408. Analyze the object image and text description information respectively through the target video generation model to obtain the analysis results, and based on the analysis results, the video generation control information corresponding to the current video frame, and the segmentation data corresponding to the current video frame, in the current video frame Generate image content corresponding to the target object in the foreground image area, and fuse the image content with the background image area of the current video frame to obtain the target video frame corresponding to the current video frame.

The segmentation data corresponding to the current video frame may include foreground data and background data corresponding to the current video frame, the foreground data may be used to indicate the foreground image area of the current video frame, the background data may be used to indicate the background image area of the current video frame, and, the object image Image content corresponding to the target object may be included.

Therefore, the electronic device can generate image content corresponding to the target object in the foreground image area of the current video frame based on the parsing result, the video generation control information corresponding to the current video frame, and the segmentation data corresponding to the current video frame, and compare the image content with the current video frame. The background image area of the video frame is fused. The fusion method may include but is not limited to Alpha blending, pixel-based fusion and other methods.

For example, the target object in the object image is a dog, the target template video is a video describing "a xxx is wearing headphones, shaking his head and listening to music", the target style is the target action, and the target action is "wearing headphones, Shaking his head and listening to the song", the video generation control information may include the posture information corresponding to the target action, the analysis result may indicate that the object category corresponding to the object image is a dog, and the model task indicated by the text description information is "generate a paragraph of a dog. "Video of a dog wearing headphones, shaking his head and listening to music", the electronic device can generate in the foreground image area of the current video frame based on the analysis results, the video generation control information corresponding to the current video frame, and the segmentation data corresponding to the current video frame. The image content of a dog wearing headphones and shaking his head while listening to music is fused with the background image area of the current video frame to obtain the target video frame corresponding to the current video frame.

The target video frame corresponding to the current video frame is a video frame in which the foreground image area of the current video frame is replaced with the image content corresponding to the target object.

It can be seen that the image content corresponding to the target object is generated in the foreground image area of the current video frame, and the image content is merged with the background image area of the current video frame to obtain the target video frame corresponding to the current video frame, which can be based on the target template video Generate videos corresponding to the target style and containing the target object, which improves the efficiency and accuracy of style video generation.

409. Generate a target video based on the target video frame corresponding to each video frame of the target template video.

The electronic device can combine the target video frames corresponding to each video frame of the target template video according to the time sequence of each video frame to obtain the target video.

In the embodiment of the present application, the corresponding target template video can be matched based on the target detection result corresponding to the target multimedia data, and feature extraction is performed on the target template video to obtain the video feature data input to the control model, so as to use the control model according to the video The feature data generates video generation control information, which improves the controllability and accuracy of the generated video. Then, image segmentation is performed on each video frame of the target template video to obtain the image data corresponding to each video frame, and the target video generation model is used according to the target The detection results, the text description information corresponding to the target template video, the segmentation data corresponding to the current video frame, and the video generation control information process the target template video, generate the image content corresponding to the target object in the foreground image area of the current video frame, and The image content is fused with the background image area of the current video frame to obtain the target video frame corresponding to the current video frame, and the target video is generated based on the target video frame corresponding to each video frame of the target template video, and the video frame of the target video corresponds to the target style and Containing the target object in the target multimedia data, the target video is generated by using the target video generation model adjusted by the target template video, making the generated style video more targeted, improving the video quality of the generated style video, and improving the generated style video. s efficiency.

Please further refer to Figure 5. Figure 5 is a timing diagram of a video generation method in an embodiment. As shown in Figure 5, the video generation method may include the following steps:

510. In response to the selection operation, the terminal device determines the selected target multimedia data.

520. The terminal device performs target detection on the target multimedia data through the target detection model, and obtains the target detection result.

The target detection result is used to indicate the target object included in the target multimedia data.

530. The terminal device obtains the target video generated based on the target detection result.

The video frames of the target video correspond to the target style, and the video frames of the target video include the target object.

Among them, the target video is generated by processing the target template video based on the target detection results through the target video generation model; the target template video corresponds to the target style, and the target template video is determined based on the target detection results; the target video generation model is based on the target template video. It is obtained by adjusting the trained video generation model.

540. The server receives the target detection result sent by the terminal device. The target detection result is obtained by the terminal device responding to the selection operation, determining the selected target multimedia data, and performing target detection on the target multimedia data through the target detection model.

550. The server obtains the target detection result corresponding to the target multimedia data, and the target detection result is used to indicate the target object included in the target multimedia data.

560. The server determines the target template video based on the target detection results.

The target template video corresponds to the target style.

570. The server determines the target video generation model based on the target template video.

The target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video.

580. The server processes the target template video according to the target detection result through the target video generation model to generate the target video.

The video frames of the target video correspond to the target style, and the video frames of the target video include the target object.

590, the server sends the target video to the terminal device.

For the implementation of steps 510 to 590, reference may be made to the above embodiments, and details will not be described again.

In the embodiment of the present application, the corresponding target template video can be matched based on the target detection result corresponding to the target multimedia data, so that the target template video is processed according to the target detection result through the target video generation model corresponding to the target template video, and the target video is generated. By using the target video generation model adjusted by the target template video to generate the target video, the generated style video is more targeted, the video quality of the generated style video can be improved, and the efficiency of generating the style video can be improved.

Please further refer to Figure 6, which is a sequence diagram of a video generation method in another embodiment. As shown in Figure 6, by selecting a pet image, a pet video of a specific action and style can be generated. The terminal device (device side) selects pet photos through the album, performs target detection on the pictures and determines whether they contain a single target pet, uploads the target detection results that meet the requirements to the server (cloud side) for video template matching, and performs video template matching on the preset prompts The word prompt and the target foreground image (object image) are used for image and text analysis; the target template video is subject to frame-by-frame detection and segmentation, the relevant features of the video frame are extracted, the video is generated through the ControlNet controlled diffusion model, and finally delivered to the end side .

Please further refer to Figure 7. Figure 7 is a sequence diagram of a video generation method in another embodiment. As shown in Figure 7, the specific steps may include: (1) Image selection and target detection: The user opens the image album to select an image, based on The target detection model performs target detection on the image and obtains the pet information of the specified category and the number of targets; (2) Image filtering: The user needs to reselect images with the following conditions: multiple pet categories are detected, multiple targets are detected, The short side size of the detected target frame is less than 512; (3) Model fine-tuning: Based on the built-in video template, extract feature information to control the model ControlNet, perform image segmentation on the video frame, and fine-tune to generate a model for a specific action template; (4) Prompt template replacement and analysis: replace the keywords in the Prompt based on the detection results, and input the corresponding model through text analysis; (5) Reference image input corresponding action model generation result: output the reference image to the corresponding action model, Combined with prompt analysis, a video animation of the specified action of the reference target is generated.

Please refer to Figure 8. Figure 8 is a schematic structural diagram of a video generation device in one embodiment. As shown in Figure 8, the video generation device 800 may include: an acquisition module 810, a template determination module 820, a model determination module 830 and a video generation module 840. .

The acquisition module 810 is used to obtain the target detection result corresponding to the target multimedia data, and the target detection result is used to indicate the target object included in the target multimedia data;

The template determination module 820 is used to determine the target template video according to the target detection result; the target template video corresponds to the target style;

The model determination module 830 is used to determine the target video generation model based on the target template video; the target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video;

The video generation module 840 is used to process the target template video according to the target detection results through the target video generation model to generate a target video, the video frames of the target video correspond to the target style, and the video frames of the target video include the target object.

In one embodiment, the video generation device 800 further includes: a control module;

The control module is used to extract features from the target template video to obtain video feature data corresponding to the target template video; input the video feature data into the control model, generate video generation control information based on the video feature data through the control model, and control the video generation Information is input to the target video generation model;

The video generation module 840 is also used to process the target template video according to the target detection results and video generation control information through the target video generation model to generate the target video.

In one embodiment, the target detection result includes an object image corresponding to the target object; the video generation module 840 is also used to: obtain text description information corresponding to the target detection result and the target template video; and generate the object image through the target video generation model. and text description information are parsed to obtain the parsing results, and the target template video is processed according to the parsing results and video generation control information to generate the target video.

In one embodiment, the video generation device 800 further includes: a segmentation module;

The segmentation module is used to perform image segmentation on each video frame of the target template video to obtain the segmentation data corresponding to each video frame;

The control module is also used to generate video generation control information corresponding to the current video frame according to the video feature data corresponding to the current video frame through the control model, and input the video generation control information corresponding to the current video frame to the target video generation model;

The video generation module 840 is also used to generate image content corresponding to the target object in the foreground image area of the current video frame based on the parsing results, the video generation control information corresponding to the current video frame, and the segmentation data corresponding to the current video frame, and convert the image into The content is fused with the background image area of the current video frame to obtain the target video frame corresponding to the current video frame; the target video is generated based on the target video frame corresponding to each video frame of the target template video.

In one embodiment, the target detection result includes the object category corresponding to the target object; the template determination module 820 is also configured to determine the target template video corresponding to the object category according to the object category corresponding to the target object.

In one embodiment, the target multimedia data satisfies video generation conditions, and the video generation conditions include one or more of the following: the object category corresponding to the target object in the target multimedia data conforms to the preset object category; the target in the target multimedia data The number of objects is less than or equal to the preset number; the image size corresponding to the target object in the target multimedia data is greater than or equal to the preset size.

In one embodiment, the target style includes a target action, and the video frames of the target video include a target object performing the target action.

In one embodiment, the acquisition module 810 is also used to receive the target detection result sent by the terminal device. The target detection result is that the terminal device determines the selected target multimedia data in response to the selection operation, and performs the target multimedia data on it through the target detection model. target detection.

In the embodiment of the present application, the corresponding target template video can be matched based on the target detection result corresponding to the target multimedia data, so that the target template video is processed according to the target detection result through the target video generation model corresponding to the target template video, and the target video is generated. By using the target video generation model adjusted by the target template video to generate the target video, the generated style video is more targeted, the video quality of the generated style video can be improved, and the efficiency of generating the style video can be improved.

Please refer to Figure 9. Figure 9 is a schematic structural diagram of a video generation device in another embodiment. As shown in Figure 9, the video generation device 900 can be applied to the above-mentioned terminal equipment. The video generation device 900 can include: a response module 910, a detection module module 920 and acquisition module 930;

Response module 910, configured to determine the selected target multimedia data in response to the selection operation;

The detection module 920 is used to perform target detection on the target multimedia data through the target detection model to obtain a target detection result; the target detection result is used to indicate the target object included in the target multimedia data;

The acquisition module 930 is used to acquire a target video generated based on the target detection result, the video frame of the target video corresponds to the target style, and the video frame of the target video includes the target object; wherein, the target video is generated based on the target detection result through the target video generation model. The target template video is processed and generated; the target template video corresponds to the target style, and the target template video is determined based on the target detection results; the target video generation model is obtained by adjusting the pre-trained video generation model based on the target template video.

Please refer to Figure 10, which is a schematic structural diagram of an electronic device in an embodiment. As shown in Figure 10, the electronic device 1000 may include: a memory 1010 storing executable program code; a processor 1020 coupled to the memory 1010; wherein the processor 1020 calls the executable program code stored in the memory 1010 to execute the program. Any video generation method disclosed in the application embodiments.

An embodiment of the present application discloses a computer-readable storage medium that stores a computer program. When the computer program is executed by the processor, the processor implements any video generation method disclosed in the embodiment of the present application.

It will be understood that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. Those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily necessary for this application.

In the various embodiments of the present application, it should be understood that the size of the sequence numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be implemented in this application. The implementation of the examples does not constitute any limitations. The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment. In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application is essentially or contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory. , including several requests to cause a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to execute some or all of the steps of the above methods in various embodiments of the present application.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only storage medium. Memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), programmable read-only memory (Programmable Read-only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), One-time Programmable Read-Only Memory (OTPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), CompactDisc Read -Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

The above is a detailed introduction to a video generation method, device, electronic equipment and storage medium disclosed in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only for To help understand the method and core ideas of this application. At the same time, for those of ordinary skill in the art, there will be changes in the specific implementation and application scope based on the ideas of the present application. In summary, the content of this description should not be understood as a limitation of the present application.

Claims (13)

1. A video generation method, characterized in that the method includes: Obtaining a target detection result corresponding to the target multimedia data, where the target detection result is used to indicate a target object included in the target multimedia data; According to the target detection result, a target template video is determined; the target template video corresponds to the target style; Determine a target video generation model according to the target template video; the target video generation model is obtained by adjusting a pre-trained video generation model according to the target template video; The target template video is processed according to the target detection result by the target video generation model to generate a target video, the video frames of the target video correspond to the target style, and the video frames of the target video include The target object. 2. The method according to claim 1, characterized in that, before processing the target template video according to the target detection result through the target video generation model to generate a target video, the method further include: Perform feature extraction on the target template video to obtain video feature data corresponding to the target template video; inputting the video feature data to a control model, generating video generation control information according to the video feature data through the control model, and inputting the video generation control information to the target video generation model; The target video generation model processes the target template video according to the target detection result to generate a target video, including: The target video generation model processes the target template video according to the target detection result and the video generation control information to generate a target video. 3. The method of claim 2, wherein the target detection result includes an object image corresponding to the target object; and the target video generation model is generated based on the target detection result and the video. Control information to process the target template video to generate a target video, including: Obtain text description information corresponding to the target detection result and the target template video; The object image and the text description information are respectively analyzed through the target video generation model to obtain an analysis result, and the target template video is processed according to the analysis result and the video generation control information to generate target video. 4. The method according to claim 3, characterized in that, before analyzing the object image and the text description information respectively through the target video generation model, the method further includes: Perform image segmentation on each video frame of the target template video to obtain segmentation data corresponding to each video frame; Generating video generation control information according to the video feature data through the control model, and inputting the video generation control information to the target video generation model includes: Generate video generation control information corresponding to the current video frame according to the video feature data corresponding to the current video frame through the control model, and input the video generation control information corresponding to the current video frame into the target video generation model; Processing the target template video according to the parsing result and the video generation control information to generate a target video includes: According to the analysis result, the video generation control information corresponding to the current video frame, and the segmentation data corresponding to the current video frame, image content corresponding to the target object is generated in the foreground image area of the current video frame, and Fusion of the image content with the background image area of the current video frame to obtain a target video frame corresponding to the current video frame; A target video is generated based on target video frames corresponding to each video frame of the target template video. 5. The method according to claim 1, wherein the target detection result includes an object category corresponding to the target object; and determining the target template video according to the target detection result includes: According to the object category corresponding to the target object, the target template video corresponding to the object category is determined. 6. The method according to claim 1, characterized in that the target multimedia data satisfies video generation conditions, and the video generation conditions include one or more of the following: The object category corresponding to the target object in the target multimedia data conforms to the preset object category; The number of target objects in the target multimedia data is less than or equal to a preset number; The image size corresponding to the target object in the target multimedia data is greater than or equal to the preset size. 7. The method according to any one of claims 1 to 6, wherein the target style includes a target action, and the video frame of the target video includes the target object performing the target action. 8. The method according to claim 1, characterized in that the method is applied to a server; the obtaining the target detection result corresponding to the target multimedia data includes: Receive a target detection result sent by the terminal device. The target detection result is obtained by the terminal device determining the selected target multimedia data in response to the selection operation, and performing target detection on the target multimedia data through a target detection model. 9. A video generation method, characterized in that it is applied to a terminal device, and the method includes: In response to the selection operation, determining the selected target multimedia data; Perform target detection on the target multimedia data through a target detection model to obtain a target detection result; the target detection result is used to indicate the target object included in the target multimedia data; Obtain a target video generated based on the target detection result, the video frames of the target video correspond to the target style, and the video frames of the target video include the target object; Wherein, the target video is generated by processing a target template video according to the target detection result through a target video generation model; the target template video corresponds to the target style, and the target template video is determined according to the target detection result. ; The target video generation model is obtained by adjusting a pre-trained video generation model based on the target template video. 10. A video generation device, characterized in that the device includes: An acquisition module, configured to acquire a target detection result corresponding to the target multimedia data, where the target detection result is used to indicate a target object included in the target multimedia data; A template determination module, configured to determine a target template video according to the target detection result; the target template video corresponds to the target style; A model determination module, configured to determine a target video generation model based on the target template video; the target video generation model is obtained by adjusting a pre-trained video generation model based on the target template video; A video generation module, configured to process the target template video according to the target detection result through the target video generation model to generate a target video, the video frames of the target video correspond to the target style, and the target The target object is included in the video frame of the video. 11. A video generation device, characterized in that it is applied to terminal equipment, and the device includes: a response module, configured to determine the selected target multimedia data in response to the selection operation; A detection module, configured to perform target detection on the target multimedia data through a target detection model to obtain a target detection result; the target detection result is used to indicate a target object included in the target multimedia data; An acquisition module, configured to acquire a target video generated based on the target detection result, the video frame of the target video corresponds to the target style, and the video frame of the target video includes the target object; wherein, the target video is The target template video is processed and generated by the target video generation model according to the target detection result; the target template video corresponds to the target style, and the target template video is determined according to the target detection result; the target video generation model It is obtained by adjusting the pre-trained video generation model according to the target template video. 12. An electronic device, characterized by comprising a memory and a processor, a computer program stored in the memory, and when the computer program is executed by the processor, the processor implements claims 1 to 8 Or any of the methods described in 9. 13. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method according to any one of claims 1 to 8 or 9 is implemented.