CN114511811A - Video processing method, video processing device, electronic equipment and medium - Google Patents

Abstract

The present disclosure provides a neural network model training method, video processing method, device, electronic device and medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning and computer vision. The implementation scheme is: obtaining a plurality of first sample images and color second sample images corresponding to the plurality of first sample images; inputting the plurality of first sample images into a feature extraction sub-network to obtain the first features; The second sample image is input into the feature extraction sub-network to obtain the second feature; the first feature and the second feature are input into the pre-coloring sub-network to obtain the pre-coloring result; the first feature is input into the object segmentation sub-network to obtain multiple object segmentation features; inputting the pre-coloring result and the plurality of object segmentation features into a final shading sub-network to obtain a final shading result; and adjusting parameters of the neural network model based on at least the final shading result and the second sample image.

Description

Video processing method, apparatus, electronic device and medium

technical field

The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning and computer vision, and in particular to a training method for a neural network model, a video processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

Background technique

Artificial intelligence is the study of making computers to simulate certain thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both hardware-level technology and software-level technology. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, and machine learning/depth Learning, big data processing technology, knowledge graph technology and other major directions.

Image and video processing tasks are a practical application of applying deep learning algorithms. Using deep learning algorithms can greatly improve the efficiency of image and video processing.

The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered to be recognized in any prior art.

SUMMARY OF THE INVENTION

The present disclosure provides a training method for a neural network model, a video processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

According to an aspect of the present disclosure, a method for training a neural network model is provided. Among them, the neural network model includes a feature extraction sub-network, a pre-coloring sub-network, an object segmentation sub-network and a final coloring sub-network. The method includes: acquiring a plurality of first sample images and color second sample images corresponding to the plurality of first sample images, wherein each first sample image in the plurality of first sample images includes a Processing object; inputting multiple first sample images into the feature extraction sub-network to obtain the first features; inputting the second sample images into the feature extraction sub-network to obtain the second features; inputting the first features and the second features into the preset a coloring sub-network to obtain a predicted pre-coloring result for at least one first sample image in a plurality of first sample images; the first feature is input into the object segmentation sub-network to obtain a plurality of object segmentation features, wherein, Each object segmentation feature in the multiple object segmentation features is respectively for a corresponding one of the multiple first sample images; the pre-coloring result and the multiple object segmentation features are input into the final coloring sub-network to obtain the final coloring the result; and adjusting parameters of the neural network model based on at least the final coloring result and the second sample image.

According to another aspect of the present disclosure, a video processing method is provided. The method includes: inputting a target coloring frame of a video to be colorized, at least one frame adjacent to the target coloring frame, and a color reference image into a neural network model to obtain a final coloring result for the target coloring frame, wherein the neural network model is obtained by training according to the above method.

According to another aspect of the present disclosure, a training apparatus for a neural network model is provided. Among them, the neural network model includes a feature extraction sub-network, a pre-coloring sub-network, an object segmentation sub-network and a final coloring sub-network. The apparatus includes: a first unit configured to acquire a plurality of first sample images and color second sample images corresponding to the plurality of first sample images, wherein each of the plurality of first sample images Each of the sample images includes the object to be processed; the second unit is configured to input the plurality of first sample images into the feature extraction sub-network to obtain the first features; the second unit is further configured to input the second sample images into the feature extraction sub-network a feature extraction sub-network to obtain a second feature; a third unit configured to input the first feature and the second feature into the pre-coloring sub-network to obtain a first sample for at least one of the plurality of first sample images The pre-coloring result predicted by the image; the fourth unit is configured to input the first feature into the object segmentation sub-network to obtain a plurality of object segmentation features, wherein each object segmentation feature in the plurality of object segmentation features is respectively for multiple object segmentation features. A corresponding one of the first sample images; a fifth unit configured to input the pre-coloring result and the plurality of object segmentation features into a final coloring sub-network to obtain a final coloring result; and a sixth unit, is configured to adjust parameters of the neural network model based at least on the final shading result and the second sample image.

According to another aspect of the present disclosure, a video processing apparatus is provided. The device includes: a coloring unit configured to input a target coloring frame of the video to be colorized, at least one frame adjacent to the target coloring frame, and a color reference image into a neural network model to obtain a final coloring result for the target coloring frame, The neural network model is obtained by training according to the above method.

According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by at least one processor A processor executes to enable at least one processor to execute any one of the methods described above.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform any one of the above methods.

According to another aspect of the present disclosure, there is provided a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, implements any one of the above methods.

According to one or more embodiments of the present disclosure, the "jitter" or "flicker" effect of the rendered video in the temporal domain can be reduced or eliminated, thereby improving the quality of video rendering.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings illustrate the embodiments by way of example and constitute a part of the specification, and together with the written description of the specification serve to explain exemplary implementations of the embodiments. The shown embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numbers refer to similar but not necessarily identical elements.

1 shows a schematic diagram of an exemplary system in which various methods described herein may be implemented, according to embodiments of the present disclosure;

FIG. 2 shows a flowchart of a training method of a neural network model according to an embodiment of the present disclosure;

FIG. 3 shows a flowchart of a portion of an example process in the method of FIG. 2 according to an embodiment of the present disclosure;

4 is a schematic diagram illustrating a process of training a neural network model by using a training method for a neural network model according to an embodiment of the present disclosure;

5 is a schematic diagram illustrating a process of training a neural network model by using a training method for a neural network model according to an embodiment of the present disclosure;

6 shows a structural block diagram of a training apparatus for a neural network model according to an embodiment of the present disclosure;

FIG. 7 shows a structural block diagram of a video processing apparatus according to an embodiment of the present disclosure; and

8 shows a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship or importance relationship of these elements, and such terms are only used for Distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases they may refer to different instances based on the context of the description.

The terminology used in the description of the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly dictates otherwise, if the number of an element is not expressly limited, the element may be one or more. Furthermore, as used in this disclosure, the term "and/or" covers any and all possible combinations of the listed items.

In the related art, when coloring a grayscale video, professionals usually fill each frame of the grayscale video with color through special video editing software, which consumes a lot of manpower and material resources and is inefficient. In another related art, the video colorization technology based on deep learning can be used to colorize the grayscale video using the neural network model obtained after learning on many colorization datasets. However, the colorization results obtained in this way are relatively fixed. Shading is not possible as the user expects. In another related art, the grayscale video can be colored based on the reference image provided by the user. Although the coloring can be performed according to the user's expectation to a certain extent, the same object in the colored video may be colored at different times. Shading results can vary, so when a shaded video is played, the video will have a noticeable "jitter" or "flickering" effect in the temporal domain, reducing the quality of the video's shading.

In order to solve the above problems, the present disclosure proposes a training method for a neural network model, by using a plurality of first sample images to be colored and corresponding colored second sample images as training samples, , corresponding object segmentation features are obtained, and the object segmentation features can be used to guide the neural network model to pay more attention to the objects to be colored in the multiple first sample images. The neural network model trained based on this, on the basis of using the reference image to colorize the image to be colored, because the model pays more attention to the object to be colored in the image, the same object in the video formed by the colored image can be The shading results at different times are as similar as possible, so as to reduce or eliminate the "jitter" or "flicker" effect of the shading video in the time domain, and improve the quality of video shading.

In the present disclosure, the "sub-network" of the neural network is not necessarily a network structure based on layers composed of neurons, and may also process the data, features, etc. input to the sub-network through other network structures and processing methods. Not limited.

In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

1 shows a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein may be implemented in accordance with embodiments of the present disclosure. Referring to FIG. 1 , the system 100 includes one or more client devices 101 , 102 , 103 , 104 , 105 and 106 , a server 120 , and one or more communication networks coupling the one or more client devices to the server 120 110. Client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable the model training method or the video processing method to be performed.

In some embodiments, server 120 may also provide other services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, eg, provided to users of client devices 101, 102, 103, 104, 105, and/or 106 under a software-as-a-service (SaaS) model .

In the configuration shown in FIG. 1 , server 120 may include one or more components that implement the functions performed by server 120 . These components may include software components executable by one or more processors, hardware components, or a combination thereof. Users operating client devices 101, 102, 103, 104, 105, and/or 106 may in turn utilize one or more client applications to interact with server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from system 100 . Accordingly, FIG. 1 is one example of a system for implementing the various methods described herein, and is not intended to be limiting.

A user may use client devices 101, 102, 103, 104, 105, and/or 106 to provide rendered video as well as reference images. A client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although FIG. 1 depicts only six types of client devices, those skilled in the art will appreciate that the present disclosure may support any number of client devices.

Client devices 101, 102, 103, 104, 105, and/or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, Smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems , such as MICROSOFT WindowsMobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices may include head mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. Client devices are capable of executing a variety of different applications, such as various Internet-related applications, communication applications (eg, e-mail applications), Short Message Service (SMS) applications, and may use various communication protocols.

Network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP/IP, SNA, IPX, and the like. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, Public Switched Telephone Network (PSTN), infrared networks, wireless networks (eg, Bluetooth, WIFI), and/or any combination of these and/or other networks.

Server 120 may include one or more general purpose computers, special purpose server computers (eg, PC (personal computer) servers, UNIX servers, midrange servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination . Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures that involve virtualization (eg, may be virtualized to maintain one or more flexible pools of logical storage devices of the server's virtual storage devices). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

The computing units in server 120 may run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 may also run any of a variety of additional server applications and/or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

In some embodiments, the server 120 may include one or more application programs, for example, application programs for services such as object detection and recognition, signal conversion, etc. based on data such as images, video, speech, text, digital signals, etc. Task requests such as voice interaction, text classification, image recognition or key point detection received by the end devices 101, 102, 103, 104, 105 and/or 106. The server can use the training samples to train the neural network model according to the specific deep learning task, and can test each sub-network in the super-network module of the neural network model, and determine the deep learning task according to the test results of each sub-network. The structure and parameters of the neural network model. Various data can be used as training sample data for deep learning tasks, such as image data, audio data, video data, or text data. After the training of the neural network model is completed, the server 120 can also automatically search for an optimal model structure through a model search technology to perform corresponding tasks.

In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology. Cloud server is a host product in the cloud computing service system to solve the defects of difficult management and weak business expansion in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

System 100 may also include one or more databases 130 . In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Database 130 may reside in various locations. For example, the database used by server 120 may be local to server 120, or may be remote from server 120 and may communicate with server 120 via a network-based or dedicated connection. Database 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

In some embodiments, one or more of the databases 130 may also be used by applications to store application data. Databases used by applications can be different types of databases such as key-value stores, object stores, or regular stores backed by a file system.

The system 100 of FIG. 1 may be configured and operated in various ways to enable application of the various methods and apparatuses described in accordance with the present disclosure.

2 shows a flowchart of a training method 200 of a neural network model according to an embodiment of the present disclosure, wherein the neural network model includes a feature extraction sub-network, a pre-coloring sub-network, an object segmentation sub-network, and a final coloring sub-network . Method 200 includes:

Step S210: Acquiring a plurality of first sample images and a color second sample image corresponding to the plurality of first sample images, wherein each first sample image in the plurality of first sample images includes to-be-processed object;

Step S220, inputting a plurality of first sample images into the feature extraction sub-network to obtain the first features;

Step S230, inputting the second sample image into the feature extraction sub-network to obtain the second feature;

Step S240, inputting the first feature and the second feature into the pre-coloring sub-network to obtain a predicted pre-coloring result for at least one first sample image in the plurality of first sample images;

Step S250, inputting the first feature into the object segmentation sub-network to obtain multiple object segmentation features, wherein each object segmentation feature in the multiple object segmentation features is respectively for a corresponding one of the multiple first sample images. sample image;

Step S260, input the pre-coloring result and multiple object segmentation features into the final coloring sub-network to obtain the final coloring result; and

Step S270, at least based on the final coloring result and the second sample image, adjust the parameters of the neural network model.

In the method 200, a plurality of first sample images to be colored and the corresponding colored second sample images are used as training samples, and for each first sample image, corresponding object segmentation features are obtained. The guided neural network model pays more attention to the objects to be colored in the plurality of first sample images. The neural network model trained based on this, on the basis of using the reference image to colorize the image to be colored, because the model pays more attention to the object to be colored in the image, the same object in the video formed by the colored image can be The shading results at different times are as similar as possible, so as to reduce or eliminate the "jitter" or "flicker" effect of the shading video in the time domain, and improve the quality of video shading.

It should be understood that the "objects" in the first sample image and the second sample image may be images of instances or objects included in the images.

According to some embodiments, each of the plurality of first sample images may include the same kind and the same number of objects, and the second sample image may include the same number of objects as each of the first sample images. The colored reference object corresponding to the object in the image.

Among the adopted training samples, the plurality of first sample images and the second sample images include objects of the same type and quantity, and the objects included in the second sample images are colored objects. In other words, the objects in the second sample image differ from the objects in the plurality of first sample images only by the colors of the included objects. In this way, the quality of pre-coloring of the first sample image by the neural network model obtained by training can be improved, thereby improving the final coloring quality of the first sample image.

According to some embodiments, the second sample image may be a color image in the Lab color space.

Different from RGB or CMYK color space, Lab color space consists of three elements, one element is luminance L; a and b are two color channels. a includes colors from dark green (low lightness value) to gray (medium lightness value) to bright pink (high lightness value); b is from light blue (low lightness value) to gray (medium lightness value) and then to yellow (high brightness value). It can be seen that the Lab color space is used to represent the second sample image, because only the features of two channels (that is, the above-mentioned two color channel features of a and b) are needed, and the features of the luminance channel L are not needed (because it is used as a reference for reference coloring). image, only the color information of the second sample image is needed, not its brightness information), which reduces the number of feature channels in the model operation process, thereby improving the speed of model training. On the other hand, compared to the RGB color space, the Lab color space has a more linear representation of color, which reduces the difficulty of model training, thereby further improving the efficiency of model training, and also makes it possible to use the trained model to colorize images. Shading quality is higher.

According to some embodiments, the plurality of first sample images may be consecutive frames in the video to be rendered. Since the positions of included objects may change between adjacent frames in consecutive frames, by using consecutive frames in the video to be colored as the first sample images, the training samples are more in line with the model The actual input value when used, the model obtained based on this training can further improve the coloring effect of multiple consecutive frames of the video to be colored, thereby reducing or eliminating the "jitter" or "flickering" effect of the colored video in the time domain, improving the video quality. Shading quality.

According to some embodiments, the consecutive plurality of frames may be consecutive 3 frames, and obtaining a predicted pre-coloring result for at least one first sample image among the plurality of first sample images may include: obtaining Predicted preshading results for the intermediate frame of the 3 frames.

Therefore, three consecutive frames are used as multiple first sample images, and the pre-coloring result is only predicted for the intermediate frame, so that the final coloring result predicted by the model for the intermediate frame can pay more attention to the adjacent frames before and after the intermediate frame. , so that in the final shading result, the shading result of the object in the intermediate frame is more uniform than the shading result of the object in the adjacent frame, that is, the shading of the same object in the shaded video in the adjacent 3 frames The results are similar, thereby reducing or eliminating the "jitter" or "flickering" effect of the video in the temporal domain and improving the quality of video rendering.

In some examples, in steps S220 and S230, existing image feature extraction backbone networks such as VGG, ResNet50, ResNet101, etc. or their combination may be used to perform feature extraction on multiple first sample images and second sample images .

In some examples, a custom feature extraction sub-network can be used. For example, the feature extraction sub-network may first perform feature extraction on multiple first sample images to obtain multiple extracted features, and further, may perform a merge operation on the multiple extracted features to obtain a first feature.

In some examples, in step S240, a nonlocal network may be used as a pre-coloring sub-network to pre-color at least one first sample image among the plurality of first sample images. It should be understood that one, two or more of the plurality of first sample images may be pre-colored; accordingly, based on the pre-coloring results, one, two or more may be acquired final coloring result.

According to some embodiments, in step S260, inputting the pre-coloring result and the plurality of object segmentation features into the final shading sub-network may include: merging the pre-coloring result and the plurality of object segmentation features in the feature channel dimension; and merging the obtained result Enter the final shading sub-network to get the final shading result.

Therefore, by first merging the pre-coloring results and multiple object segmentation features in the feature channel dimension, a feature block is obtained, and then the merged feature block is input into the final shading sub-network, which simplifies the input of the final shading sub-network and improves the model. operational efficiency.

It should be understood that merging in the feature channel dimension refers to accumulating the number of feature channels without changing the height and width of the merged features.

According to some embodiments, the object segmentation feature may be an object segmentation mask. The object segmentation mask (mask) can consist of 0 and 1, that is, 1 is used in the mask to represent the part of the image in which the object is segmented; 0 is used in the mask to represent the part of the image that does not segment the object (eg background), thus, combined with the object segmentation mask, the final shading sub-network can accurately finalize the part of the image where the object is segmented by 1, without mistakenly coloring the part around the object (for example, represented by 0). the background part) are also colored in the same way. As a result, the final shaded result is more accurate between adjacent frames, reducing or eliminating the "shaky" effect of shaded video.

FIG. 3 shows a flowchart of part of the example process (ie, step S270 ) in the method 200 of FIG. 2 according to an embodiment of the present disclosure. As shown in FIG. 3, according to some embodiments, at least based on the final coloring result and the second sample image, the step of adjusting the parameters of the neural network model may include:

S371, based on the final coloring result and the second sample image, calculate the first loss value;

S372. Calculate a second loss value based on at least the object segmentation feature; and

S373. Based on the first loss value and the second loss value, adjust the parameters of the neural network model.

Therefore, not only the parameters of the neural network model are adjusted according to the first loss value obtained based on the final coloring result and the second sample image, but also the parameters of the neural network model are adjusted according to the second loss value obtained based on at least the object segmentation feature, so that The model parameters can be more optimized for the object segmentation features, so that the trained model pays more attention to the object segmentation features and further improves the final coloring quality.

It can be understood that the second loss value can be calculated based on the object segmentation features and the corresponding ground truth.

According to an embodiment of the present disclosure, a video processing method is also provided, the video processing method comprising: inputting a target coloring frame of a video to be colorized, at least one frame adjacent to the target coloring frame, and a color reference image into a neural network model , to obtain the final coloring result for the target coloring frame, wherein the neural network model is obtained by training according to the above-mentioned neural network model training method.

Thus, using the model trained according to the above-mentioned neural network model training method, the video frame to be colored is colored, so that the coloring result of the object in the target colored frame is more uniform than the coloring result of the object in the adjacent frame, that is, The same object in the shaded video has similar shaded results in the target shaded frame and adjacent frames, thereby reducing or eliminating the "jitter" or "flickering" effect of the video in the temporal domain and improving the quality of video coloring.

The method for training a neural network model according to an embodiment of the present disclosure will be further described below with reference to FIG. 4 and FIG. 5 .

4 and 5 are schematic diagrams illustrating a process of training a neural network model by using a training method for a neural network model according to an embodiment of the present disclosure.

As shown in FIG. 4 , the neural network model 400 includes: a feature extraction sub-network 410 , a pre-coloring sub-network 420 , an object segmentation sub-network 430 and a final coloring sub-network 440 . In addition, Figure 5 shows a pre-shading sub-network 520, an object segmentation sub-network 530, and a final shading sub-network 540, which are the same as the pre-shading sub-network 420, the object segmentation sub-network 430, and the final shading sub-network 440 shown in Figure 4, respectively resemblance.

A plurality of first sample images 401, 402, 403 (for example, three consecutive frames of the video to be rendered, ie, t-1 frame, t frame, and t+1 frame) and the corresponding color second sample images 404 ( e.g. reference image). Wherein, each of the first sample images 401 , 402 , and 403 in the plurality of first sample images includes the same type and the same number of objects (that is, the t-1 frame, the t frame, and the t+1 frame all include dolphin, toy, arm), and the second sample image 404 (reference image) includes color reference objects (eg, colored dolphins, toys, arms) that correspond one-to-one with the objects in each of the first sample images. Here, the second sample image is a color image in the Lab color space.

The acquired multiple first sample images and the color second sample images are input into the feature extraction sub-network 410 of the neural network model 400 to perform feature extraction on the multiple first sample images and the second sample images respectively. For example, for each of the three first sample images, a feature map with dimensions (H11, W11, Cx) can be extracted respectively, and the three dimensions are (H1, W1, Cx) ) feature maps are merged to obtain the first feature of dimension (H11, W11, C), where H and W represent the height and width of the feature map respectively; C represents the number of feature channels. For the second sample image, a second feature with a dimension of (H22, W22, C) can be extracted.

After the first feature and the second feature are obtained, the first feature and the second feature can be input into the pre-coloring sub-network 420 . Here, continuing to refer to FIG. 5 , the first feature and the second feature are input into the pre-coloring sub-network 520. In order to facilitate the calculation, the two features can be flattened first to obtain the first feature with dimension ((H11*W11), C). A flattened feature 501-1 and a second flattened feature 501-2 of dimension ((H22*W22), C). Then, the second flattened feature 501-2 may be transposed to obtain a feature matrix with dimension (C, (H22*W22)). Then, for example, using matrix multiplication, the above-mentioned two-way features are multiplied to obtain a similarity matrix 502 (((H11*W11), C)*(C, (H22*W22))), and the similarity matrix 502 is used to represent A similarity map between each pixel in the frame to be rendered (eg, the t-th frame) and each pixel in the reference image.

Then, the pre-coloring sub-network can sample the color information of each pixel in the reference image according to the obtained similarity matrix 502 (for example, sample the color information of the a and b channels of the reference image), and then sample the obtained color information Color information is mapped into the frame to be rendered, resulting in a pre-colored result. Specifically, the similarity matrix 502 can be multiplied by the Lab map of the reference image, that is, ((H11*W11), (H22*W22)*(H22*H22, C) to obtain ((H11*W11), C ), and then convert it to (H11, W11, C) to get the pre-colored result.

Further, referring to FIG. 5 , further optimization of the pre-coloring results is required. Therefore, the first feature can be input into the object segmentation sub-network 530 to obtain a plurality of object segmentation features (eg, the above-mentioned object segmentation mask), wherein each object segmentation feature of the plurality of object segmentation features is for A corresponding one of the plurality of first sample images. Specifically, for t-1 frame, t frame and t+1 frame, the first feature can be read as a three-channel object segmentation mask.

Further, the pre-coloring result and the plurality of object segmentation features are input into the final shading sub-network 540, so as to obtain the final shading result.

Continuing to refer to FIG. 4, the parameters of the neural network model can be adjusted based at least on the final colorization result and the second sample image. Specifically, the first loss value may be calculated based on the final colorization result and the second sample image 404 (ie, the reference image); and the second loss value may be calculated based on the object segmentation feature and the object segmentation ground truth 405 corresponding thereto . Here, the first loss value may be the L1 loss of the final shading result and the reference image in the ab color space. Therefore, not only the parameters of the neural network model are adjusted according to the first loss value obtained based on the final coloring result and the second sample image, but also the parameters of the neural network model are adjusted according to the second loss value obtained based on at least the object segmentation feature, so that The model parameters can be more optimized for the object segmentation features, so that the trained model pays more attention to the object segmentation features, which can make the coloring results of the same object in the video formed by the colored images at different times as close as possible, thereby reducing the Or remove the "jitter" or "flicker" effect in the time domain of the shaded video to improve the quality of video shaders.

FIG. 6 shows a structural block diagram of an apparatus 600 for training a neural network model according to an embodiment of the present disclosure. Wherein, the neural network model includes a feature extraction sub-network, a pre-coloring sub-network, an object segmentation sub-network and a final coloring sub-network. As shown in FIG. 6, apparatus 600 includes:

The first unit 610 is configured to acquire a plurality of first sample images and a color second sample image corresponding to the plurality of first sample images, wherein each first sample in the plurality of first sample images Images include objects to be processed;

The second unit 620 is configured to input the plurality of first sample images into the feature extraction sub-network to obtain the first features;

The second unit 620 is further configured to input the second sample image into the feature extraction sub-network to obtain the second feature;

a third unit 630, configured to input the first feature and the second feature into the pre-coloring sub-network to obtain a predicted pre-coloring result for at least one first sample image in the plurality of first sample images;

The fourth unit 640 is configured to input the first feature into the object segmentation sub-network to obtain a plurality of object segmentation features, wherein each object segmentation feature in the plurality of object segmentation features is respectively for the plurality of first sample images. A corresponding first sample image of ;

a fifth unit 650, configured to input the pre-coloring result and the plurality of object segmentation features into the final shading sub-network to obtain the final shading result; and

The sixth unit 660 is configured to adjust parameters of the neural network model based on at least the final coloring result and the second sample image.

According to some embodiments, the sixth unit 660 may be further configured to: calculate a first loss value based on the final colorization result and the second sample image; calculate a second loss value based on at least the object segmentation feature; and based on the first loss value and the second loss value to adjust the parameters of the neural network model.

According to some embodiments, the fifth unit 650 may be further configured to: combine the pre-coloring result and the plurality of object segmentation features in the feature channel dimension; and input the combined result into the final shading sub-network to obtain the final shading result.

According to some embodiments, the plurality of first sample images may be consecutive frames in the video to be rendered.

According to some embodiments, the consecutive plurality of frames may be consecutive 3 frames, and wherein the third unit may be further configured to obtain a predicted pre-coloring result for an intermediate frame among the 3 frames.

According to some embodiments, the object segmentation feature may be an object segmentation mask.

According to some embodiments, each of the plurality of first sample images may include the same kind and the same number of objects, and wherein the second sample image may include the same objects as in each of the first sample images The object corresponding to the color reference object.

According to some embodiments, the second sample image may be a color image in the Lab color space.

FIG. 7 shows a structural block diagram of a video processing apparatus 700 according to an embodiment of the present disclosure.

As shown in FIG. 7 , the apparatus 700 includes: a coloring unit 710 configured to input the target coloring frame of the video to be colorized, at least one frame adjacent to the target coloring frame, and the color reference image into the neural network model, so as to obtain the target coloring frame The final coloring result of the coloring frame, where the neural network model is obtained by training according to the above method.

The present disclosure also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to At least one processor is enabled to perform any of the methods described above.

The present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform any one of the above methods.

The present disclosure also provides a computer program product, including a computer program, wherein the computer program implements any one of the above methods when executed by a processor.

Referring to FIG. 8 , a structural block diagram of an electronic device 800 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 8 , the device 800 includes a computing unit 801 that can be executed according to a computer program stored in a read only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803 Various appropriate actions and handling. In the RAM 803, various programs and data necessary for the operation of the device 800 can also be stored. The computing unit 801 , the ROM 802 , and the RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to bus 804 .

Various components in the device 800 are connected to the I/O interface 805 , including: an input unit 806 , an output unit 807 , a storage unit 808 , and a communication unit 809 . The input unit 806 may be any type of device capable of inputting information to the device 800, the input unit 806 may receive input numerical or character information, and generate key signal input related to user settings and/or function control of the electronic device, and may Including but not limited to mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and/or remote control. The output unit 807 may be any type of device capable of presenting information, and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. The storage unit 808 may include, but is not limited to, magnetic disks and optical disks. Communication unit 809 allows device 800 to exchange information/data with other devices through computer networks such as the Internet and/or various telecommunication networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and/or chipsets , such as Bluetooth ^™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and/or the like.

Computing unit 801 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 801 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 801 performs the various methods and processes described above, eg, method 200 . For example, in some embodiments, methods according to embodiments of the present disclosure may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 800 via ROM 802 and/or communication unit 809 . When a computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured by any other suitable means (eg, by means of firmware) to perform methods according to embodiments of the present disclosure.

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-described methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is limited only by the appended claims and their equivalents. Various elements of the embodiments or examples may be omitted or replaced by equivalents thereof. Furthermore, the steps may be performed in an order different from that described in this disclosure. Further, various elements of the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims (21)

1. A training method for a neural network model, wherein the neural network model comprises a feature extraction sub-network, a pre-coloring sub-network, an object segmentation sub-network and a final coloring sub-network, the method comprising: Acquire a plurality of first sample images and color second sample images corresponding to the plurality of first sample images, wherein each first sample image in the plurality of first sample images includes processing object; inputting the plurality of first sample images into the feature extraction sub-network to obtain first features; inputting the second sample image into the feature extraction sub-network to obtain a second feature; inputting the first feature and the second feature into the pre-coloring sub-network to obtain a predicted pre-coloring result for at least one first sample image of the plurality of first sample images; The first feature is input into the object segmentation sub-network to obtain a plurality of object segmentation features, wherein each object segmentation feature in the plurality of object segmentation features is respectively for the plurality of first sample images. A corresponding first sample image of ; inputting the pre-shading result and the plurality of object segmentation features into the final shading sub-network to obtain a final shading result; and The parameters of the neural network model are adjusted based on at least the final coloring result and the second sample image. 2. The method of claim 1, wherein adjusting parameters of the neural network model based on at least the final colorization result and the second sample image comprises: calculating a first loss value based on the final shading result and the second sample image; calculating a second loss value based on at least the object segmentation feature; and Based on the first loss value and the second loss value, parameters of the neural network model are adjusted. 3. The method of claim 1, wherein inputting the pre-shading results and the plurality of object segmentation features into the final shading sub-network comprises: combining the pre-colored result and the plurality of object segmentation features in the feature channel dimension; and The combined result is input into the final shading sub-network to obtain the final shading result. 4. The method of claim 1, wherein the plurality of first sample images are consecutive frames in the video to be rendered. 5. The method of claim 4, wherein the consecutive plurality of frames are consecutive 3 frames, and wherein the acquisition is for at least one first sample image of the plurality of first sample images Predicted preshading results include: Obtain the predicted pre-shading result for the intermediate frame among the 3 frames. 6. The method of any one of claims 1 to 5, wherein the object segmentation feature is an object segmentation mask. 7. The method of any one of claims 1 to 5, wherein each first sample image of the plurality of first sample images comprises the same kind and the same number of objects, and wherein, The second sample images include color reference objects corresponding to objects in each of the first sample images. 8. The method of any one of claims 1 to 5, wherein the second sample image is a color image in the Lab color space. 9. A video processing method, comprising: Input the target coloring frame of the video to be colorized, at least one frame adjacent to the target coloring frame, and the color reference image into the neural network model to obtain a final coloring result for the target coloring frame, wherein the neural network The model is trained by the method according to any one of claims 1 to 8. 10. A training device for a neural network model, wherein the neural network model comprises a feature extraction sub-network, a pre-coloring sub-network, an object segmentation sub-network and a final coloring sub-network, the device comprising: A first unit configured to acquire a plurality of first sample images and color second sample images corresponding to the plurality of first sample images, wherein each of the plurality of first sample images A sample image includes the object to be processed; a second unit, configured to input the plurality of first sample images into the feature extraction sub-network to obtain first features; The second unit is further configured to input the second sample image into the feature extraction sub-network to obtain a second feature; a third unit configured to input the first feature and the second feature into the pre-coloring sub-network to obtain a prediction for at least one first sample image among the plurality of first sample images The pre-coloring result of ; The fourth unit is configured to input the first feature into the object segmentation sub-network to obtain a plurality of object segmentation features, wherein each object segmentation feature in the plurality of object segmentation features is for the multiple object segmentation features respectively. a corresponding one of the first sample images; a fifth unit configured to input the pre-shading result and the plurality of object segmentation features into the final shading sub-network to obtain a final shading result; and A sixth unit, configured to adjust parameters of the neural network model based on at least the final coloring result and the second sample image. 11. The apparatus of claim 10, wherein the sixth unit is further configured to: calculating a first loss value based on the final shading result and the second sample image; calculating a second loss value based on at least the object segmentation feature; and Based on the first loss value and the second loss value, parameters of the neural network model are adjusted. 12. The apparatus of claim 10, wherein the fifth unit is further configured to: combining the pre-colored result and the plurality of object segmentation features in the feature channel dimension; and The combined result is input into the final shading sub-network to obtain the final shading result. 13. The apparatus of claim 10, wherein the plurality of first sample images are consecutive frames in the video to be rendered. 14. The apparatus of claim 13, wherein the consecutive plurality of frames are consecutive 3 frames, and wherein the third unit is further configured to: Obtain the predicted pre-shading result for the intermediate frame among the 3 frames. 15. The apparatus of any of claims 10 to 14, wherein the object segmentation feature is an object segmentation mask. 16. The apparatus of any one of claims 10 to 14, wherein each first sample image of the plurality of first sample images comprises the same kind and the same number of objects, and wherein, The second sample images include color reference objects corresponding to objects in each of the first sample images. 17. The apparatus of any one of claims 10 to 14, wherein the second sample image is a color image in the Lab color space. 18. A video processing device, comprising: a coloring unit, configured to input a target coloring frame of the video to be colorized, at least one frame adjacent to the target coloring frame, and a color reference image into the neural network model, so as to obtain a final coloring result for the target coloring frame, Wherein, the neural network model is obtained by training the method according to any one of claims 1 to 8. 19. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-9 Methods. 20. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any of claims 1-9. 21. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-9.

Images