CN111402130A - Data processing method and data processing device - Google Patents

Abstract

The embodiment of the application discloses a data processing method, which is applied to the field of artificial intelligence, in particular to an image processing technology, and comprises the following steps: acquiring a sequence of frames, frames in the sequence of frames having a first resolution; determining at least two frame groups from the frame sequence, wherein the frame groups comprise a first target frame and at least two adjacent frames of the first target frame, the first target frame is any one frame in the frame sequence, and the adjacent frames are frames except the first target frame in the frame sequence; determining the characteristics of each frame group in at least two frame groups through a three-dimensional convolution neural network, wherein the size of a convolution kernel in the three-dimensional convolution neural network in a time dimension is positively correlated with the number of frames in the frame groups; fusing the characteristics of each frame group of the at least two frame groups to determine the detail characteristics of the first target frame; and acquiring a first target frame with a second resolution according to the detail features and the first target frame, wherein the second resolution is greater than the first resolution.

Description

Data processing method and data processing device

technical field

The present application relates to the technical field of image processing, and in particular, to a data processing method and a data processing device.

Background technique

Super-resolution (SR), referred to as super-resolution, refers to the technology of reconstructing corresponding high-resolution images from low-resolution images. By upsampling and enlarging the low-resolution image, and filling in the details by means of image prior knowledge, the corresponding high-resolution image is generated. Super-resolution technology has important application value in the fields of high-definition television, surveillance equipment, satellite imagery and medical imaging.

Video super resolution (VSR), referred to as video super-resolution, is to generate a corresponding high-resolution video from a low-resolution video. A core operation that distinguishes video super-resolution from image super-resolution is motion compensation, that is, to The multi-frame images near the target frame are subjected to information extraction and fusion, and the similarity of adjacent frames before and after the target frame is used to obtain detailed information, thereby generating high-resolution video. Specifically, in the prior art, at least 7 low-resolution frame sequences including target frames are input into a three-dimensional (3D) convolutional neural network, and detailed information is implicitly extracted through a 3*3*3 convolution kernel. And fusion, because the size of the 3*3*3 convolution kernel in the time dimension is 3, that is, 3 frames can be processed at the same time, and then the convolution kernel slides 1 frame in turn for information extraction to obtain detailed information, according to the details. information and enlarge the low-resolution target frame by upsampling, and finally obtain the high-resolution target frame.

When the prior art uses a 3D convolutional neural network for motion compensation, it is limited by the size of the convolution kernel, and 3 frames are processed each time the sliding process is performed. When the accumulated kernel slides to the 1st, 2nd, and 3rd frames, due to the lack of guidance information of the target frame, the feature extraction will be relatively blind and the feature extraction efficiency will be low.

SUMMARY OF THE INVENTION

The embodiment of the present application provides a data processing method for frame sequence super-resolution, which can improve the efficiency of feature extraction and reduce the amount of calculation.

A first aspect of an embodiment of the present application provides a data processing method, including: a data processing device acquires a frame sequence, where the frames in the frame sequence have a first resolution; the data processing device determines from the frame sequence at least Two frame groups, the frame group includes a first target frame and at least two adjacent frames of the first target frame, the first target frame is any frame in the frame sequence, and the adjacent frame is Frames other than the first target frame in the frame sequence; the data processing device determines the characteristics of each frame group in the at least two frame groups through a three-dimensional convolutional neural network, and the The feature indication is based on the detailed information obtained from the adjacent frames in each frame group based on the first target frame, and the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension is the same as the frame in the frame group. are positively correlated; the data processing device fuses the features of each frame group in the at least two frame groups to determine the detail feature of the first target frame, and the detail feature indication is based on the first target frame , the detail information obtained from the adjacent frames in the at least two frame groups; the data processing device obtains the first target frame with the second resolution according to the detail feature and the first target frame, so The second resolution is greater than the first resolution. Optionally, the larger the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension, the larger the number of frames in the frame group. The dimensions of the dimension are equal to the number of frames in the frame group.

In the data processing method provided by the embodiment of the present application, in the process of super-dividing the first target frame in the frame sequence, at least two frame groups are first determined from the frame sequence, the frame groups include the first target frame, and the frame groups are respectively input into the three-dimensional frame. The convolutional neural network extracts the group features of the frame group, and then fuses the group features to determine the detail features of the first target frame, and converts the first target frame of the first resolution to the first target of the second resolution according to the detail features. frame. In this data processing method, since the frame group determined from the frame sequence includes the first target frame, and the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension is positively related to the number of frames in the frame group, according to the three-dimensional The size of the convolution kernel in the convolutional neural network in the time dimension is set in the room of the frame in the frame group. Therefore, when extracting the features of the frame group through the three-dimensional convolutional neural network, it can not only reduce the sliding of the convolution kernel, but also reduce the amount of calculation. The guidance of the target frame can also be obtained, and the detailed feature extraction efficiency is high.

In a possible implementation manner of the first aspect, the frame group includes the first target frame and two adjacent frames.

In the data processing method provided in this embodiment of the present application, the determined number of frame groups is 3 frames, that is, the size of the convolution kernel in the time dimension is 3, and a 3D convolutional neural network with a convolution kernel size of 3*3*3 can be used. The network performs feature extraction, and the amount of computation is small.

In a possible implementation manner of the first aspect, the two adjacent frames include a first adjacent frame and a second adjacent frame, and the first adjacent frame is associated with the first target in the frame sequence The interval between frames is equal to the interval between the second adjacent frame and the first target frame in the frame sequence.

In the data processing method provided by the embodiment of the present application, the first adjacent frame and the second adjacent frame in the frame group are symmetrical with respect to the target frame in the frame sequence. Considering the continuity of motion, for the frame sequence obtained continuously through the same time interval, the two The adjacent frames are symmetrical about the target frame in the time dimension, which can extract features more efficiently.

In a possible implementation manner of the first aspect, the at least two frame groups include three frame groups.

Since the number of frame groups is larger, the computational complexity of feature extraction is larger, and the smaller the number of frame groups, the less detailed information can be obtained. The data processing method provided by the embodiment of the present application can achieve a better balance between providing sufficient information and reducing the amount of calculation by fusing the features of three frame groups to perform detailed feature extraction.

In a possible implementation manner of the first aspect, the method further includes: the data processing apparatus aligns the frames in each of the at least two frame groups, and determines the aligned at least two frame groups Determining the characteristics of each frame group in the at least two frame groups by the data processing device through a three-dimensional convolutional neural network includes: the data processing device determining the aligned at least two frame groups through a three-dimensional convolutional neural network Features of each frame group in .

In the data processing method provided by the embodiment of the present application, before extracting the features of the frame group through the three-dimensional convolutional neural network, frame alignment processing can be performed on the first frame group, thereby, the features of the frame group can be extracted more effectively.

In a possible implementation manner of the first aspect, the data processing apparatus aligns the frames in each of the at least two frame groups, and determining the aligned at least two frame groups includes: the data processing The apparatus determines a homography matrix between all two consecutive frames in a queue composed of frames in the at least two frame groups; the data processing apparatus determines the aligned at least two frame groups according to the homography matrices.

In the data processing method provided by the embodiment of the present application, the frame group of the first target frame is aligned by the method of the homography matrix, which can reduce the amount of calculation.

In a possible implementation manner of the first aspect, the method further includes: the data processing apparatus determines a weight of the feature of each frame group in the at least two frame groups; the data processing apparatus fuses the The characteristics of each frame group in the at least two frame groups to determine the detailed characteristics of the first target frame include: the data processing device fuses each frame group in the at least two frame groups according to the weights features to determine the detailed features of the first target frame.

In the data processing method provided by the embodiments of the present application, when multiple features are fused, an attention mask can be calculated through a deep learning network attention mechanism, the weights of the features of each frame group can be determined, and the features can be fused accordingly to finally determine the target. Details of the frame.

A second aspect of an embodiment of the present application provides a data processing apparatus, the apparatus includes: an acquisition unit, configured to acquire a frame sequence, where the frames in the frame sequence have a first resolution; a determination unit, configured to obtain a frame sequence from the At least two frame groups are determined in the frame sequence, the frame group includes a first target frame and at least two adjacent frames of the first target frame, and the first target frame is any frame in the frame sequence, The adjacent frames are frames other than the first target frame in the frame sequence; the determining unit is further configured to determine the feature of each frame group in the at least two frame groups through a three-dimensional convolutional neural network , the feature indication of each frame group is based on the first target frame, the detailed information obtained from the adjacent frames in each frame group, and the convolution kernel in the three-dimensional convolutional neural network is in the time dimension. The size is positively related to the number of frames in the frame group; the processing unit is used for fusing the features of each frame group in the at least two frame groups to determine the detail features of the first target frame, the detail features Indicates the detail information obtained from the adjacent frames in the at least two frame groups based on the first target frame; the obtaining unit is further configured to obtain, according to the detail feature and the first target frame A first target frame of a second resolution, the second resolution being greater than the first resolution.

In a possible implementation manner of the second aspect, the frame group includes the first target frame and two adjacent frames.

In a possible implementation manner of the second aspect, the two adjacent frames include a first adjacent frame and a second adjacent frame, and the first adjacent frame is associated with the first target in the frame sequence The interval between frames is equal to the interval between the second adjacent frame and the first target frame in the frame sequence.

In a possible implementation manner of the second aspect, the at least two frame groups include three frame groups.

In a possible implementation manner of the second aspect, the determining unit is further configured to align the frames in each of the at least two frame groups, and determine the aligned at least two frame groups; the The determining unit is specifically configured to: determine the feature of each frame group in the aligned at least two frame groups through a three-dimensional convolutional neural network.

In a possible implementation manner of the second aspect, the determining unit is specifically configured to: determine a homography matrix between all consecutive two frames in a queue composed of frames in the at least two frame groups; The homography matrix determines the aligned at least two frame groups.

In a possible implementation manner of the second aspect, the determining unit is further configured to: the determining unit is further configured to: determine, through a deep neural network, a weight of the feature of each frame group in the at least two frame groups value; the processing unit is specifically configured to: fuse the features of each frame group in the at least two frame groups according to the weight value to determine the detailed feature of the first target frame.

In a possible implementation manner of the second aspect, the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension is equal to the number of frames in the frame group.

A third aspect of the embodiments of the present application provides a computer program product containing instructions, characterized in that, when it runs on a computer, the computer is caused to execute the first aspect or any one of the possible implementations of the first aspect. Methods.

A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, including instructions, characterized in that, when the instructions are executed on a computer, the computer is made to execute the first aspect or any one of the first aspects. method of implementation.

A fifth aspect of the embodiments of the present application provides a chip system, where the chip system includes a processor, and the processor is configured to read and execute a computer program stored in a memory, so as to execute the functions involved in any possible implementation manner of any of the foregoing aspects . In one possible design, the system-on-a-chip further includes a memory that is electrically connected to the processor. Further optionally, the chip further includes a communication interface, and the processor is connected to the communication interface. The communication interface is used for receiving data and/or information to be processed, the processor obtains the data and/or information from the communication interface, processes the data and/or information, and outputs the processing result through the communication interface. The communication interface may be an input-output interface. The chip system may be composed of chips, or may include chips and other discrete devices.

Wherein, for the technical effect brought by any one of the second aspect, the third aspect, the fourth aspect and the fifth aspect, please refer to the technical effect brought by the corresponding implementation in the first aspect, which will not be repeated here. .

The data processing method provided by the embodiment of the present application has the advantages of:

In the data processing method provided by the embodiment of the present application, in the process of super-dividing the first target frame in the frame sequence, at least two frame groups are first determined from the frame sequence, the frame groups include the first target frame, and the frame groups are respectively input into the three-dimensional frame. The convolutional neural network extracts the group features of the frame group, and then fuses the group features to determine the detail features of the first target frame, and converts the first target frame of the first resolution to the first target of the second resolution according to the detail features. frame. In this data processing method, since the frame group determined from the frame sequence includes the first target frame, and the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension matches the number of frames in the frame group, therefore, by When the 3D convolutional neural network extracts the group features of the frame group, it can not only reduce the sliding of the convolution kernel and reduce the amount of calculation, but also obtain the guidance of the target frame, and the extraction efficiency of details is high.

Description of drawings

1 is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present application;

2 is a schematic diagram of an application environment provided by an embodiment of the present application;

3 is a schematic structural diagram of a convolutional neural network provided by an embodiment of the present application;

4 is a schematic structural diagram of another convolutional neural network provided by an embodiment of the present application;

5 is a schematic diagram of an application scenario of the data processing method in the embodiment of the present application;

6 is a schematic diagram of another application scenario of the data processing method in the embodiment of the present application;

FIG. 7 is a schematic diagram of an embodiment of a data processing method provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of an embodiment of a frame alignment method in an embodiment of the present application;

FIG. 9 is a schematic diagram of an embodiment of homography matrix calculation in the embodiment of the present application;

FIG. 10 is a schematic diagram of an embodiment of timing grouping in an embodiment of the present application;

11 is a schematic diagram of an embodiment of feature fusion between groups in an embodiment of the present application;

12 is a schematic diagram of another embodiment of performing feature fusion between groups in an embodiment of the present application;

13 is a schematic diagram of an embodiment of sampling amplification in the embodiment of the present application;

FIG. 14 is a schematic diagram of another embodiment of the data processing method provided by the embodiment of the present application;

FIG. 15 is a schematic diagram of an embodiment of a data processing apparatus provided by an embodiment of the present application;

FIG. 16 is a structural diagram of a chip hardware provided by an embodiment of the present application.

Detailed ways

The embodiment of the present application provides a data processing method, which is used for super-resolution of a frame sequence obtained by continuous shooting, which can reduce the amount of calculation and improve the effect of detail feature extraction.

The terms involved in the embodiments of the present application are briefly introduced below.

Video: Video consists of a series of static images, in which the image is usually called a frame, and the number of frames transmitted or displayed per second is called the frame rate (frames per second, FPS). The higher the frame rate, the smoother the picture; the frame The smaller the rate, the more jumpy the picture is. When the video frame rate is not lower than 24fps, due to the phenomenon of "persistence of vision", the human eye cannot distinguish a single static picture, and it looks like a smooth and continuous visual effect, such a continuous frame sequence is a video. The video in the embodiment of the present application refers to a frame sequence obtained by continuous shooting.

Image resolution (resolution): The resolution refers to the amount of information stored in the image, which is how many pixels per inch of the image. The unit of resolution is PPI (pixels per inch), usually called pixels per inch. The expression is: the number of horizontal pixels × the number of vertical pixels, and the common resolutions are 1280*720PPI; 1920*1080PPI and other specifications. The higher the resolution, the larger the image; conversely, the smaller the image.

Video super-resolution: referred to as "video super-resolution", a corresponding high-resolution video is reconstructed from a low-resolution video. By upsampling and enlarging the low-resolution image, the corresponding high-resolution image is generated by filling in the details by means of image prior knowledge, image self-similarity, and multi-frame image complementary information.

In a sequence of frames obtained by continuous shooting, adjacent frames are usually very similar. For the super-resolution of frame sequences, such as video streaming super-resolution, video surveillance super-resolution, and old movie high-definition tasks, simply applying image super-resolution methods to video frame-by-frame processing cannot achieve ideal results. On the one hand, since the image super-resolution does not consider the information of the previous and previous frames and lacks temporal continuity, the generated high-resolution video will have artifacts such as flicker and jitter, which greatly affects viewing and subsequent applications; on the other hand, the image super-resolution The resolution method lacks the complementary information of the front and rear frames, and the low information utilization limits its super-resolution performance. Therefore, on the basis of image super-resolution, video super-resolution (VSR), which can effectively utilize the information of previous and previous frames, has received more attention.

Motion compensation: is a method of describing the difference between adjacent frames. In the frame sequence obtained by continuous shooting, adjacent frames are usually very similar, that is to say, they contain a lot of redundancy. Simple motion compensation is to subtract the reference frame from the current frame to obtain the difference between frames, that is, super-division technology. Details to be obtained in .

A core operation of video super-resolution is the extraction and fusion of multi-frame spatiotemporal information, so motion compensation is needed to deal with the motion between video frames. According to the way of motion compensation, existing video super-resolution methods can be divided into two categories: explicit motion compensation and implicit motion compensation.

The explicit motion compensation method uses optical flow and other means to directly distort and align the image in the preprocessing stage, which is computationally intensive and has obvious artifacts.

Implicit motion compensation uses operations such as 3D convolution or deformable convolution to implicitly perform motion compensation in neural networks. Such methods are limited by the structure of 3D convolution and deformable convolution, and require a huge amount of computation. , causing it to run very slowly. Therefore, a video super-segmentation method that more effectively fuses the timing information of multiple frames in a video sequence under the condition of limited computation has become a hot research topic in the current industry and academia.

Three-dimensional convolution, referred to as 3D convolution, is a convolution applied to three-dimensional data. Its convolution kernel contains three dimensions, which is one more depth dimension than the commonly used 2D convolution kernel in addition to the length and width dimensions. The depth dimension can be multiple frames of a video or different slices of a stereoscopic image. 3D convolution can effectively extract spatiotemporal information in video sequences or 3D images, and is often used in tasks such as motion recognition, medical image processing, and video processing. The depth dimension in the three-dimensional convolution kernel in the embodiment of the present application refers to the time dimension, which refers to multiple frames obtained at different time points in the video.

Frame Sequence: Refers to a sequence of multiple frames of images.

Frame group: refers to multiple frames that are simultaneously input to the three-dimensional convolutional neural network for information extraction in the embodiment of the present application, including the target frame and the adjacent frames of the target frame.

The embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Those of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The term "and/or" that appears in this application can be an association relationship to describe associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, and A and B exist at the same time , the case of B alone, where A and B can be singular or plural. In addition, the character "/" in this application generally indicates that the related objects are an "or" relationship. In this application, "at least one" means one or more, and "plurality" means two or more. "At least one item(s) below" or similar expressions thereof refer to any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (a) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c may be single or multiple .

The terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a distinguishing manner adopted when describing objects with the same attributes in the embodiments of the present application. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or device comprising a series of elements is not necessarily limited to those elements, but may include no explicit or other units inherent to these processes, methods, products, or devices.

Figure 1 shows a schematic diagram of an artificial intelligence main frame, which describes the overall workflow of an artificial intelligence system and is suitable for general artificial intelligence field requirements.

The above artificial intelligence theme framework will be explained from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data has gone through the process of "data-information-knowledge-wisdom".

The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.

(1) Infrastructure:

The infrastructure provides computing power support for artificial intelligence systems, realizes communication with the outside world, and supports through the basic platform. Communication with the outside world through sensors; computing power is provided by smart chips (hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA); the basic platform includes distributed computing framework and network-related platform guarantee and support, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with external parties to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.

(2) Data

The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, and text, as well as IoT data from traditional devices, including business data from existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

Among them, machine learning and deep learning can perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on data.

Reasoning refers to the process of simulating human's intelligent reasoning method in a computer or intelligent system, using formalized information to carry out machine thinking and solving problems according to the reasoning control strategy, and the typical function is search and matching.

Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

(4) General ability

After the above-mentioned data processing, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, image identification, etc.

(5) Smart products and industry applications

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, and the productization of intelligent information decision-making and implementation of applications. Its application areas mainly include: intelligent manufacturing, intelligent transportation, Smart home, smart medical care, smart security, autonomous driving, safe city, smart terminals, etc.

Referring to FIG. 2 , an embodiment of the present application provides a system architecture 200 . The data collection device 260 is used to collect continuously shot frame sequences and store them in the database 230 , and the training device 220 generates the target model/rule 201 based on the frame sequence data maintained in the database 230 . The following will describe in more detail how the training device 220 obtains the target model/rule 201 based on the frame sequence data. The target model/rule 201 can be used in application scenarios such as video super-score and image sequence super-score.

The target model/rule 201 may be obtained based on a deep neural network, and the deep neural network will be introduced below.

来描述：从物理层面深度神经网络中的每一层的工作可以理解为通过五种对输入空间(输入向量的集合)的操作，完成输入空间到输出空间的变换(即矩阵的行空间到列空间)，这五种操作包括：1、升维/降维；2、放大/缩小；3、旋转；4、平移；5、“弯曲”。其中1、2、3的操作由

完成，4的操作由+b完成，5的操作则由a()来实现。这里之所以用“空间”二字来表述是因为被分类的对象并不是单个事物，而是一类事物，空间是指这类事物所有个体的集合。其中，W是权重向量，该向量中的每一个值表示该层神经网络中的一个神经元的权重值。该向量W决定着上文所述的输入空间到输出空间的空间变换，即每一层的权重W控制着如何变换空间。训练深度神经网络的目的，也就是最终得到训练好的神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。因此，神经网络的训练过程本质上就是学习控制空间变换的方式，更具体的就是学习权重矩阵。The work of each layer in a deep neural network can be expressed mathematically

To describe: from the physical level, the work of each layer in the deep neural network can be understood as completing the transformation from the input space to the output space (that is, the row space of the matrix to the column through five operations on the input space (set of input vectors). Space), these five operations include: 1. Dimension raising/lowering; 2. Enlarging/reducing; 3. Rotation; 4. Translation; 5. "Bending". Among them, the operations of 1, 2, and 3 are determined by

Complete, the operation of 4 is completed by +b, and the operation of 5 is realized by a(). The reason why the word "space" is used here is because the object to be classified is not a single thing, but a type of thing, and space refers to the collection of all individuals of this type of thing. Among them, W is the weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. This vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how the space is transformed. The purpose of training the deep neural network is to finally obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by the vectors W of many layers). Therefore, the training process of the neural network is essentially learning the way to control the spatial transformation, and more specifically, learning the weight matrix.

Because it is hoped that the output of the deep neural network is as close as possible to the value you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then update each layer of neural network according to the difference between the two. The weight vector of the network (of course, there is usually an initialization process before the first update, that is, the parameters are pre-configured for each layer in the deep neural network), for example, if the predicted value of the network is high, adjust the weight vector to make it Predict lower and keep adjusting until the neural network can predict the actual desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function (loss function) or objective function (objective function), which are important for measuring the difference between the predicted value and the target value equation. Among them, taking the loss function as an example, the higher the output value of the loss function (loss), the greater the difference, then the training of the deep neural network becomes the process of reducing the loss as much as possible.

The target models/rules obtained by training the device 220 can be applied in different systems or devices. In FIG. 2 , the execution device 210 is configured with an I/O interface 212 for data interaction with external devices, and a “user” can input data to the I/O interface 212 through the client device 240 .

The execution device 210 can call data, codes, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

The calculation module 211 uses the target model/rule 201 to process the input data. Taking image super-score as an example, the calculation module 211 can analyze the input image or image sequence to obtain image features.

The association function module 213 may preprocess the image data in the calculation module 211, such as frame alignment or image grouping.

The association function module 214 may preprocess the image data in the calculation module 211, such as frame alignment or image grouping.

Finally, the I/O interface 212 returns the processing result to the client device 240, which is provided to the user.

More deeply, the training device 220 can generate corresponding target models/rules 201 based on different data for different targets, so as to provide users with better results.

In the case shown in FIG. 2 , the user can manually specify data in the input execution device 210 , eg, operate in the interface provided by the I/O interface 212 . In another case, the client device 240 can automatically input data to the I/O interface 212 and obtain the result. If the client device 240 automatically inputs data and needs to obtain the user's authorization, the user can set the corresponding permission in the client device 240 . The user can view the result output by the execution device 210 on the client device 240, and the specific presentation form can be a specific manner such as display, sound, and action. The client device 240 may also serve as a data collection terminal to store the collected training data in the database 230 .

It is worth noting that FIG. 2 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship among the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 2 , The data storage system 250 is an external memory relative to the execution device 210 , and in other cases, the data storage system 250 may also be placed in the execution device 210 .

A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. Learning at multiple levels. As a deep learning architecture, CNN is a feed-forward artificial neural network, taking image processing as an example, in which each neuron responds to overlapping regions in the image input into it .

As shown in FIG. 3 , a convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional/pooling layer 120 , where the pooling layer is optional, and a neural network layer 130 .

Convolutional layer/pooling layer 120:

Convolutional layer:

As shown in FIG. 3, the convolutional/pooling layer 120 may include layers 121-126 as examples. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, and layer 124 is a convolutional layer. Layers are pooling layers, 125 are convolutional layers, and 126 are pooling layers; in another implementation, 121 and 122 are convolutional layers, 123 are pooling layers, 124 and 125 are convolutional layers, and 126 are pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or it can be used as the input of another convolutional layer to continue the convolution operation.

Taking the convolution layer 121 as an example, the convolution layer 121 may include many convolution operators, which are also called kernels, and their role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator can be essentially a weight matrix. This weight matrix is usually pre-defined. In the process of convolving an image, the weight matrix is usually pixel by pixel along the horizontal direction on the input image ( Or two pixels after two pixels...depending on the value of stride), which completes the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will extend to the input the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension of the convolutional output, but in most cases a single weight matrix is not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to extract unwanted noise in the image. Perform fuzzification... The dimensions of the multiple weight matrices are the same, and the dimension of the feature maps extracted from the weight matrices with the same dimensions are also the same, and then the multiple extracted feature maps with the same dimensions are combined to form the output of the convolution operation .

Depending on the dimension of the data to be processed, convolution kernels also come in a variety of formats. Commonly used convolution kernels include two-dimensional convolution kernels and three-dimensional convolution kernels. The two-dimensional convolution kernel is mainly used to process two-dimensional image data, while the three-dimensional convolution kernel can be applied to video processing, stereo image processing, etc. due to the increased depth/time dimension. Compared with the two-dimensional convolution kernel, the amount of parameters and computation required for the three-dimensional convolution kernel with one additional dimension increases significantly. In practical applications, it is necessary to carefully design the three-dimensional convolution network: when the convolution kernel is large, the number of slips is small, but the amount of calculation increases greatly; when the convolution kernel is small, the number of slips is large, and the features of the depth/time dimension Extraction is blind and inefficient, which limits the performance of the network.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications, and each weight matrix formed by the weight values obtained by training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features; with the convolutional neural network As the depth of the network 100 deepens, the features extracted by the later convolutional layers (eg 126) become more and more complex, such as features such as high-level semantics. Features with higher semantics are more suitable for the problem to be solved. For the convenience of describing the network structure, multiple convolutional layers can be referred to as a block.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer, that is, each layer 121-126 exemplified by 120 in Figure 3, which can be a convolutional layer followed by a layer The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. During image processing, the only purpose of pooling layers is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a max pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the average value of the pixel values in the image within a certain range. The max pooling operator can take the pixel with the largest value within a specific range as the result of max pooling. Also, just as the size of the weight matrix used in the convolutional layer should be related to the size of the image, the operators in the pooling layer should also be related to the size of the image. The size of the output image after processing by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.

Neural network layer 130:

After being processed by the convolutional layer/pooling layer 120, the convolutional neural network 100 is not sufficient to output the required output information. Because as mentioned before, the convolutional layer/pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (required class information or other related information), the convolutional neural network 100 needs to utilize the neural network layer 130 to generate one or a set of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (131, 132 to 13n as shown in FIG. 3) and the output layer 140, and the parameters contained in the multiple hidden layers may be based on specific task types The relevant training data is pre-trained, for example, the task type can include image recognition, image classification, image super-resolution reconstruction and so on.

After the multi-layer hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, the output layer 140 has a loss function similar to the classification cross entropy, and is specifically used to calculate the prediction error, Once the forward propagation of the entire convolutional neural network 100 (as shown in Fig. 3 from 110 to 140 is forward propagation) is completed, the back propagation (as shown in Fig. 3 from 140 to 110 as back propagation) will start to update The weight values and biases of the aforementioned layers are used to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

It should be noted that the convolutional neural network 100 shown in FIG. 3 is only used as an example of a convolutional neural network. In a specific application, the convolutional neural network can also exist in the form of other network models, for example, such as The multiple convolutional layers/pooling layers shown in FIG. 4 are in parallel, and the extracted features are input to the full neural network layer 130 for processing.

The application scenarios of the video super-resolution method are broad, which will be introduced below with reference to Figures 5 and 6 as examples.

Application Scenario 1: HD Streaming Video System

Please refer to FIG. 5, which is a schematic diagram of an application scenario of the data processing method in the embodiment of the present application;

With the popularity of smartphones and tablet computers, streaming video has gradually become one of the current mainstream video entertainment methods. The resolution of video resources provided by streaming video platforms is gradually increasing, which puts forward higher requirements for network bandwidth and network stability. Based on video super-resolution technology, lower-resolution video images can be directly transmitted to users, and the client can perform video super-resolution on low-quality videos with its own computing power, and finally present high-quality images to users. In this way, the bandwidth requirements of the video stream can be greatly reduced without significantly reducing the quality of the high-definition picture.

Application Scenario 2: HD Surveillance System

Please refer to FIG. 6 , which is a schematic diagram of another application scenario of the data processing method in the embodiment of the present application;

Video surveillance is an important part of the safe city system. More and more video surveillance is deployed in every corner of the city to protect the safety of the city. Limited by the camera quality, installation location, limited storage space and other unfavorable conditions, the picture quality of some video surveillance is poor, which limits its subsequent application. As shown in FIG. 2 , in this embodiment of the present application, a low-resolution video surveillance picture can be converted into a high-resolution high-definition picture. Through the information of the frames before and after and the prior knowledge of the image, the effective recovery of a large number of details in the monitoring picture is realized, which provides more effective and rich information for the subsequent video analysis, and improves the robustness of the safe city system.

Existing video super-resolution methods mainly include optical flow-based video super-resolution methods and implicit motion compensation-based video super-resolution methods, which are briefly introduced below.

The optical flow-based video super-segmentation method, that is, an explicit motion compensation method, estimates the dense optical flow from adjacent frames to the intermediate frame for the input multi-frame image frame by frame, and aligns the adjacent frame distortion to the intermediate frame based on the optical flow estimation results. , form an aligned image sequence, perform feature extraction and fusion on the aligned image sequence, and output the super-resolution result of the intermediate frame.

Because the explicit motion compensation method uses optical flow and other means to directly distort and align the image in the preprocessing stage, limited by the accuracy of the optical flow calculation, there are often large artifacts.

In the video super-segmentation method based on implicit motion compensation, in order to avoid the huge computational load of dense optical flow and its artifacts, a neural network module with motion compensation capability, such as 3D convolutional neural network, is used in image information extraction and fusion. During the process, motion compensation is performed implicitly. Structurally, the 3D convolution kernel adds one dimension to the 2D convolution. Increasing the convolution kernel will lead to a sharp increase in the amount of parameters and computation, so it is difficult to deepen the depth of the 3D convolution kernel in the time dimension. In practical applications, a 3D convolution kernel with a size of 3x3x3 is often used to achieve a balance between performance and computation, and a 3D convolution kernel with a size of 3x3x3 extracts 3 frames of information at a time. In the video super-resolution task, when 7 frames of images are directly input to the 3D convolutional neural network, 3 frames are processed each time, and the convolution kernel slides 1 frame for processing, a total of 5 times, with the fourth frame as the target Take the frame as an example, the distance between the farthest adjacent frame and the target frame is 3 frames, and the first to third frames are in the first calculation. Since the target frame is not included, the guidance of the target frame cannot be obtained when extracting features. The fusion process is indirect and blind, and the feature extraction efficiency is low.

The data processing method provided in this embodiment of the present application may be applied to video, and the video frame rate may be 24, 30, 60, 120, or 300, etc. The specific frame rate of the video is not limited here. In addition, the data processing method can also be applied to a sequence of images captured continuously, for example, a sequence of images captured continuously by a user at different time intervals. In the embodiments of the present application, the data processing objects are collectively referred to as frame sequences.

The data processing method proposed in the embodiment of the present application is used to efficiently fuse multi-frame information in the super-resolution of the frame sequence, which can reduce the amount of calculation and improve the processing speed. Please refer to FIG. 7 , which is a schematic diagram of an embodiment of the data processing method provided by the embodiment of the present application.

701. The data processing apparatus performs frame alignment processing on the frame sequence;

The data processing device performs frame alignment processing on the input frame sequence, and aligns the same picture content in different frames to obtain aligned video frames, which can reduce picture differences between frames and reduce the difficulty of subsequent information extraction and fusion.

Optionally, the adjacent frames used for information extraction are aligned with the target frame, for example, for the first target frame, multiple adjacent frames for extracting detailed information from the first target frame are aligned with the first target frame, such as the first target frame. The 6 adjacent frames of the target frame are aligned with the first target frame. Similarly, a frame alignment process is performed for each target frame that is over-segmented.

Optionally, align multiple frames that are simultaneously input to the convolutional neural network for information extraction, for example, 3 frames containing the target frame are aligned before being input to the 3D convolutional neural network.

Optionally, the data processing method provided by this embodiment of the present application implements fast frame alignment by using a homography matrix method. As an example to facilitate understanding, the following briefly introduces the principle of fast frame alignment and the characteristics of the homography matrix.

For the video frames obtained by continuous shooting, the motion between frames, that is, the change of the picture content between the two frames before and after, can be composed of two parts: camera motion and object motion, and the camera motion can be roughly described by the homography matrix, so this method Coarse motion compensation is achieved using homography-based frame alignment.

One plane is transformed into another plane by perspective transformation, and the homography matrix can describe the perspective transformation of the plane. A homography matrix has the following properties:

1) The homography matrix of A to C can be calculated by the homography matrix of A to B and the homography matrix of B to C:

H _A→C = H _A→B ·H _B→C

2), the homography matrix from B to A is the inverse matrix of the homography matrix from A to B

Please refer to FIG. 8 , which is a schematic diagram of an embodiment of a frame alignment method in an embodiment of the present application.

The data processing device calculates the homography matrix between two consecutive frames. Optionally, the data processing device checks the homography matrix. Due to the complexity of motion, some samples are not suitable for alignment, such as fixed surveillance cameras. There is no camera operation, or the motion of objects in the inter-frame motion is too large, and inter-frame alignment of such samples will lead to alignment errors. In order to avoid erroneous alignment from affecting subsequent feature extraction and fusion, optionally, an applicability test is performed before frame alignment, and the data processing device judges whether the frame sequence is suitable for alignment through the homography matrix check. If it is not suitable, the frame sequence of the original input is directly output.

In the embodiment of the present application, the method of homography matrix is used to realize frame alignment, which can reduce the amount of calculation and speed up the frame alignment. The following is an introduction in comparison with the existing optical flow method.

Please refer to FIG. 9 , which is a schematic diagram of an embodiment of the calculation of the homography matrix in the embodiment of the present application. For each frame, the data processing device only calculates the basic homography matrix with the previous frame, and the homography matrix between other frames is calculated by the basic homography matrix according to the properties of the homography matrix. Considering a video of M frames For processing, each time 2N+1 frames are taken as a set of network inputs, that is, a target frame and 2N adjacent frames, the optical flow-based method needs to calculate between each adjacent frame and the target frame, a total of 2NM times, and In the homography matrix-based alignment method, for each input of 2N+1 frames, the homography matrix only needs to be calculated once, so the total number of calculations is M times, which greatly reduces the amount of calculation and achieves a significant speed improvement. At the same time, this method reduces the deformation of the pixel level in the optical flow calculation, and provides a good alignment for the subsequent super-resolution processing.

It should be noted that step 701 is an optional step, which may or may not be performed, which is not specifically limited here.

Optionally, frame alignment may be performed on the frame groups of the first target frame separately, or frame alignment may be performed on all frame groups of the first target frame jointly, which is not specifically limited here.

702. The data processing apparatus determines at least two frame groups of the first target frame;

When extracting the detail information for the target frame in the frame sequence, it can be considered to obtain the detail information from the adjacent frames of the target frame. The adjacent frame of the target frame, hereinafter referred to as the adjacent frame, is any frame in the frame sequence except the target frame. Optionally, it is a frame that is relatively close to the target frame when the image is captured, and contains partially overlapping image information with the target frame, usually providing detailed information that the target frame does not have.

The data processing device groups the frame sequence, and determines at least two frame groups for information extraction for each frame in the frame sequence. The frame sequence may be an aligned frame sequence output after aligning the frame sequence in step 701, or a frame sequence without frame alignment, which is not specifically limited here.

Taking the first target frame as an example, the data processing apparatus determines a frame group including at least two frames including the first target frame and adjacent frames of the first target frame. The specific number of frame groups is not limited here, for example, it may be 3, 4, or 7, and so on. It can be understood that under the condition that each frame group contains a certain number of frames, the more frame groups, the more detailed information that can be obtained, and the greater the amount of calculation. On the contrary, the fewer the frame groups, the less detailed information can be obtained. , the smaller the amount of computation. In practical applications, the number of frame groups can be determined according to the over-division requirement.

In the embodiment of the present application, the frame group used to simultaneously input the 3D convolutional neural network includes the target frame. Therefore, the target frame can provide guidance information for feature extraction during the convolution process, and improve the effectiveness of the feature extraction. The number of frames matches the size of the depth dimension in the kernel size, and the number of frames in the frame group is odd. Optionally, if a convolution kernel of size 3*3*3 is used, the number of frames in the frame group is 3 frames, that is, one target frame and two adjacent frames. If the convolution kernel is used, the number of frames in the frame group is 5. If a convolution kernel with a size of 7*7*7 is used, the number of frames in the frame group is 7, and the specific size is not limited. As the size of the convolution kernel increases, the amount of computation will increase rapidly. Usually, terminal data processing equipment with limited computing power uses a 3*3*3 size convolution kernel for feature extraction.

There are various methods for selecting adjacent frames in the frame group. Optionally, all adjacent frames in the plurality of frame groups determined for the first target frame and the frame set formed by the first target frame are a group of consecutive frame sequences, and There are no duplicate adjacent frames among all adjacent frames in the plurality of frame groups.

Optionally, the frame group is determined according to the time interval with the target frame, and the N frame groups are denoted as {G_1, G_2, . . . , G_n}, n∈[1:N]. According to the different time distances between the corresponding adjacent frames and the intermediate frames, each group of 3 frames G_n={I _(tn) , I _t , I _(t+n) }, where I _t is the target frame, I _(tn) is the front adjacent frame, specifically n frames before the target frame, I _(t+n) is the rear adjacent frame, specifically n frames after the target frame, n∈[1:N]. For the frame sequence obtained at the same time interval, in the frame group determined by this method, the time interval between the front adjacent frame and the back adjacent frame and the target frame is the same, that is, the time dimension is symmetrical with respect to the target frame, considering the continuity of motion , two adjacent frames symmetrical about the target frame can extract features more effectively.

Taking the input of 7 frames as an example, please refer to FIG. 10 , which is a schematic diagram of an embodiment of timing grouping in the embodiment of the present application.

This module divides adjacent frames into 3 groups according to the time distance from the adjacent frame to the target frame. Frames 1, 4, and 7 are the first frame group, frames 2, 4, and 6 are the second frame group, and frames 3, 4, and 5 are the third frame group .

Since the target frame is included in each frame group as a guide for information fusion within the frame group, the information extraction efficiency can be effectively improved.

In addition, in the prior art, 3 frames are processed each time by means of sliding the convolution kernel, and a total of 6 groups need to be calculated. The data processing method provided in the embodiment of the present application only needs to perform 3 groups of calculations, which can significantly reduce the amount of calculation.

703. The data processing device acquires the group feature of the frame group;

According to the frame group determined in step 702, the data processing device extracts and fuses the features of the frame group respectively through the three-dimensional convolutional neural network. For the convenience of description, the features of the frame group are hereinafter referred to as group features. Optionally, feature extraction and fusion can also be performed in combination with a two-dimensional convolutional neural network and a three-dimensional convolutional neural network. The data processing device obtains the group feature of each frame group in the N frame groups

Optionally, when the frame group is determined according to the temporal distance from the adjacent frame to the target frame, the data processing apparatus may extract and fuse features by using a weight sharing method for different frame groups of the target frame. Weight sharing refers to using the same network structure entity to process different batches of data.

Taking the input of 7 frames as an example, the 4th frame is the target frame. Since it is in the middle of the frame sequence, it is also called the middle frame below. Frames 1, 4, and 7 are the first frame group, frames 2, 4, and 6 are the second frame group, and frames 3, 4, and 5 are the third frame group . Considering that the inter-frame distances corresponding to the three frame groups are 1, 2, and 3, respectively, when extracting the features of each frame group, the same 3D network is used, and the dilation rate of the convolution kernel in the network is set to this value. The inter-frame distances of the frame groups, that is, the first frame group, the second frame group and the third frame group are 1, 2 and 3, respectively. The dilation rate corresponds to the receptive field of the convolution kernel, that is, the spatial coverage. Using a larger dilation rate for groups with large motion can better extract spatial motion information and achieve more efficient intra-group fusion.

Similarly, the data processing apparatus separately extracts group features for multiple frame groups of each target frame, and details are not repeated here.

704. The data processing device fuses the group feature of the first target frame to obtain the inter-group feature of the first target frame;

融合到一起，得到目标帧的组间特征F_A。该组间特征即为从邻近帧中提取的细节特征，可用于下一步骤中进行目标帧的超分。The data processing device converts the group features of a plurality of frame groups of the same target frame

fused together to obtain the inter _- group feature FA of the target frame. The inter-group feature is the detailed feature extracted from the adjacent frames, which can be used for the super-score of the target frame in the next step.

There are many ways to fuse multiple group features to determine the feature _FA of the target frame, which is not specifically limited here. Optionally, please refer to FIG. 11 , which is a schematic diagram of an embodiment of feature fusion between groups in an embodiment of the present application.

首先使用2D卷积对其进行特征提取得到1维的特征

之后基于

计算注意力掩膜M_n。注意力掩膜M_n可以理解为特征

的权重，计算公式如下：Group features for all frame groups of the first target frame

First use 2D convolution to perform feature extraction on it to obtain 1-dimensional features

based on

Compute the attention mask M _n . The attention mask _Mn can be understood as a feature

The weight of , the calculation formula is as follows:

代表第i组的组特征图；(x，y)_j代表特征图中位置j的像素坐标，i代表第i个帧组，N代表帧组总数。where M _n (x, y) _i represents the attention mask,

Represents the group feature map of the ith group; (x, y) _j represents the pixel coordinates of position j in the feature map, i represents the ith frame group, and N represents the total number of frame groups.

The group features are weighted according to the calculated attention mask, and the features of the weighted target frame are obtained according to the following formula

where ⊙ is the element-wise dot product (Hadamard product).

进行进一步融合，生成融合之后的特征F_A。请参阅图12，为本申请实施例中进行组间特征融合的另一个实施例示意图。Optionally, use a three-dimensional convolution block (3D Block) containing 3D convolution and a two-dimensional convolution block (2D Block) containing 2D convolution to weight the group features

Further fusion is performed to generate the fused feature F _A . Please refer to FIG. 12 , which is a schematic diagram of another embodiment of feature fusion between groups in an embodiment of the present application.

Similarly, the data processing apparatus performs inter-group fusion on the frame group features of each target frame in the frame sequence, and obtains the features of each target frame respectively, and details are not repeated here.

705. The data processing apparatus acquires a high-resolution target frame according to the inter-group feature.

_The feature FA of the target frame can be understood as a multi-channel image, also called a feature map. The data processing module enlarges the feature map to the target resolution, and outputs a residual map, which is a detailed feature. Optionally, the data processing apparatus uses cascaded 2D convolution and pixel reshuffle algorithm (PixelShuffle) to realize the enlargement of the feature map, optionally, each time enlarges by 2 times until the magnification reaches the target resolution.

The data processing device amplifies the low-resolution target frame through up-sampling to obtain an enlarged blurred image, which can generally be considered as: clear image=blurred image+residual image. The blurred image can be directly obtained by interpolation from the input low-resolution target frame, and then combined with the residual map, a clear high-resolution target frame can be obtained.

Exemplarily, please refer to FIG. 13 , which is a schematic diagram of sample amplification of a target frame in an embodiment of the present application.

The data processing device amplifies the fused feature FA through the cascaded 2D convolution and pixel reshuffle algorithm ( _PixelShuffle ), and each time amplifies it by 2 times, until it is enlarged to the target resolution and then outputs a residual image. The data processing device samples the original target frame through a bicubic interpolation (Bicubic) method to obtain a blurred enlarged image obtained after upsampling and amplification, and the data processing device adds the residual image according to the enlarged image to obtain the target frame, that is, the corresponding intermediate frame I4. high-resolution target frame.

Similarly, the data processing device performs up-sampling and amplification according to the characteristics of each target frame in the frame sequence, and obtains high-resolution frames of each target frame respectively, and details are not repeated here.

Please refer to FIG. 14 , which is a schematic diagram of another embodiment of the data processing method provided by the embodiment of the present application;

In the data processing method provided by the embodiment of the present application, after the data processing device acquires the input frame sequence, the frame sequence is subjected to fast frame alignment processing, and then time sequence grouping is performed according to the time interval between the adjacent frame and the target frame, and grouping 1 to group N are grouped. According to the 3D convolutional neural network, the intra-group fusion of features is performed, and then the weight of the intra-group fusion features is obtained according to the attention mechanism, and the inter-group fusion is performed according to the weight and the N intra-group fusion features to obtain the detailed features of the target frame. Finally, Output high-resolution frames by upsampling.

The following describes a data processing apparatus for implementing data processing. Please refer to FIG. 15 , which is a schematic diagram of an embodiment of the data processing apparatus provided by the embodiment of the present application;

The data processing device provided by the embodiment of the present application includes:

an acquiring unit 1501, configured to acquire a frame sequence, where the frames in the frame sequence have a first resolution;

The determining unit 1502 is configured to determine at least two frame groups from the frame sequence, where the frame group includes a first target frame and at least two adjacent frames of the first target frame, and the first target frame is the any frame in the frame sequence, the adjacent frame is a frame other than the first target frame in the frame sequence;

The determining unit 1502 is further configured to determine, by using a three-dimensional convolutional neural network, the characteristics of each frame group in the at least two frame groups, where the characteristics of each frame group indicate that the first target frame is obtained from all the frame groups. The detailed information obtained in the adjacent frames in each frame group, the size of the convolution kernel in the time dimension in the three-dimensional convolutional neural network is positively correlated with the number of frames in the frame group;

The processing unit 1503 is configured to fuse the features of each frame group in the at least two frame groups to determine the detail feature of the first target frame, where the detail feature indicates that based on the first target frame, from the Detailed information obtained from adjacent frames in at least two frame groups;

The obtaining unit 1501 is further configured to obtain, according to the detailed feature and the first target frame, a first target frame with a second resolution, where the second resolution is greater than the first resolution.

Optionally, the frame group includes the first target frame and the two adjacent frames.

Optionally, the two adjacent frames include a first adjacent frame and a second adjacent frame, and the interval between the first adjacent frame and the first target frame in the frame sequence is the same as the interval between the first adjacent frame and the first target frame. A second adjacent frame is equally spaced from the first target frame in the sequence of frames.

Optionally, the at least two frame groups include three frame groups.

Optionally, the determining unit 1502 is further configured to align the frames in each frame group of the at least two frame groups, and determine the aligned at least two frame groups; the determining unit is specifically configured to: through the three-dimensional A convolutional neural network determines features for each of the aligned at least two frame groups.

Optionally, the determining unit 1502 is specifically configured to: determine a homography matrix between all consecutive two frames in a queue composed of frames in the at least two frame groups; determine the alignment according to the homography matrix. of at least two frame groups.

Optionally, the determining unit 1502 is further configured to: the determining unit is further configured to: determine the weight of the feature of each frame group in the at least two frame groups through a deep neural network; the processing unit specifically uses In: fusing the features of each of the at least two frame groups according to the weight to determine the detailed features of the first target frame.

Optionally, the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension is equal to the number of frames in the frame group.

FIG. 16 is a structural diagram of a chip hardware provided by an embodiment of the present application.

The algorithms based on the convolutional neural network shown in Figures 3 and 4 can be implemented in the NPU chip shown in Figure 16.

The neural network processor NPU 50 is mounted on the main CPU (Host CPU) as a co-processor, and the Host CPU assigns tasks. The core part of the NPU is the arithmetic circuit 503, which is controlled by the controller 504 to extract the matrix data in the memory and perform multiplication operations.

In some implementations, the arithmetic circuit 503 includes multiple processing units (process engines, PEs). In some implementations, arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 503 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit fetches the data corresponding to the matrix B from the weight memory 502 and buffers it on each PE in the operation circuit. The arithmetic circuit fetches the matrix A data from the input memory 501 and performs matrix operations on the matrix B, and the obtained partial result or final result of the matrix is stored in the accumulator 508 accumulator.

Unified memory 506 is used to store input data and output data. The weight data is directly transferred to the weight memory 502 through a storage unit access controller 505 (direct memory access controller, DMAC). Input data is also moved to unified memory 506 via the DMAC.

The BIU is the Bus Interface Unit, that is, the bus interface unit 510, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch memory 509 Instruction Fetch Buffer.

The bus interface unit 510 (BIU for short) is used for the instruction fetch memory 509 to obtain instructions from the external memory, and also for the storage unit access controller 505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 506 , the weight data to the weight memory 502 , or the input data to the input memory 501 .

The vector calculation unit 507 may include a plurality of operation processing units, if necessary, further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. Mainly used for non-convolutional/FC layer network calculation in neural network, such as Pooling (pooling), Batch Normalization (batch normalization), LocalResponse Normalization (local response normalization), etc.

In some implementations, the vector computation unit can 507 store the processed output vectors to the unified buffer 506 . For example, the vector calculation unit 507 may apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the vector of processed outputs can be used as activation input to the arithmetic circuit 503, eg, for use in subsequent layers in a neural network.

an instruction fetch buffer 509 connected to the controller 504 for storing instructions used by the controller 504;

The unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all On-Chip memories. External memory is private to the NPU hardware architecture.

The operations of each layer in the convolutional neural network shown in FIG. 3 and FIG. 4 may be performed by the matrix computing unit 212 or the vector computing unit 507 .

The foregoing method embodiments of the present application may be applied to a processor, or the processor may implement the steps of the foregoing method embodiments. A processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiments may be completed by a hardware integrated logic circuit in a processor or an instruction in the form of software. The above processor may be a central processing unit (CPU), a network processor (NP) or a combination of CPU and NP, a digital signal processor (DSP), an application specific integrated circuit (application specific integrated circuit) integrated circuit, ASIC), off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps and logic block diagrams disclosed in this application can be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps in combination with the method disclosed in this application can be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware. Although only one processor is shown, the apparatus may include multiple processors or processors include multiple processing units. Specifically, the processor may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor.

Memory is used to store computer instructions for execution by the processor. The memory may be a storage circuit or a memory. The memory can be volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically programmable Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory may be random access memory (RAM), which acts as an external cache. The memory may be independent of the processor, or may be a storage unit in the processor, which is not limited herein. Although only one memory is shown in the figures, the apparatus may include multiple memories or the memories include multiple storage units.

The transceiver is used to realize the content interaction between the processor and other units or network elements. Specifically, the transceiver may be a communication interface of the device, a transceiver circuit or a communication unit, or a transceiver. The transceiver may also be a communication interface or a transceiver circuit of the processor. In a possible implementation manner, the transceiver may be a transceiver chip. The transceiver may also include a transmitting unit and/or a receiving unit. In one possible implementation, the transceiver may include at least one communication interface. In another possible implementation, the transceiver may also be a unit implemented in the form of software. In various embodiments of the present application, the processor may interact with other units or network elements through the transceiver. For example, the processor obtains or receives content from other network elements through the transceiver. If the processor and the transceiver are two physically separate components, the processor can interact with other units of the device without going through the transceiver.

In one possible implementation, the processor, the memory, and the transceiver may be connected to each other through a bus. The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or illustrations. Any embodiment or design described in the embodiments of the present application as "exemplary" or "such as" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present the related concepts in a specific manner.

In the various embodiments of the present application, various illustrations are provided for the sake of understanding. However, these examples are merely examples and are not meant to be the best way to implement the present application.

The above embodiments may be implemented in whole or in part by software, hardware, firmware or any combination thereof, and when implemented in software, may be implemented in whole or in part in the form of computer program products.

The computer program product includes one or more computer instructions. When the computer-executed instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a server, data center, etc., which includes one or more available media integrated. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), among others.

The technical solutions provided by the present application have been introduced in detail above, and the principles and implementations of the present application have been described with specific examples in the present application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application. At the same time, for those skilled in the art, according to the idea of the application, there will be changes in the specific implementation and application scope. To sum up, the content of this specification should not be construed as a limitation to the application.

Claims (19)

1. a data processing method, is characterized in that, described method comprises: The data processing device acquires a sequence of frames, the frames in the sequence of frames have a first resolution; The data processing device determines at least two frame groups from the frame sequence, the frame groups including a first target frame and at least two adjacent frames of the first target frame, the first target frame being the any frame in the frame sequence, the adjacent frame is a frame other than the first target frame in the frame sequence; The data processing device determines the characteristics of each frame group in the at least two frame groups through a three-dimensional convolutional neural network, and the characteristic indication of each frame group is based on the first target frame, from the each frame. Detailed information obtained from adjacent frames in the group, the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension is positively correlated with the number of frames in the frame group; The data processing device fuses the features of each of the at least two frame groups to determine a detail feature of the first target frame, the detail feature indicating that based on the first target frame, from the at least Detail information obtained from adjacent frames within two frame groups; The data processing apparatus acquires, according to the detailed feature and the first target frame, a first target frame with a second resolution, where the second resolution is greater than the first resolution. 2. The method of claim 1, wherein the frame group includes the first target frame and two of the adjacent frames. 3. The method of claim 2, wherein the two adjacent frames include a first adjacent frame and a second adjacent frame, the first adjacent frame being the same as the first adjacent frame in the frame sequence. The interval between a target frame is equal to the interval between the second adjacent frame and the first target frame in the frame sequence. 4. The method of any one of claims 1 to 3, wherein the at least two frame groups comprise three frame groups. 5. The method according to claim 1, wherein the method further comprises: The data processing device aligns the frames in each of the at least two frame groups, and determines the aligned at least two frame groups; The characteristics of each frame group in the at least two frame groups determined by the data processing device through a three-dimensional convolutional neural network include: The data processing device determines the feature of each frame group in the aligned at least two frame groups through a three-dimensional convolutional neural network. 6. The method according to claim 5, wherein the data processing device aligns the frames in each of the at least two frame groups, and determining the aligned at least two frame groups comprises: The data processing device determines the homography matrix between all two consecutive frames in the queue formed by the frames in the at least two frame groups; The data processing apparatus determines the aligned at least two frame groups according to the homography matrix. 7. The method according to any one of claims 1 to 6, wherein the method further comprises: The data processing device determines the weight of the feature of each frame group in the at least two frame groups through a deep neural network; The data processing device fuses the features of each frame group in the at least two frame groups to determine the detailed features of the first target frame including: The data processing device fuses the features of each frame group in the at least two frame groups according to the weight value to determine the detailed features of the first target frame. 8. The method according to any one of claims 1 to 7, wherein the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension is equal to the number of frames in the frame group. 9. A data processing device, comprising: an acquisition unit, configured to acquire a frame sequence, the frames in the frame sequence have a first resolution; a determining unit, configured to determine at least two frame groups from the frame sequence, the frame group including a first target frame and at least two adjacent frames of the first target frame, and the first target frame is the any frame in the frame sequence, the adjacent frame is a frame other than the first target frame in the frame sequence; The determining unit is further configured to determine, by using a three-dimensional convolutional neural network, a feature of each frame group in the at least two frame groups, where the feature indication of each frame group is based on the first target frame and obtained from the Detailed information obtained from adjacent frames in each frame group, the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension is positively correlated with the number of frames in the frame group; A processing unit, configured to fuse the features of each frame group in the at least two frame groups to determine the detail feature of the first target frame, the detail feature indicating that based on the first target frame, from the at least Detail information obtained from adjacent frames within two frame groups; The obtaining unit is further configured to obtain, according to the detailed feature and the first target frame, a first target frame with a second resolution, where the second resolution is greater than the first resolution. 10. The apparatus of claim 9, wherein the frame group includes the first target frame and two of the adjacent frames. 11. The apparatus of claim 10, wherein the two adjacent frames comprise a first adjacent frame and a second adjacent frame, the first adjacent frame being the same as the first adjacent frame in the frame sequence. The interval between a target frame is equal to the interval between the second adjacent frame and the first target frame in the frame sequence. 12. The device according to any one of claims 9 to 11, wherein The at least two frame groups include three frame groups. 13. The apparatus according to claim 9, wherein the determining unit, is also used to align the frames in each of the at least two frame groups, and determine the aligned at least two frame groups; The determining unit is specifically used for: A feature of each of the aligned at least two frame groups is determined by a three-dimensional convolutional neural network. 14. The apparatus according to claim 13, wherein the determining unit is specifically configured to: determining a homography matrix between all two consecutive frames in the queue consisting of frames in the at least two frame groups; The aligned at least two frame groups are determined from the homography matrix. 15. The device according to any one of claims 9 to 14, wherein The determining unit is also used for: Determine the weight of the feature of each frame group in the at least two frame groups by a deep neural network; The processing unit is specifically used for: The features of each of the at least two frame groups are fused according to the weights to determine the detailed features of the first target frame. The apparatus according to any one of claims 9 to 15, wherein the size of the convolution kernel in the three-dimensional convolutional neural network in the time dimension is equal to the number of frames in the frame group. 17. A data processing device, comprising a processor and a memory, wherein the processor and the memory are connected to each other, wherein the memory is used to store a computer program, the computer program includes program instructions, the The processor is configured to invoke the program instructions to execute the method as claimed in any one of claims 1 to 8. 18. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 8. 19. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 8.

Images