CN113191489B - Training method of binary neural network model, image processing method and device - Google Patents

Abstract

The application relates to an image processing technology in the field of computer vision in the field of artificial intelligence, and discloses a binary neural network model training method, an image processing method and an image processing device. The training method comprises the following steps: s1: determining a knowledge distillation framework; the teacher network is a trained neural network model, and the student network is an initial binary neural network model M ₀ (ii) a S2: training a binary neural network model M by using a j +1 th batch of images and a target loss function _j To obtain a binary neural network model M _j+1 (ii) a The target loss function comprises an angle loss item, and the angle loss item is used for describing the difference between an included angle between the characteristic matrix and the weight matrix in the teacher network and an included angle between the characteristic matrix and the weight matrix in the student network; s3: when a preset condition is met, the binary neural network model M is used _j+1 As a target binary neural network model; otherwise let j = j +1 and repeat step S2. According to the application embodiment, the prediction accuracy of the binary neural network model can be improved.

Description

Training method of binary neural network model, image processing method and device

Technical Field

The present application relates to the field of artificial intelligence technology, and in particular to a training method, an image processing method and a device for a binary neural network model.

Background Art

Computer vision is an integral part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, and medical diagnosis. It is a discipline about how to use cameras/camcorders and computers to obtain the data and information we need about the objects being photographed. Figuratively speaking, it is to equip computers with eyes (cameras/camcorders) and brains (algorithms) to replace human eyes in identifying, tracking, and measuring targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

Image classification (IC), object detection (OD) and image segmentation (IS) are important problems in high-level visual semantic understanding tasks. With the rapid development of artificial intelligence technology, the above three basic tasks are increasingly widely used in the field of computer vision. Deep convolutional neural networks have played an increasingly important role in the above three basic tasks, especially in object detection. However, deep convolutional neural network models usually have millions of parameters and require billions of floating point operations (FLOPs) to calculate, which limits their deployment on resource-limited platforms. In order to achieve efficient online inference on embedded devices, deep convolutional neural network models are usually quantized to obtain binary neural network models to perform the above basic tasks in computer vision; this is mainly because the storage space occupied by the parameters in the binary neural network model is much smaller than the storage space occupied by the parameters of the deep convolutional neural network model before quantization.

However, the binary neural network model in the prior art has a much lower prediction accuracy than the neural network model.

Summary of the invention

The embodiments of the present application provide a training method, an image processing method and a device for a binary neural network model, which can effectively improve the prediction accuracy of the trained binary neural network model.

In a first aspect, the present application provides a method for training a binary neural network model, the method comprising: S1: determining a knowledge distillation framework; wherein the teacher network in the knowledge distillation framework is a trained neural network model, the student network in the knowledge distillation framework is an initial binary neural network model M ₀ , the teacher network and the student network respectively include N layers of neural networks, N is a positive integer; S2: using the j+1th batch of images and the target loss function to train the binary neural network model M _j to obtain a binary neural network model M _j+1 ; wherein the binary neural network model M _j is obtained based on the training of the j-th batch of images, and j is a positive integer; the target loss function includes an angle loss term, which is used to describe the difference between a first angle corresponding to the i-th layer of the neural network in the teacher network and a second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix of the j+1-th batch of images in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix of the j+1-th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N; S3: when the preset conditions are met, the binary neural network model M _j+1 is used as the target binary neural network model; otherwise, let j=j+1 and repeat step S2.

It should be understood that in each training process, the adjustment of the model parameters in the student network is carried out in the direction of minimizing the value of the target loss function. That is, when the angle loss term in the target loss function is smaller, the performance difference between the student network and the teacher network is smaller.

It can be seen that in the embodiment of the present application, the teacher network trained in the knowledge distillation framework is used to guide the training process of the student network, and the angle loss term is designed in the target loss function to update the parameters in the student network. On the one hand, the feature extraction results of the student network for the input samples can be made close to the feature extraction results of the teacher network for the input samples. On the other hand, the angle between the binary weight matrix and the binary input matrix in the student network is made close to the angle between the weight matrix and the input matrix in the teacher network. In summary, compared with the prior art in which the training process of the binary neural network model does not consider the quantized angle loss, the present application can make the performance of the trained student network as close to the performance of the teacher network as possible by introducing the knowledge distillation framework and the angle loss term in the loss function, thereby improving the prediction accuracy of the target binary neural network model trained in the embodiment of the present application.

In a feasible implementation, the above-mentioned target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; the first convolution output result is based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network in the teacher network for the j+1-th batch of images; the second convolution output result is based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, as well as the binary input matrix of the j+1-th batch of images in the i-th layer neural network in the student network.

It should be understood that the smaller the convolution result loss term in the target loss function, the smaller the loss value of the target loss function, which means that the performance of the student network and the teacher network are closer.

It can be seen that in the embodiment of the present application, the convolution result loss term is introduced into the target loss function to make the second convolution output result in the student network as close as possible to the first convolution output result in the teacher network, that is, the output result of each layer of the neural network in the student network is made as close as possible to the output result of each layer of the neural network in the teacher network, thereby ensuring that the predicted value of the output of the student network is close to the predicted value of the teacher network, thereby improving the prediction accuracy of the target binary neural network model obtained after training.

In a feasible implementation, the above-mentioned objective loss function also includes a weight loss term; wherein the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network.

It can be seen that the embodiment of the present application can also introduce a weight loss term that characterizes the difference between the weight matrix in the teacher network and the binary weight matrix in the student network into the target loss function, and train the student network together with the convolution result loss term and the angle loss term. By introducing the above three model performance measurement indicators into the target loss function to train the student network, the performance of the target neural network model obtained after training can be maximized, making it as close to the performance of the teacher network as possible.

In a feasible implementation, the above-mentioned training of the binary neural network model M _j using the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 includes: inputting the j+1th batch of images into the binary neural network model M _j to obtain prediction values of the j+1th batch of images; updating the parameters of each layer of the neural network in the binary neural network model M _j based on the prediction values of the j+1th batch of images, the labels of the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 .

It should be understood that after the forward propagation process of a training is completed, the embodiment of the present application can obtain an image prediction value based on the end of the forward propagation process; then calculate the loss function value in the training process based on the image prediction value, the image label, and the above-mentioned target loss function, and update the model parameters based on the target loss function value.

It can be seen that in the embodiment of the present application, in each training process of the binary neural network model, the parameters in the student network can be updated layer by layer through the predicted value of the image used in each training, the label of the image used, and the angle loss term, convolution result loss term and weight loss term contained in the above-mentioned target loss function. In summary, the embodiment of the present application compares the difference between the student network prediction value and the image label on the one hand, and compares the difference between the corresponding model parameters of the student network and the teacher network (corresponding to the weight loss term in the target loss function) and the intermediate calculation results of the model (corresponding to the angle loss term and convolution result loss term in the target loss function) on the other hand, so that the trained target binary neural network model can be closer to the performance of the teacher network in feature extraction and prediction value, that is, the model prediction accuracy is improved.

In a feasible implementation, the j+1th batch of images is input into the binary neural network model _Mj to obtain the predicted values of the j+1th batch of images, including: P1: obtaining the binary weight matrix of the i-th layer of the neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer of the neural network in the binary neural network model _Mj ; P2: obtaining the second convolution output result of the i-th layer of the neural network based on the binary input matrix and binary weight matrix of the j+1th batch of images in the i-th layer of the neural network; wherein, the element at any position in the probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P3: let i=i+1, and repeat steps P1-P2, and obtain the predicted values of the j+1th batch of images based on the second convolution output result of the N-th layer of the neural network.

It can be seen that in the embodiment of the present application, during the forward propagation process of each layer of the neural network during each training, the binary weight matrix of each layer of the neural network is first determined based on the reference weight matrix and probability matrix corresponding to each layer of the neural network, and then the second convolution output result of each layer of the neural network is calculated based on the binary input matrix and the binary weight matrix of each layer of the neural network, and then when forward propagating to the Nth layer of the neural network, the image prediction value of the model output during this training process can be calculated based on the second convolution output result of the Nth layer of the neural network. Then, in the subsequent back propagation process, the second convolution output result and the first convolution output result obtained during the forward propagation process, as well as the image prediction value and the image label can be used to calculate the loss function value of this training process, and then the model parameters are adjusted according to the loss function value to ensure that the optimal target binary neural network model is obtained.

In a feasible implementation, the reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; the binary weight matrix of the i-th layer neural network is obtained based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network _model Mj, including: determining the element at any position in the target binary weight matrix based on the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the second probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix; wherein the element at any position in the first probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix.

It can be seen that in the embodiment of the present application, the reference weight matrix in each layer of the neural network in the student network includes a first reference weight matrix and a second reference weight matrix; and the probability matrix in each layer of the neural network includes a first probability matrix and a second probability matrix, the first probability matrix corresponds to the first reference weight matrix, and the second probability matrix corresponds to the second reference weight matrix. Based on the probability values of the elements at the same position in the first reference weight matrix and the second reference weight matrix, the elements at that position in the first reference weight matrix or the second reference weight matrix are selected as the elements at that position in the binary weight matrix; based on this rule, the binary weight matrix corresponding to each layer of the neural network in the word training process is obtained; then, based on the binary weight matrix, the second convolution output result of each layer of the neural network corresponding to the above embodiment and the target loss function value in each training process are calculated, thereby ensuring the correct conduct of the training process, and then obtaining the optimal binary neural network model.

In a feasible implementation, the above-mentioned second convolution output result of the i-th layer neural network is obtained based on the binary input matrix and binary weight matrix of the j+1-th batch of images in the i-th layer neural network, including: performing convolution operations on the binary input matrix and binary weight matrix of each image in the j+1-th batch of images in the i-th layer neural network respectively to obtain a reference feature matrix of each image; and scaling the reference feature matrix of each image using the weight scaling factor of the i-th layer neural network to obtain a second convolution output result.

It can be seen that in the embodiment of the present application, the reference feature matrix of each image is obtained based on the binary input matrix and the binary weight matrix in each layer of the neural network; then the reference feature matrix of each image is scaled using the weight scaling factor of each layer of the neural network to obtain the second convolution output result. The second convolution output result characterizes the characteristics of the input image, and the design of the convolution result loss term in the target loss function can make the trained target binary neural network model retain the feature extraction ability of the teacher network as much as possible, thereby improving the prediction accuracy of the target binary neural network model.

In a feasible implementation, the above parameters include at least one of a probability matrix or a weight scaling factor.

It can be seen that in the embodiment of the present application, each training back propagation process will update at least one of the probability matrix or weight scaling factor in each layer of the neural network, so that during the forward propagation process of the next training, the binary weight matrix in each layer of the neural network is updated, and then based on the updated binary weight matrix and/or weight scaling factor, the image prediction value and loss function value output by the training model are obtained, and based on the obtained loss function value, the model parameters are further adjusted to ensure that a student network with performance close to that of the teacher network is obtained.

In the second aspect, the present application provides a model training method, wherein the above-mentioned model includes a teacher network and a student network, the teacher network is a trained neural network model, the student network is a binary neural network model, the teacher network and the student network respectively include N layers of neural networks, N is a positive integer, and the method includes: using the teacher network and the target loss function to train the binary neural network model; wherein the target loss function includes an angle loss term, and the angle loss term is used to describe the difference between a first angle corresponding to the i-th layer of the neural network in the teacher network and a second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N; repeat the above steps until the iteration termination condition is met to obtain the target binary neural network model.

In a feasible implementation, the target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; the first convolution output result is based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network in the teacher network; the second convolution output result is based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, as well as the binary input matrix in the i-th layer neural network in the student network.

In a feasible implementation, the objective loss function also includes a weight loss term; wherein the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network.

In a feasible implementation, the above-mentioned training of the binary neural network model using the teacher network and the target loss function includes: inputting the training image into the binary neural network model to obtain the predicted value of the training image; and updating the parameters in the binary neural network model based on the predicted value of the training image, the label of the training image and the target loss function.

In a feasible implementation, the above-mentioned input of the training image into the binary neural network model to obtain the predicted value of the training image includes: P1: obtaining the binary weight matrix of the i-th layer neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model; wherein, the element at any position in the probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P2: obtaining the second convolution output result of the i-th layer neural network according to the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network;

P3: Let i=i+1 and repeat steps P1-P2 to obtain the predicted value of the training image based on the second convolution output result of the Nth layer neural network.

In a feasible implementation, the reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; the binary weight matrix of the i-th layer neural network is obtained based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model, including: determining the element at any position in the target binary weight matrix based on the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the second probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix; wherein the element at any position in the first probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix.

In a feasible implementation, the above-mentioned second convolution output result of the i-th layer neural network is obtained according to the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network, including: performing a convolution operation on the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network to obtain a reference feature matrix of the training image; scaling the reference feature matrix of the training image using the weight scaling factor of the i-th layer neural network to obtain a second convolution output result.

In one possible implementation, the parameter includes at least one of a probability matrix or a weight scaling factor.

It should be understood that the beneficial effects of each embodiment in the second aspect can be referred to the description of the corresponding embodiment in the first aspect, and will not be repeated here.

In a third aspect, the present application provides an image processing method, the method comprising: obtaining an image to be processed; performing image processing on the image to be processed using a target binary neural network model to obtain a predicted value of the image to be processed; wherein the target binary neural network model is obtained through K trainings, and in the j+1th training of the K trainings: using the j+1th batch of images and the target loss function to train the binary neural network model M _j to obtain the binary neural network model M _j+1 ; the binary neural network model M _j is the student network in the knowledge distillation framework; the teacher network in the knowledge distillation framework is a trained neural network model, and the teacher network and the student network respectively contain N layers of neural networks, where N is a positive integer; the target loss function contains an angle loss term; K is a positive integer, and j is an integer greater than or equal to zero and less than or equal to K; the angle loss term is used to describe the difference between the first angle corresponding to the i-th layer of the neural network in the teacher network and the second angle corresponding to the i-th layer of the neural network in the student network; the first angle is based on the weight matrix corresponding to the i-th layer of the neural network in the teacher network and the input matrix of the j+1th batch of images in the i-th layer of the neural network in the teacher network; the second angle is based on the binary weight matrix corresponding to the i-th layer of the neural network in the student network and the binary input matrix of the j+1th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N.

It can be seen that in the embodiments of the present application, since the method in the first aspect introduces a knowledge distillation framework during training and introduces a corresponding angle loss term in the target loss function, the target binary neural network model trained by the method in the first aspect has a greatly improved model accuracy compared to the existing binary neural network model; at the same time, since the binary neural network model parameters occupy less storage space than the teacher network and are more lightweight, it is more suitable for use on embedded devices and has a broader application prospect.

In a feasible implementation, the above-mentioned image processing includes at least one of image classification, target detection or image segmentation.

It can be seen that the method in the embodiment of the present application can be used for any task of image classification, target detection and image segmentation. By applying the image processing method in the embodiment of the present application to the above three tasks, the image processing effect can be improved, that is, the versatility of this model is good.

In a fourth aspect, the present application provides an image processing method, the method comprising: obtaining an image to be processed; performing image processing on the image to be processed using a target binary neural network model to obtain a predicted value of the image to be processed; wherein the target binary neural network model is obtained by training an initial binary neural network model _M0 in a knowledge distillation framework through a target loss function, the initial binary neural network model _M0 is a student network in the knowledge distillation framework, and the teacher network in the knowledge distillation framework is a trained neural network model; the target loss function includes an angle loss term, and the angle loss term is used to describe the difference between the angle between the feature matrix and the weight matrix in the teacher network and the angle between the feature matrix and the weight matrix in the student network.

It can be seen that in the embodiments of the present application, since the method in the first aspect introduces a knowledge distillation framework during training and introduces a corresponding angle loss term in the target loss function, the target binary neural network model trained by the method in the first aspect has a greatly improved model accuracy compared to the existing binary neural network model; at the same time, since the binary neural network model parameters occupy less storage space and are more lightweight than the neural network model in the teacher network, it has good application prospects in embedded devices.

In a fifth aspect, the present application provides a training device for a binary neural network model, the device comprising: a determination unit, configured to execute step S1. A training unit, configured to execute step S2. A decision unit, configured to execute step S3. Step S1: Determine a knowledge distillation framework; wherein the teacher network in the knowledge distillation framework is a trained neural network model, the student network in the knowledge distillation framework is an initial binary neural network model M ₀ , the teacher network and the student network each contain N layers of neural networks, and N is a positive integer. Step S2: Use the j+1th batch of images and the target loss function to train the binary neural network model M _j to obtain the binary neural network model M _j+1 ; wherein the binary neural network model M _j is obtained based on the jth batch of images, and j is a positive integer; the target loss function includes an angle loss term, which is used to describe the difference between the first angle corresponding to the i-th layer of the neural network in the teacher network and the second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix of the j+1th batch of images in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix of the j+1th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N. Step S3: When the preset conditions are met, the binary neural network model M _j+1 is used as the target binary neural network model; otherwise, let j=j+1 and repeat step S2.

In a feasible implementation, the above-mentioned target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; the first convolution output result is based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network in the teacher network for the j+1-th batch of images; the second convolution output result is based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, as well as the binary input matrix of the j+1-th batch of images in the i-th layer neural network in the student network.

In a feasible implementation, the above-mentioned objective loss function also includes a weight loss term; wherein the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network.

In a feasible implementation, the training unit is specifically used to: input the j+1th batch of images into the binary neural network model M _j to obtain the predicted values of the j+1th batch of images; update the parameters of each layer of the binary neural network model M _j based on the predicted values of the j+1th batch of images, the labels of the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 .

In a feasible implementation, in the aspect of inputting the j+1th batch of images into the binary neural network model _Mj to obtain the predicted values of the j+1th batch of images, the training unit is specifically used for: P1: obtaining the binary weight matrix of the i-th layer of the neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer of the neural network in the binary neural network model _Mj ; P2: obtaining the second convolution output result of the i-th layer of the neural network based on the binary input matrix and binary weight matrix of the j+1th batch of images in the i-th layer of the neural network; wherein, the element at any position in the probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P3: let i=i+1, repeat steps P1-P2, and obtain the predicted values of the j+1th batch of images based on the second convolution output result of the N-th layer of the neural network.

In a feasible implementation, the above-mentioned reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; in terms of obtaining the binary weight matrix of _{the i-th layer neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in} the binary neural network model M j, the training unit is specifically used to: determine the element at any position in the target binary weight matrix based on the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the second probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix; wherein, the element at any position in the first probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix.

In a feasible implementation, in the aspect of obtaining the second convolution output result of the i-th layer neural network based on the binary input matrix and the binary weight matrix of the j+1-th batch of images in the i-th layer neural network, the training unit is specifically used to: perform convolution operations based on the binary input matrix and the binary weight matrix of each image in the j+1-th batch of images in the i-th layer neural network, respectively, to obtain a reference feature matrix for each image; and scale the reference feature matrix of each image using the weight scaling factor of the i-th layer neural network to obtain the second convolution output result.

In a feasible implementation, the above parameters include at least one of a probability matrix or a weight scaling factor.

In a sixth aspect, the present application provides a model training device, which includes a teacher network and a student network, the teacher network is a trained neural network model, the student network is a binary neural network model, the teacher network and the student network respectively include N layers of neural networks, N is a positive integer, and the device includes: a training unit, which is used to train the binary neural network model using the teacher network and the target loss function; wherein the target loss function includes an angle loss term, and the angle loss term is used to describe the difference between a first angle corresponding to the i-th layer of the neural network in the teacher network and a second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N; a decision unit, which is used to repeatedly execute the above steps until the iteration termination condition is met to obtain the target binary neural network model.

In a feasible implementation, the target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; the first convolution output result is based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network in the teacher network; the second convolution output result is based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, as well as the binary input matrix in the i-th layer neural network in the student network.

In a feasible implementation, the objective loss function also includes a weight loss term; wherein the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network.

In a feasible implementation, in the above-mentioned aspect of training the binary neural network model using the teacher network and the target loss function, the training unit is specifically used to: input the training image into the binary neural network model to obtain the predicted value of the training image; and update the parameters in the binary neural network model based on the predicted value of the training image, the label of the training image and the target loss function.

In a feasible implementation, in the aspect of inputting the training image into the binary neural network model to obtain the predicted value of the training image, the training unit is specifically used for: P1: obtaining the binary weight matrix of the i-th layer neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model; wherein, the element at any position in the probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P2: obtaining the second convolution output result of the i-th layer neural network according to the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network; P3: setting i=i+1, and repeating steps P1-P2, and obtaining the predicted value of the training image based on the second convolution output result of the N-th layer neural network.

In a feasible implementation, the reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; in terms of obtaining the binary weight matrix of the i-th layer neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model, the training unit is specifically used to: determine the element at any position in the target binary weight matrix based on the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the second probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix; wherein the element at any position in the first probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix.

In a feasible implementation, in the aspect of obtaining the second convolution output result of the i-th layer neural network based on the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network, the training unit is specifically used to: perform a convolution operation on the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network to obtain a reference feature matrix of the training image; and scale the reference feature matrix of the training image using the weight scaling factor of the i-th layer neural network to obtain the second convolution output result.

In a feasible implementation, the above parameters include at least one of a probability matrix or a weight scaling factor.

In a seventh aspect, the present application provides an image processing device, the device comprising: an acquisition unit, for acquiring an image to be processed; a processing unit, for performing image processing on the image to be processed using a target binary neural network model to obtain a predicted value of the image to be processed; wherein the target binary neural network model is obtained through K trainings, and in the j+1th training of the K trainings: the binary neural network model M _j is trained using the j+1th batch of images and the target loss function to obtain a binary neural network model M _j+1 ; the binary neural network model M _j is the student network in the knowledge distillation framework; the teacher network in the knowledge distillation framework is a trained neural network model, and the teacher network and the student network respectively contain N layers of neural networks, where N is a positive integer; the target loss function contains an angle loss term; K is a positive integer, and j is an integer greater than or equal to zero and less than or equal to K; the angle loss term is used to describe the difference between the first angle corresponding to the i-th layer of the neural network in the teacher network and the second angle corresponding to the i-th layer of the neural network in the student network; the first angle is based on the weight matrix corresponding to the i-th layer of the neural network in the teacher network and the input matrix of the j+1th batch of images in the i-th layer of the neural network in the teacher network; the second angle is based on the binary weight matrix corresponding to the i-th layer of the neural network in the student network and the binary input matrix of the j+1th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N.

In a feasible implementation, the above-mentioned image processing includes at least one of image classification, target detection or image segmentation.

In an eighth aspect, the present application provides an image processing device, which includes: an acquisition unit for acquiring an image to be processed; a processing unit for performing image processing on the image to be processed using a target binary neural network model to obtain a predicted value of the image to be processed; wherein the target binary neural network model is obtained by training an initial binary neural network model _M0 in a knowledge distillation framework through a target loss function, the initial binary neural network model _M0 is a student network in the knowledge distillation framework, and the teacher network in the knowledge distillation framework is a trained neural network model; the target loss function includes an angle loss term, and the angle loss term is used to describe the difference between the angle between the feature matrix and the weight matrix in the teacher network and the angle between the feature matrix and the weight matrix in the student network.

In a ninth aspect, the present application provides a model training device, comprising a processor and a memory, the memory being used to store program instructions, and the processor being used to call the program instructions to execute any of the methods in the first aspect or the second aspect.

In the tenth aspect, the present application provides a chip system, which includes a processor and a memory; wherein the memory is used to store a target binary neural network model and program instructions; the target binary neural network model is trained based on the method described in any one of the first aspect or the second aspect; the processor is used to read the program instructions to call the target binary neural network model to execute the method described in any one of the third aspect or the fourth aspect.

The above chip can specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

In the eleventh aspect, the present application provides a terminal device, which includes the chip system in the tenth aspect, and a discrete device coupled to the chip system; wherein the terminal device includes a car, a camera, a computer, a mobile phone or a wearable device.

In a twelfth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes a method for executing any one of the above-mentioned first aspect, second aspect, third aspect or fourth aspect.

In a thirteenth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods in the first, second, third or fourth aspects above.

BRIEF DESCRIPTION OF THE DRAWINGS

The following is an introduction to the drawings used in the embodiments of the present application.

FIG1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

FIG2 is a schematic diagram of the structure of a backbone network provided in an embodiment of the present application;

FIG3 is a schematic diagram of a chip hardware structure provided in an embodiment of the present application;

FIG4 is a schematic diagram of another system architecture structure provided in an embodiment of the present application;

FIG5 is a schematic diagram of a convolution operation process in a binary neural network provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of a knowledge distillation framework provided in an embodiment of the present application;

FIG7 is a flow chart of a training method for a binary neural network model provided in an embodiment of the present application;

FIG8 is a flow chart of another model training method provided in an embodiment of the present application;

9-A to 9-C are schematic diagrams of distribution of extracted features of different network models provided in embodiments of the present application;

FIG10 is a schematic diagram of the angle between a feature matrix and a weight matrix in an embodiment of the application;

FIG11 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

FIG12 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

FIG13 is a schematic diagram of a binary neural network model training device provided in an embodiment of the present application;

FIG14 is a schematic diagram of a model training device provided in an embodiment of the present application;

FIG15 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

FIG16 is a schematic diagram of the hardware structure of a model training device in an embodiment of the present application;

FIG. 17 is a schematic diagram of the hardware structure of the image processing device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

The embodiments of the present application can be applied to basic processing tasks in computer vision such as image classification, image segmentation and target detection, such as picture detection, album management, video recording, safe city, human-computer interaction and other scenarios that require image processing.

It should be understood that the image in the embodiments of the present application may be a static image (or a static picture) or a dynamic image (or a dynamic picture). For example, the image in the present application may be a video or a dynamic picture, or the image in the present application may be a static picture or a photo. For ease of description, the present application will uniformly refer to static images or dynamic images as images in the following embodiments.

The method of the embodiment of the present application can be specifically applied to album management and target detection scenarios. These two scenarios are introduced in detail below.

Album management:

A large number of images may be stored in the photo album of a user's terminal device, such as a mobile phone, for example, a large number of images obtained by taking photos with a camera, taking screenshots, or downloading from the Internet. When a user needs to find the image he needs from a large amount of image data, the method in the embodiment of the present application can be used to classify a large number of images in the photo album. Different types of images are stored in different directories, for example, animals, landscapes, and people, etc. Among them, the animal category can also be subdivided into different subcategories, for example, the animal in the image is identified according to the specific animal category in the image, and classified into the subcategory to which the image belongs.

It can be seen that the method of the present application can quickly and accurately help users locate the category to which the image they want to find belongs, thereby saving user time and improving user experience.

Object Detection:

Object detection is to find the object of interest in the image and determine the position and size of the object. For example, if a user wants to find some images containing cats in the photo album of his terminal device, the method in the embodiment of the present application can be used to identify all images containing cats in the user terminal device for the user to select.

It can be seen that the method in the embodiment of the present application can accurately detect the target in the image, thereby screening the images containing the objects of interest to the user and improving the user experience.

It should be understood that the album management and target detection described above are only two specific scenarios in which the method of the embodiment of the present application is applied. The method of the embodiment of the present application is not limited to the above two scenarios when applied. The method of the embodiment of the present application can be applied to any scenario that requires image processing, such as image segmentation. Alternatively, the method of the embodiment of the present application can also be similarly applied to other fields, such as speech recognition and natural language processing, etc., which is not limited in the embodiment of the present application.

The following describes the method provided by this application from the model training side and the model application side:

The training method of the binary neural network model provided in the embodiment of the present application relates to the processing of computer vision, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, and the training data (such as the image to be processed in the present application) is symbolized and formalized for intelligent information modeling, extraction, preprocessing, training, etc., and finally a trained target binary neural network model is obtained; and the image processing method provided in the embodiment of the present application can use the above-mentioned trained target binary neural network model to input the input data (such as the image to be processed in the present application) into the trained target binary neural network model to obtain output data (such as the predicted value of the image to be processed in the present application). It should be noted that the training method of the binary neural network model and the image processing method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

The embodiments of the present application involve a large number of neural network-related applications. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts in the field of neural networks and computer vision that may be involved in the embodiments of the present application are first introduced below.

(1) Image classification

Determine what category of objects are contained in the image or video to be processed.

(2) Object Detection

Identify all the targets (objects) of interest from a given image to be processed and determine their categories and locations. Since various objects have different appearances, shapes, and postures, and are interfered by factors such as illumination and occlusion during imaging, target detection is one of the core and most challenging problems in the field of computer vision.

(3) Image segmentation

Image segmentation is divided into instance segmentation and scene segmentation. Image segmentation is mainly used to determine which target or object each pixel in the image to be processed belongs to.

(4) Neural Network

A neural network can be composed of neural units. A neural unit can refer to an operation unit with x _s and intercept 1 as input. The output of the operation unit can be:

Where s=1, 2, ...n, n is a natural number greater than 1, _Ws is the weight of _xs , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

(5) Deep Neural Networks

其中，

是输入向量，

是输出向量，

是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量

经过如此简单的操作得到输出向量

由于DNN层数多，则系数W和偏移向量

的数量也就很多了。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为

上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。总结就是：第L-1层的第k个神经元到第L层的第j个神经元的系数定义为

需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression:

in,

is the input vector,

is the output vector,

is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector

After such a simple operation, the output vector

Since DNN has many layers, the coefficient W and the offset vector

The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as

The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

(6) Convolutional Neural Networks

Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). Convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

(7) Loss function

In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

(8) Back propagation algorithm

Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

(9) Pixel value

The pixel value of an image can be a red, green, and blue (RGB) color value, and the pixel value can be a long integer representing the color. For example, the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For grayscale images, the pixel value can be a grayscale value.

(10) Entropy

It can represent the certainty of things. The higher the certainty, the lower the entropy, and vice versa. For classification tasks, if the confidence of the classification result of an image is closer to 0 or 1, its entropy is lower, and the closer the classification result is to 0.5, the higher the entropy, which means the classification result is uncertain.

(11) Knowledge Distillation

Knowledge distillation is a common method of model compression. Model compression refers to distilling the feature representation "knowledge" learned by the complex and powerful teacher network in the teacher-student framework and passing it to the student network with small parameters and weak learning ability.

(12) Binary Neural Network

A binary neural network refers to a neural network that uses only two values, 1 and -1, to represent the neural network parameters (weights) and the convolution operation outputs (activations) activated by nonlinear functions in the neural network. Compared with full-precision neural networks, it can save a lot of memory and computing, which is conducive to the deployment of models on resource-constrained devices.

(13) Teacher Network

The teacher network usually refers to a more complex network with very good performance and generalization ability. The teacher network in the embodiment of the present application can be a neural network whose neural network parameters (weights) and convolution operation outputs (activations) activated by nonlinear functions in the neural network are full-precision (32-bit floating point numbers), half-precision (16-bit floating point numbers), or commonly used integer types (8bit, 4bit, 2bit integers) and other types of data.

The following introduces the system architecture provided by the embodiments of the present application.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of a system architecture 100 provided in an embodiment of the present application. As shown in the system architecture 100, the data acquisition device 160 is used to acquire training data. In the embodiment of the present application, the training data includes labeled image data, wherein the label of the image may be a category corresponding to the image, or a category corresponding to an object in the image, or a category corresponding to each pixel of the image, and the above categories are mathematically represented as a multidimensional vector.

After collecting the training data, the data collection device 160 stores the training data in the database 130, and the training device 120 obtains the target model 101 (that is, the target binary neural network model in the embodiment of the present application) through training based on the training data maintained in the database 130.

The following will describe in more detail how the training device 120 obtains the target model 101 based on the training data with the first embodiment. The target model 101 can be used to implement the image processing method provided in the embodiment of the present application, that is, the image to be processed is input into the target model 101 after relevant preprocessing, and the predicted value of the image to be processed can be obtained. The target model 101 in the embodiment of the present application can specifically be a target binary neural network model. In the embodiment provided in the present application, the target binary neural network model is obtained by training the initial binary neural network model _M0 . It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the collection of the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not necessarily train the target model 101 completely based on the training data maintained by the database 130, and it may also be possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiment of the present application.

The target model 101 obtained by training the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 1 . The execution device 110 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR)/virtual reality (VR), a vehicle terminal, etc., and can also be a server or a cloud, etc. In FIG. 1 , the execution device 110 is configured with an input/output (I/O) interface 112 for data interaction with an external device. The user can input data to the I/O interface 112 through the client device 140. The input data may include various image or video data in the embodiment of the present application.

When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

Finally, the I/O interface 112 returns the processing results, such as the predicted value of the image to be processed obtained above (i.e., the category label of the image to be processed, or the target identified from the image to be processed, or the result of segmenting the image to be processed) to the client device 140, thereby providing it to the user.

It is worth noting that the training device 120 can generate a corresponding target model 101 based on different training data for different goals or different tasks, and the corresponding target model 101 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results.

In the case shown in FIG. 1 , the user can manually give input data, and the manual giving can be operated through the interface provided by the I/O interface 112. In another case, the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I/O interface 112 and the output results of the output I/O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I/O interface 112 directly stores the input data of the input I/O interface 112 and the output results of the output I/O interface 112 as new sample data in the database 130.

It is worth noting that FIG1 is only a schematic diagram of a system architecture provided by an embodiment of the present invention, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG1, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

As shown in Figure 1, the target model 101 is obtained by training with the training device 120. In the embodiment of the present application, the target model 101 can be a target binary neural network model obtained by training based on the training method of the binary neural network model in the embodiment of the present application. Specifically, the target binary neural network model provided in the embodiment of the present application can be a convolutional neural network or other neural network with similar functions, and this solution does not make any specific limitations on this.

As mentioned in the previous basic concept introduction, convolutional neural network is a deep neural network with convolution structure, which is a deep learning (DL) architecture. Deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the image input into it.

As shown in FIG. 2 , a convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional layer/pooling layer 220 (wherein the pooling layer is optional), and a neural network layer 230 .

Convolutional layer/pooling layer 220:

Convolutional Layer:

As shown in FIG2 , the convolution layer/pooling layer 220 may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layers 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to blur unwanted noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. In the layers 221-226 illustrated in the convolution layer/pooling layer 220 in FIG. 2, a convolution layer may be followed by a pooling layer, or multiple convolution layers may be followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

Neural Network Layer 230:

After being processed by the convolution layer/pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. Because as mentioned above, the convolution layer/pooling layer 220 will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (231, 232 to 23n as shown in Figure 2) and an output layer 240, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, target detection, image classification, and image super-resolution reconstruction.

After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240. The output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (the propagation from 210 to 240 in FIG. 2 is the forward propagation) is completed, the back propagation (the propagation from 240 to 210 in FIG. 2 is the back propagation) will begin to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

It should be noted that the convolutional neural network 200 shown in FIG. 2 is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

The following introduces a chip hardware structure provided by an embodiment of the present application.

FIG3 is a chip hardware structure provided by an embodiment of the present invention, and the chip includes a neural network processor 50. The chip can be set in the execution device 110 shown in FIG1 to complete the calculation work of the calculation module 111. The chip can also be set in the training device 120 shown in FIG1 to complete the training work of the training device 120 and output the target model 101. The algorithms of each layer in the convolutional neural network shown in FIG2 can be implemented in the chip shown in FIG3.

The neural network processor NPU 50 is mounted on the host CPU as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations.

In some implementations, the operation circuit 503 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is stored in the accumulator 508 (accumulator).

The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional/non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

In some implementations, the vector calculation unit 507 can store the processed output vector to the unified memory 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, such as for use in a subsequent layer in a neural network.

The unified memory 506 is used to store input data and output data.

The weight data is directly transferred from the external memory to the input memory 501 and/or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

An instruction fetch buffer 509 connected to the controller 504 is used to store instructions used by the controller 504 .

The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

Among them, the operations of each layer in the convolutional neural network shown in Figure 2 can be performed by the operation circuit 503 or the vector calculation unit 507.

The training device 120 in Figure 1 introduced above can execute the various steps of the method for training a binary neural network model in the embodiment of the present application. The execution device 110 in Figure 1 can execute the various steps of the image processing method (for example, image classification, image segmentation, and target detection) in the embodiment of the present application. The neural network model shown in Figure 2 and the chip shown in Figure 3 can also be used to execute the various steps of the image processing method in the embodiment of the present application. The chip shown in Figure 3 can also be used to execute the various steps of the method for training a binary neural network model in the embodiment of the present application.

As shown in Figure 4, Figure 4 is a schematic diagram of a system architecture 300 provided in an embodiment of the present application. The system architecture includes a local device 301, a local device 302, an execution device 210 and a data storage system 250; wherein the local device 301 and the local device 302 are connected to the execution device 210 via a communication network.

The execution device 210 can be implemented by one or more servers. Optionally, the execution device 210 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and other devices. The execution device 210 can be arranged at a physical site, or distributed at multiple physical sites. The execution device 210 can use the data in the data storage system 250, or call the program code in the data storage system 250 to implement the method for training a binary neural network or the image processing method (e.g., image super-resolution method, image denoising method, image demosaicing method, and image deblurring method) of the embodiment of the present application.

Specifically, the execution device 210 may perform the following process:

S1: Determine the knowledge distillation framework; wherein the teacher network in the knowledge distillation framework is a trained neural network model, and the student network in the knowledge distillation framework is an initial binary neural network model _M0 , and the teacher network and the student network respectively include N layers of neural networks, where N is a positive integer; S2: Use the j+1th batch of images and the target loss function to train the binary neural network model _Mj to obtain the binary neural network model _Mj+1 ; wherein the binary neural network model _Mj is obtained based on the jth batch of images, and j is a positive integer; the target loss function includes an angle loss term, which is used to describe the difference between the first angle corresponding to the i-th layer of the neural network in the teacher network and the second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix of the j+1th batch of images in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix of the j+1th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N; S3: When the preset conditions are met, the binary neural network model Mj is trained. _j+1 is used as the target binary neural network model; otherwise, j=j+1 is set and step S2 is repeated.

Through the above-mentioned execution device 210, a target binary neural network model can be trained and obtained, and the target binary neural network model can be used for image processing, speech processing, natural language processing, etc. For example, the target binary neural network model can be used to implement the image classification, target detection and image segmentation methods in the embodiments of the present application.

Alternatively, the device 210 executing the above process can be constructed into an image processing apparatus, which can be used for image processing (for example, it can be used to implement the image classification, target detection and image segmentation methods in the embodiments of the present application).

Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

The local device of each user can interact with the execution device 210 through a communication network of any communication mechanism/communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

In one implementation, the local device 301 and the local device 302 obtain relevant parameters of the neural network from the execution device 210, deploy the neural network on the local device 301 and the local device 302, and use the neural network to perform image processing on the image to be processed to obtain a processing result of the image to be processed.

In another implementation, the neural network can be directly deployed on the execution device 210. The execution device 210 obtains the image to be processed from the local device 301 and the local device 302, and uses the neural network to perform image processing on the image to be processed to obtain a processing result of the image to be processed.

In one implementation, the local device 301 and the local device 302 obtain relevant parameters of the image processing apparatus from the execution device 210, deploy the image processing apparatus on the local device 301 and the local device 302, and use the image processing apparatus to perform image processing on the image to be processed to obtain a processing result of the image to be processed.

In another implementation, the image processing apparatus may be directly deployed on the execution device 210. The execution device 210 obtains the image to be processed from the local device 301 and the local device 302, and uses the image processing apparatus to perform image processing on the image to be processed to obtain a processing result of the image to be processed.

That is to say, the above-mentioned execution device 210 can also be a cloud device, in which case the execution device 210 can be deployed in the cloud; or, the above-mentioned execution device 210 can also be a terminal device, in which case the execution device 210 can be deployed on the user terminal side, which is not limited in the embodiments of the present application.

The following is a detailed introduction to the method for training a binary neural network model and an image processing method (for example, the image processing method may include image classification, target detection, and image segmentation) according to an embodiment of the present application in conjunction with the accompanying drawings.

Please refer to Figure 6, which is a schematic diagram of the structure of a knowledge distillation framework 600 in an embodiment of the present application. The knowledge distillation framework is the knowledge distillation framework used in the embodiment of the present application. The knowledge distillation framework 600 includes a teacher network and a student network, the teacher network is a trained neural network model, and the student network is an initial binary neural network model _M0 ; the teacher network and the student network each include N layers of neural networks, where N is a positive integer. The knowledge distillation framework 600 is also referred to as a layer-wise search (LWS-Det) architecture in the embodiment of the present application.

It should be noted that FIG6 only shows the partial structure of the i-th layer neural network in the teacher network and the student network, and the structures of the other layers of the neural network are the same as those of the i-th layer, where i is a positive integer greater than 1.

基于第一参考权重矩阵

第二参考权重矩阵

第一概率矩阵和第二概率矩阵确定第i层神经网络的二值权重矩阵

二值权重矩阵

的确定过程会在后文图7实施例中进行详细描述；然后对二值权重矩阵

和二值输入矩阵

进行卷积运算，得到参考特征矩阵；最后，通过权重缩放尺度因子a_i、PReLU和BN层等处理，得到第i层神经网络的输出结果。In the teacher network, the output of the i-1th layer of the neural network is input to the i-th layer of the neural network. After the batch normalization (BN) layer and the activation function (not shown in FIG6 ), the input matrix a _i-1 of the i-th layer is obtained. Then, the input matrix a _i-1 and the weight matrix w _i are convolved to obtain the first convolution output result. Finally, the output of the i-th layer of the neural network is obtained after the parametric linear rectifier function (PReLU) and the BN layer. In the student network, the output of the i-1th layer of the neural network is input to the i-th layer of the neural network. After the batch normalization (BN) layer, the activation function and the binarization (not shown in FIG6 ), the binary input matrix of the i-th layer is obtained.

Based on the first reference weight matrix

The second reference weight matrix

The first probability matrix and the second probability matrix determine the binary weight matrix of the i-th layer neural network

Binary weight matrix

The determination process will be described in detail in the embodiment of FIG. 7 later; then the binary weight matrix

and the binary input matrix

A convolution operation is performed to obtain a reference feature matrix; finally, the output result of the i-th layer neural network is obtained through weight scaling factor a _i , PReLU and BN layers.

It should be understood that the PReLU in FIG6 can also be replaced by other activation functions, and the present application does not limit this. The specific process of performing convolution operation on the input matrix a _i-1 and the weight matrix w _i in the i-th layer neural network can be seen in FIG5 , which will not be repeated here.

Please refer to Figure 7, which is a flow chart of a training method 700 for a binary neural network model in an embodiment of the present application. As shown in Figure 7, the method 700 includes step S1, step S2 and step S3.

In some examples, the method 700 may be executed by the execution device 110 in FIG. 1 , the chip shown in FIG. 3 , the execution device 210 in FIG. 4 , and the like.

Step S1: determine the knowledge distillation framework; wherein the teacher network in the knowledge distillation framework is a trained neural network model, the student network in the knowledge distillation framework is an initial binary neural network model M ₀ , the teacher network and the student network each contain N layers of neural networks, N is a positive integer.

Among them, the trained neural network model in the teacher network can be a neural network model whose neural network parameters (weights) and convolution operation outputs (activations) activated by nonlinear functions are real number types, and the real number types include but are not limited to full precision (32-bit floating point numbers), half precision (16-bit floating point numbers), or integers (8bit, 4bit, 2bit integers). The initial binary neural network model _M0 is obtained after initializing the parameters in the model.

It should be understood that the embodiments of the present application can be applied to all binary neural network training processes, and the present application is not limited to this.

Step S2: Use the j+1th batch of images and the target loss function to train the binary neural network model M _j to obtain the binary neural network model M _j+1 ; wherein the binary neural network model M _j is obtained based on the jth batch of images, and j is a positive integer; the target loss function includes an angle loss term, which is used to describe the difference between a first angle corresponding to the i-th layer of the neural network in the teacher network and a second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix of the j+1th batch of images in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix of the j+1th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N.

Specifically, in the process of using the j+1th batch of images for the j+1th training, the j+1th batch of images is input into the above-mentioned knowledge distillation framework, that is, simultaneously input into the teacher network and the student network, and the teacher network is used to guide the training of the student network. The j+1th batch of images includes at least one image.

For each image in the j+1th batch of images, the angle between the input matrix corresponding to each image in the i-th layer of the neural network in the teacher network and the weight matrix of the neural network in this layer is calculated, that is, the first angle; at the same time, the angle between the binary input matrix corresponding to each image in the i-th layer of the neural network in the student network and the binary weight matrix of the neural network in this layer is calculated, that is, the second angle; the angle loss of each image in the i-th layer of the neural network is calculated based on formula (2). Then, based on the angle loss of each image in the j+1th batch of images in the i-th layer of the neural network, the average angle loss is calculated, and the average angle loss is the angle loss term of the j+1th batch of images in the i-th layer of the neural network. Finally, the angle loss terms of the j+1th batch of images in each layer of the neural network are accumulated to obtain the loss value of the angle loss term in the objective function in the j+1th training.

为每张图像在第i层神经网络中的角度损失项；cosθ_i为每张图像对应的第一角度；

为每张图像对应的第二角度；a_i-1为每张图像在教师网络中第i层神经网络中的输入矩阵；w_i为教师网络中第i层神经网络中的权重矩阵；

为每张图像在学生网络中第i层神经网络中的二值输入矩阵；

为学生网络中第i层神经网络中的二值权重矩阵；

为张量积(tensor product)。in,

is the angle loss term of each image in the i-th layer of the neural network; cosθ _i is the first angle corresponding to each image;

is the second angle corresponding to each image; a _i-1 is the input matrix of each image in the i-th layer of the neural network in the teacher network; w _i is the weight matrix in the i-th layer of the neural network in the teacher network;

The binary input matrix for each image in the i-th layer of the neural network in the student network;

is the binary weight matrix in the i-th layer of the neural network in the student network;

is the tensor product.

It can be seen that in the embodiment of the present application, a trained teacher network is used to guide the training of the student network, and an angle loss term is designed in the target loss function to update the parameters in the student network. On the one hand, the feature extraction results of the student network for the input samples can be made close to the feature extraction results of the teacher network for the input samples. On the other hand, the angle between the binary weight matrix and the binary input matrix in the student network is made close to the angle between the weight matrix and the input matrix in the teacher network. In summary, compared with the prior art in which the training process of the binary neural network model does not consider the quantized angle loss, the present application can introduce the knowledge distillation framework and the angle loss term in the loss function to make the performance of the trained student network as close to the performance of the teacher network as possible, thereby improving the prediction accuracy of the target binary neural network model trained in the embodiment of the present application.

In a feasible implementation, the target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; the first convolution output result is based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network in the teacher network for the j+1-th batch of images; the second convolution output result is based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, as well as the binary input matrix of the j+1-th batch of images in the i-th layer neural network in the student network.

Specifically, for each image in the j+1th batch of images, the convolution result loss corresponding to the i-th layer of the neural network can be calculated using formula (3): that is, the input matrix of the i-th layer of the neural network in the teacher network and the weight matrix of the i-th layer of the neural network in the teacher network are convolved to obtain the first convolution output result corresponding to each image; the binary input matrix of the i-th layer of the neural network in the student network and the binary weight matrix of the i-th layer of the neural network in the student network are convolved to obtain the reference feature matrix, and the reference feature matrix is element-wise multiplied by the weight scaling factor of the i-th layer of the neural network in the student network to obtain the second convolution output result corresponding to each image. After subtracting the first convolution output result and the second convolution output result of each image in the i-th layer of the neural network, the bi-norm is calculated to obtain the convolution result loss value of each image in the i-th layer of the neural network. Then, based on the convolution result loss value of each image in the j+1th batch of images, the average convolution result loss is calculated. The average convolution result loss is the convolution result loss of the j+1th batch of images in the i-th layer of the neural network. Finally, the convolution result loss corresponding to the j+1th batch of images in each layer of the neural network is accumulated to obtain the loss value of the convolution result loss item in the objective function in the j+1th training.

为每张图像在第i层神经网络中的卷积结果损失；E_i为求取每张图像在第i层神经网络中的卷积结果损失的函数；α_i为第i层神经网络对应的权重缩放尺度因子；⊙为哈达玛积(Hadamard product)，表示对两个矩阵中对应位置元素相乘；°为逐元素相乘；公式(3)中其余参数的物理意义参见公式(2)中相关解释，此处不再赘述。in,

is the convolution loss of each image in the i-th layer of the neural network; E _i is the function for obtaining the convolution loss of each image in the i-th layer of the neural network; α _i is the weight scaling factor corresponding to the i-th layer of the neural network; ⊙ is the Hadamard product, which means the multiplication of the elements at the corresponding positions in the two matrices; ° is the element-by-element multiplication; the physical meanings of the other parameters in formula (3) refer to the relevant explanations in formula (2) and will not be repeated here.

It can be seen that in the embodiment of the present application, the convolution result loss term is introduced into the target loss function to make the second convolution output result in the student network as close as possible to the first convolution output result in the teacher network, that is, the output result of each layer of the neural network in the student network is made as close as possible to the output result of each layer of the neural network in the teacher network, thereby ensuring that the predicted value of the output of the student network is close to the predicted value of the teacher network, thereby improving the prediction accuracy of the target binary neural network model obtained after training.

In a feasible implementation, the objective loss function also includes a weight loss term; wherein the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network.

Specifically, in the j+1th training, the weight loss corresponding to the i-th layer of the neural network can be calculated using formula (4): that is, the binary weight matrix of the i-th layer of the neural network in the student network is multiplied element by element using the weight scaling factor of the i-th layer of the neural network in the student network to obtain the first weight matrix; then the first weight matrix is subtracted from the weight matrix of the i-th layer of the neural network in the teacher network and the binary norm is calculated to obtain the weight loss corresponding to the i-th layer of the neural network. According to the above steps, the weight loss corresponding to each layer of the neural network in the j+1th training is calculated, and the weight loss of each layer of the neural network is accumulated to obtain the loss value of the weight loss term in the target loss function in the j+1th training.

为第j+1次训练中第i层神经网络对应的权重损失；其余参数物理意义参见公式(2)和公式(3)中相关解释，此处不再赘述。in,

is the weight loss corresponding to the i-th layer of the neural network in the j+1-th training. The physical meanings of the remaining parameters refer to the relevant explanations in formulas (2) and (3) and will not be repeated here.

It can be seen that the embodiment of the present application can also introduce a weight loss term that characterizes the difference between the weight matrix in the teacher network and the binary weight matrix in the student network into the target loss function, and train the student network together with the convolution result loss term and the angle loss term. By introducing the above three model performance measurement indicators into the target loss function to train the student network, the performance of the target neural network model obtained after training can be maximized, making it as close to the performance of the neural network as possible.

In a feasible implementation, the above-mentioned training of the binary neural network model M _j using the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 includes: inputting the j+1th batch of images into the binary neural network model M _j to obtain prediction values of the j+1th batch of images; updating the parameters of each layer of the neural network in the binary neural network model M _j based on the prediction values of the j+1th batch of images, the labels of the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 .

Specifically, the target loss function also includes a detection loss term, which is obtained based on the predicted value of the j+1th batch of images and the label of the j+1th batch of images. The specific calculation process of the detection loss term is as follows: calculate the difference between the predicted value and the label of each image in the j+1th batch of images, and then take the average value, that is, obtain the loss value of the detection loss term in the target loss function during the j+1th training process. It should be understood that the above-mentioned target loss function may include one or more of the angle loss term, the convolution result loss term, the weight loss term or the detection result loss term. The loss function value in the j+1th training process is calculated based on the target loss function. The smaller the loss function value, the closer the performance of the student network and the teacher network. Update the parameters in each layer of the neural network in the binary neural network model _Mj in the direction of making the target loss function value smaller and smaller, and obtain the binary neural network model Mj ₊₁ .

Further, optionally, when updating the parameters in each layer of the neural network in the binary neural network model _Mj, the parameters of each layer of the neural network can be updated layer by layer in the order from the Nth layer of the neural network to the 1st layer of the neural network, which is not limited in the present application.

It should be understood that after the forward propagation process of a training is completed, the embodiment of the present application can obtain an image prediction value based on the end of the forward propagation process; then calculate the loss function value in the training process based on the image prediction value, the image label, and the above-mentioned target loss function, and update the model parameters based on the target loss function value.

In a feasible implementation, the j+1th batch of images is input into the binary neural network model _Mj to obtain the predicted values of the j+1th batch of images, including: P1: obtaining the binary weight matrix of the i-th layer of the neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer of the neural network in the binary neural network model _Mj ; P2: obtaining the second convolution output result of the i-th layer of the neural network based on the binary input matrix and binary weight matrix of the j+1th batch of images in the i-th layer of the neural network; wherein, the element at any position in the probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P3: let i=i+1, and repeat steps P1-P2, and obtain the predicted values of the j+1th batch of images based on the second convolution output result of the N-th layer of the neural network.

Specifically, during the forward propagation of the i-th layer of the neural network in the j+1th training, the binary weight matrix of the i-th layer of the neural network is obtained based on the reference weight matrix and probability matrix corresponding to the i-th layer of the neural network, and the binary weight matrix is the model parameter in the i-th layer of the student network. For each image in the j+1th batch of images, when forward propagating to the N-th layer of the neural network, the second convolution output result of each image is obtained based on the binary weight matrix of the N-th layer of the neural network and the binary input matrix corresponding to each image in the N-th layer of the neural network; then the second convolution output result of each image is input into the activation function, BN layer, and fully connected Softmax layer structures to obtain the prediction value of the binary neural network model M _j for each image in the j+1th batch of images.

It should be understood that the above activation function may be a rectified linear unit (ReLU), PReLU or other possible activation functions, which is not limited in this application.

It can be seen that in the embodiment of the present application, during the forward propagation process of each layer of the neural network during each training, the binary weight matrix of each layer of the neural network is first determined based on the reference weight matrix and probability matrix corresponding to each layer of the neural network, and then the second convolution output result of each layer of the neural network is calculated based on the binary input matrix and the binary weight matrix of each layer of the neural network, and then when forward propagating to the Nth layer of the neural network, the image prediction value of the model output during this training process can be calculated based on the second convolution output result of the Nth layer of the neural network. Then, in the subsequent back propagation process, the second convolution output result and the first convolution output result obtained during the forward propagation process, as well as the image prediction value and the image label can be used to calculate the loss function value of this training process, and then the model parameters are adjusted according to the loss function value to ensure that the optimal target binary neural network model is obtained.

In a feasible implementation, the reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; the binary weight matrix of the i-th layer neural network is obtained based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network _model Mj, including: determining the element at any position in the target binary weight matrix based on the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the second probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix; wherein the element at any position in the first probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix.

Optionally, during the j+1th training process, a differentiable binary search (DBS) method can be used to search for the binary weight matrix of the i-th layer neural network in the student network from the search space O; wherein the search space can be the first reference weight matrix and the second reference weight matrix corresponding to the i-th layer neural network in the student network. Further, optionally, all elements in the first reference weight matrix and the second reference weight matrix are 1 or -1, when all elements in the first reference weight matrix are 1, all elements in the second reference weight matrix are -1, and when all elements in the first reference weight matrix are -1, all elements in the second reference weight matrix are 1. The first reference weight matrix corresponds to the first probability matrix, and the second reference weight matrix corresponds to the second probability matrix. It should be understood that the width and height of the first reference weight matrix, the second reference weight matrix, the first probability matrix and the second probability matrix are the same.

Specifically, the element at any position of the binary weight matrix in the student network is determined according to the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the first probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix. Further, optionally, the element corresponding to the larger probability value of the first probability value and the second probability value at any position in the reference weight matrix can be used as the element at any position in the binary weight matrix.

和

第一概率矩阵和第二概率矩阵可以分别为图6中所示的

和

当

的值大于

的值时，将

在第一参考权重矩阵

对应位置上的元素，即-1，确定为二值权重矩阵

中第一行第一列的元素，按照此规则依次确定二值权重矩阵

中的每个元素。For example, the first reference weight matrix and the second reference weight matrix in the i-th layer neural network can be respectively as shown in FIG.

and

The first probability matrix and the second probability matrix can be respectively as shown in FIG. 6

and

when

The value is greater than

When the value of

In the first reference weight matrix

The element at the corresponding position, i.e. -1, is determined as a binary weight matrix

The elements in the first row and first column of the binary weight matrix are determined in turn according to this rule

Each element in .

Optionally, before determining the binary weight matrix of the i-th layer neural network, the first probability matrix and the second probability matrix may be normalized with reference to formula (5); and then the binary weight matrix corresponding to the i-th layer neural network may be determined according to formula (6).

为归一化后的概率矩阵；操作o_k,o′_k∈O；s.t.表示满足于；

表示操作o_k的第i层神经网络对应的二值权重矩阵中第l个权重对应的第一概率值和第二概率值；

表示取第一概率值和第二概率值中的最大值；

为第i层神经网络对应的二值权重矩阵

第l个位置上的元素。In formulas (5) and (6),

is the normalized probability matrix; operation o _k ,o′ _k ∈O; st means satisfied;

Represents the first probability value and the second probability value corresponding to the lth weight in the binary weight matrix corresponding to the i-th layer neural network of operation o _k ;

Indicates taking the maximum value of the first probability value and the second probability value;

is the binary weight matrix corresponding to the i-th layer of the neural network

The element at position l.

It can be seen that in the embodiment of the present application, the reference weight matrix in each layer of the neural network in the student network includes a first reference weight matrix and a second reference weight matrix; and the probability matrix in each layer of the neural network includes a first probability matrix and a second probability matrix, the first probability matrix corresponds to the first reference weight matrix, and the second probability matrix corresponds to the second reference weight matrix. Based on the probability value of the elements at the same position in the first reference weight matrix and the second reference weight matrix, the element at that position in the first reference weight matrix or the second reference weight matrix is selected as the element at that position in the binary weight matrix; based on this rule, the binary weight matrix corresponding to each layer of the neural network in the word training process is obtained; then, based on the binary weight matrix, the second convolution output result of each layer of the neural network corresponding to the above embodiment and the target loss function value in each training process are calculated, thereby ensuring the correct conduct of the training process, and then obtaining the optimal binary neural network model.

In a feasible implementation, the above-mentioned second convolution output result of the i-th layer neural network is obtained based on the binary input matrix and binary weight matrix of the j+1-th batch of images in the i-th layer neural network, including: performing convolution operations on the binary input matrix and binary weight matrix of each image in the j+1-th batch of images in the i-th layer neural network respectively to obtain a reference feature matrix of each image; and scaling the reference feature matrix of each image using the weight scaling factor of the i-th layer neural network to obtain a second convolution output result.

然后对每张图像的二值输入矩阵和第i层神经网络的二值权重矩阵进行卷积运算，得到每张图像的参考特征矩阵；最后利用第i层神经网络的权重缩放尺度因子α_i对每张图像的参考特征矩阵进行逐元素相乘，得到每张图像的第二卷积输出结果。Specifically, as shown in FIG6 , in the i-th layer neural network of the student network, for each image in the j+1-th batch of images, after activating, standardizing, and binarizing the input matrix of the i-th layer neural network for each image, the binary input matrix of each image in the i-th layer neural network is obtained.

Then, the binary input matrix of each image and the binary weight matrix of the i-th layer neural network are convolved to obtain the reference feature matrix of each image; finally, the reference feature matrix of each image is element-by-element multiplied using the weight scaling factor α _i of the i-th layer neural network to obtain the second convolution output result of each image.

It can be seen that in the embodiment of the present application, the reference feature matrix of each image is obtained based on the binary input matrix and the binary weight matrix in each layer of the neural network; then the reference feature matrix of each image is scaled using the weight scaling factor of each layer of the neural network to obtain the second convolution output result. The second convolution output result characterizes the characteristics of the input image, and the design of the convolution result loss term in the target loss function can make the trained target binary neural network model retain the feature extraction ability of the teacher network as much as possible, thereby improving the prediction accuracy of the target binary neural network model.

In a feasible implementation manner, the above parameters include at least one of the probability matrix or the weight scaling factor.

Specifically, during the j+1th training process, after the loss function value of the j+1th training process is calculated by referring to the process in the above embodiment, at least one of the probability matrix or weight scaling factor in each layer of the neural network in the student network can be adjusted according to the trend of gradually reducing the target loss function value.

It can be seen that in the embodiment of the present application, each training back propagation process will update at least one of the probability matrix and weight scaling factor in each layer of the neural network, so that during the forward propagation process of the next training, the binary weight matrix in each layer of the neural network is updated, and then based on the updated binary weight matrix and/or weight scaling factor, the image prediction value and loss function value output by the training model are obtained, and based on the obtained loss function value, the model parameters are further adjusted to ensure that a student network with performance close to that of the teacher network is obtained.

Optionally, when the number of images contained in the j+1th batch of images is one, the expression of the above objective loss function can be shown as formula (7):

为角度损失项；

为卷积结果损失项；

为权重损失项；λL^Lim为细纹理损失项，用于限制对检测头的辅助学习；λ、μ和γ为模型中的超参数，可选地，可以将其分别设置为0.01、0.01和0.0001，本申请对此不限定。Among them, L is the target loss function; L ^GT is the detection result loss term;

is the angle loss term;

is the convolution result loss term;

is the weight loss term; λL ^Lim is the fine texture loss term, which is used to limit the auxiliary learning of the detection head; λ, μ and γ are hyperparameters in the model, which can be set to 0.01, 0.01 and 0.0001 respectively, which is not limited in this application.

Step S3: When the preset conditions are met, the binary neural network model M _j+1 is used as the target binary neural network model; otherwise, j=j+1 is set and step S2 is repeated.

Optionally, after the j+1th training is completed, when any of the following preset conditions is met, the binary neural network model Mj+1 is used as the target binary neural network model:

Preset condition 1: The number of training images used in the previous j+1 training processes reaches a preset number. The preset number can be the total number of images in the training set or any other value, which is not limited in this application.

Preset condition 2: During the j+1th training process, the target loss function value is less than a preset value. The preset value can be set according to a specific scenario, and this application does not limit this.

Preset condition three: The prediction accuracy rate obtained based on the predicted values and labels of multiple images obtained in the j+1th training process is higher than a preset ratio. The preset value can be set according to a specific scenario, and this application does not limit this.

It can be seen that in the embodiment of the present application, the teacher network trained in the knowledge distillation framework is used to guide the training process of the student network, and the angle loss term, the convolution result loss term and the weight loss term are designed in the target loss function to jointly update the parameters in the student network. Specifically: the angle loss term in the target loss function is used to minimize the angle loss of the trained target binary neural network after quantization, and the convolution result loss term and the weight loss term are used to minimize the amplitude loss of the trained target binary neural network after quantization. Thereby, the performance of the trained target binary neural network model is close to the performance of the teacher network to the greatest extent, and the prediction accuracy of the trained target binary neural network model in the embodiment of the present application is improved.

Please refer to Figure 8, which is a flow chart of another model training method 800 provided in an embodiment of the present application. Method 800 includes step S810 and step S820. The model includes a teacher network and a student network, the teacher network is a trained neural network model, the student network is a binary neural network model, and the teacher network and the student network each include N layers of neural networks, where N is a positive integer.

Step S810, training the binary neural network model using the teacher network and the target loss function; wherein the target loss function includes an angle loss term, which is used to describe the difference between a first angle corresponding to the i-th layer of the neural network in the teacher network and a second angle corresponding to the i-th layer of the neural network in the student network; the first angle is based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix in the i-th layer of the neural network in the teacher network; the second angle is based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N.

Step S820, repeat the above step S810 until the iteration termination condition is met to obtain the target binary neural network model.

In a feasible implementation, the target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; the first convolution output result is based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network in the teacher network; the second convolution output result is based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, as well as the binary input matrix in the i-th layer neural network in the student network.

In a feasible implementation, the objective loss function also includes a weight loss term; wherein the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network.

In a feasible implementation, the above-mentioned training of the binary neural network model using the teacher network and the target loss function includes: inputting the training image into the binary neural network model to obtain the predicted value of the training image; and updating the parameters in the binary neural network model based on the predicted value of the training image, the label of the training image and the target loss function.

In a feasible implementation, the above-mentioned input of the training image into the binary neural network model to obtain the predicted value of the training image includes: P1: obtaining the binary weight matrix of the i-th layer neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model; wherein, the element at any position in the probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P2: obtaining the second convolution output result of the i-th layer neural network according to the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network; P3: setting i=i+1, and repeating steps P1-P2, and obtaining the predicted value of the training image based on the second convolution output result of the N-th layer neural network.

In a feasible implementation, the reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; the binary weight matrix of the i-th layer neural network is obtained based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model, including: determining the element at any position in the target binary weight matrix based on the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the second probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix; wherein the element at any position in the first probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix.

In a feasible implementation, the above-mentioned second convolution output result of the i-th layer neural network is obtained according to the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network, including: performing a convolution operation on the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network to obtain a reference feature matrix of the training image; scaling the reference feature matrix of the training image using the weight scaling factor of the i-th layer neural network to obtain a second convolution output result.

In one possible implementation, the parameter includes at least one of a probability matrix or a weight scaling factor.

It should be understood that the training images in the above-mentioned embodiment of FIG. 8 can be understood as the images used in each training process in FIG. 7 . The specific training process in the embodiment of FIG. 8 can refer to the description of the corresponding process in FIG. 7 , which will not be repeated here.

Please refer to Figures 9-A to 9-C, which are schematic diagrams of the distribution of extracted features of different network models in the embodiments of the present application. As shown in Figure 9-A, Figure 9-A is a distribution diagram of the extracted features of the first layer of neural network and the last layer of neural network in the teacher network; Figure 9-B is a distribution diagram of the extracted features of the first layer of neural network and the last layer of neural network in the binary neural network in the embodiment of the present application; Figure 9-C is a distribution diagram of the extracted features of the first layer of neural network and the last layer of neural network in the binary neural network obtained by using an efficient binarized object detector (An Efficient Binarized Object Detector, BiDet) method.

It can be seen that the distribution of features extracted by the binary neural network obtained by the embodiment of the present application is close to the distribution of features extracted by the teacher network, while the distribution of features extracted by the binary neural network obtained by the BiDet method is quite different from the distribution of features extracted by the teacher network, indicating that the binary neural network model trained by the method of the embodiment of the present application has a performance close to that of the teacher network model, and the prediction accuracy is higher than other binary neural network models in the prior art.

和二值输入向量

之间的夹角

以及各自的幅值，与教师网络中对应向量之间的夹角θ和幅值基本相同，即图10中的(d)所示。采用图10中的(b)所示方法量化得到的二值权重向量

和二值输入向量

发生了重合，且量化后得到的二值权重向量的幅值大于教师网络中权重向量的幅值。采用图10中的(c)所示方法进行量化得到的二值权重向量

和二值输入向量

发生了重合，即夹角为零；且二值权重向量的幅值远小于教师网络中权重向量的幅值。Please refer to Figure 10, which is a schematic diagram of the angle between the feature matrix and the weight matrix in the embodiment of the present application. The feature matrix angle refers to the angle between the weight matrix and the input matrix in any layer of the neural network in the above embodiments. In Figure 10, the feature matrices in the teacher network and the binary neural network are simplified into three-dimensional vectors for representation. (a) in Figure 10 represents the angle between the weight vector w and the input vector a in the teacher network, as well as their respective amplitudes. (d) in Figure 10 represents the angle between the binary weight vector and the binary input vector of the binary neural network in the embodiment of the present application, as well as their respective amplitudes. (b) and (c) in Figure 10 respectively represent the angle between the binary weight vector and the binary input vector in the remaining binary neural networks, as well as their respective amplitudes. Comparing (b), (c) and (d) in Figure 10 with (a), it can be seen that the binary weight vector quantized using the method in the embodiment of the present application is

and the binary input vector

The angle between

and their respective amplitudes are basically the same as the angle θ and amplitude between the corresponding vectors in the teacher network, as shown in (d) in Figure 10. The binary weight vector quantized using the method shown in (b) in Figure 10 is

and the binary input vector

The binary weight vector obtained by quantization is larger than the weight vector in the teacher network.

and the binary input vector

There is an overlap, that is, the angle is zero; and the amplitude of the binary weight vector is much smaller than the amplitude of the weight vector in the teacher network.

It can be seen that in the embodiment of the present application, the angle loss term is introduced into the objective function to minimize the difference between the angle between the feature matrix and the weight matrix in the target binary neural network and the angle between the feature matrix and the weight matrix in the teacher network, so that the performance of the trained target binary neural network is close to the performance of the teacher network, that is, the prediction accuracy.

FIG11 is a flowchart of the image processing method of the present application. The method 1100 in FIG11 includes step 1110 and step 1120.

In some examples, the method 1100 may be executed by a device such as the execution device 110 in FIG. 1 , the chip shown in FIG. 3 , and the execution device 210 in FIG. 4 .

Step 1110, obtaining an image to be processed.

Step 1120, using the target binary neural network model to perform image processing on the image to be processed to obtain a predicted value of the image to be processed.

Among them, the target binary neural network model is obtained through K trainings, and in the j+1th training among the K trainings: the binary neural network model _Mj is trained using the j+1th batch of images and the target loss function to obtain the binary neural network model Mj ₊₁ ; the binary neural network model _Mj is the student network in the knowledge distillation framework; the teacher network in the knowledge distillation framework is a trained neural network model, and the teacher network and the student network respectively include N layers of neural networks, where N is a positive integer; the target loss function includes an angle loss term; K is a positive integer, j is an integer greater than or equal to zero and less than or equal to K; the angle loss term is used to describe the difference between the first angle corresponding to the i-th layer of the neural network in the teacher network and the second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix corresponding to the i-th layer of the neural network in the teacher network and the input matrix of the j+1th batch of images in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix corresponding to the i-th layer of the neural network in the student network and the binary input matrix of the j+1th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N.

In a feasible implementation, the above-mentioned image processing includes at least one of image classification, target detection or image segmentation.

It can be seen that the method in the embodiment of the present application can be used for any task in image classification, target detection or image segmentation. By applying the image processing method in the embodiment of the present application to the above three tasks, the image processing effect can be improved, that is, the versatility of this model is good.

The specific training process of the above-mentioned target binary neural network model can refer to the specific description of method 700 in Figure 7 and method 800 in Figure 8, which will not be repeated here.

Optionally, the above methods 700 and 800 can be processed by a CPU, or by a CPU and a GPU together, or other processors suitable for neural network calculations can be used without a GPU, and this application does not impose any restrictions.

The above-mentioned image processing may include image classification, image segmentation, target detection or other related image processing, which is not specifically limited in the present application. The following specifically describes the application of method 1100 in the fields of image classification, image segmentation and target detection.

Image classification: The image to be processed is input into the target binary neural network model. The backbone network in the model extracts the features of the image to be processed, obtains the feature vector of the image to be processed, and performs corresponding calculations layer by layer based on the feature vector. Finally, the predicted value of the image to be processed is obtained. The predicted value can be a multidimensional vector. Each element in the multidimensional vector corresponds to an image category, and each element in the multidimensional vector is used to represent the probability value of the image to be processed being the image category corresponding to each element.

Image segmentation: The image to be processed is input into the target binary neural network model. The backbone network in the model extracts the features of the image to be processed, obtains the feature vector of the image to be processed, and performs corresponding calculations layer by layer based on the feature vector to obtain multiple predicted values of the image to be processed. The multiple predicted values correspond to multiple pixel points of the image to be processed. Each predicted value in the multiple predicted values is a multidimensional vector, and each element in any multidimensional vector corresponds to an image category. Each element in any multidimensional vector is used to represent the probability value that the pixel point corresponding to any multidimensional vector is the image category corresponding to each element.

Target detection: The image to be processed is input into the target binary neural network model. The backbone network in the model extracts the features of the image to be processed and obtains the feature vector of the image to be processed. The model first identifies and segments the target object in the image to be processed based on the extracted feature vector, obtains the target area corresponding to the target object, and finally outputs a multidimensional vector corresponding to the target area. The meaning of the multidimensional vector is the same as that in the above image classification, which will not be repeated here.

It can be understood that the embodiments described in Figures 7 and 8 are the training stage of the binary neural network model (the stage executed by the training device 120 shown in Figure 1), and the specific training is carried out using the embodiments shown in Figure 7 or 8 or any possible implementation method based on the embodiments; and the embodiment described in Figure 11 can be understood as the application stage of the binary neural network model (the stage executed by the execution device 110 shown in Figure 1), which can be specifically embodied by using the target binary neural network model trained by the embodiments shown in Figures 7 or 8, and obtaining the predicted value of the image to be processed based on the image to be processed input by the user.

Please refer to FIG12 , which is a flowchart of another image processing method 1200 provided in an embodiment of the present application. The method 1200 in FIG12 includes step 1210 and step 1220 .

In some examples, the method 1200 may be executed by a device such as the execution device 110 in FIG. 1 , the chip shown in FIG. 3 , and the execution device 210 in FIG. 4 .

Step 1210, obtaining an image to be processed.

Step 1220, using the target binary neural network model to perform image processing on the image to be processed to obtain a predicted value of the image to be processed; wherein the target binary neural network model is obtained by training the initial binary neural network model _M0 in the knowledge distillation framework through the target loss function, the initial binary neural network model _M0 is the student network in the knowledge distillation framework, and the teacher network in the knowledge distillation framework is a trained neural network model; the target loss function includes an angle loss term, and the angle loss term is used to describe the difference between the angle between the feature matrix and the weight matrix in the teacher network and the angle between the feature matrix and the weight matrix in the student network.

It can be seen that in the embodiments of the present application, since the method in the first aspect introduces a knowledge distillation framework during training and introduces a corresponding angle loss term in the target loss function, the target binary neural network model trained by the method in the first aspect has a greatly improved model accuracy compared to the existing binary neural network model; at the same time, since the binary neural network has fewer model parameters and is more lightweight than the neural network model, it has good application prospects in embedded devices.

Please refer to Table 1, which describes the detection effect and model-related performance of the binary neural network model and the teacher network trained on the Pattern Analysis, Statistical Modeling and Computational Learing (PASCAL VOC) dataset using different methods for target detection.

In this simulation experiment, the target detection framework uses Faster Region-based Convolutional Neural Networks (Faster RCNN) and Single Shot Multi-Box Detector (SSD) respectively; among them, Faster RCNN is a general two-stage target detection framework, and SSD is a general single-stage target detection framework.

Under the condition that the short side and long side of Faster RCNN are 600×1000 pixels respectively (i.e., the input in Table 1 is 600×1000), the backbone network uses three networks of ResNet (Residual Network, ResNet), namely ResNet-18, ResNet-34 and ResNet-50. Then, in the framework containing different backbone networks, different quantization methods are used to quantize the models, including: Real-valued, in which the numerical type in the network after quantization is 32-bit floating point type; Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (DeRoFa-Net), in which the numerical type in the network after quantization is 4-bit floating point type; and Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithms. TrainingAlgorithm, Bi-Real-Net); An Efficient Binarized Object Detector, BiDet; Towards Precise Binary Neural Network with Generalized Activation Functions, ReActNet and layer-wise search, LWS-Det; Among them, after quantization by the four quantization methods of Bi-Real-Net, BiDet, ReActNet and LWS-Det, the numerical type in the network is 1-bit integer, and the LWS-Det method is the quantization method used in the embodiments of the present application. Table 1 also includes the memory usage, Giga floating point operations Per Second, GFLOPs, and mean average precision (mAP) when quantized by different methods.

It can be seen that when tested on the PASCAL VOC dataset, the model accuracies obtained by training in the Real-valued mode were 76.4%, 77.8% and 79.5% respectively; then based on the distillation method of the embodiment of the present application, the binary target detection model LWS-Det with ResNet-18/34/50 as the backbone network was trained respectively, and the accuracies on the test set were 73.2%, 75.8% and 76.9%, which greatly accelerated the calculation speed and saved 6.79/5.88/5.57 times of storage space respectively. Compared with other binary quantization methods, LWS-Det has a significant performance improvement. Using the ResNet-18 backbone network, LWS-Det has improved the mAP performance by 12.3%, 10.5% and 3.6% respectively compared with Bi-Real-Net, BiDet and ReActNet with the same memory and computing resource usage. Similarly, using the ResNet-34 backbone network, LWS-Det outperforms Bi-Real-Net, BiDet, and ReActNet by 12.7%, 10.0%, and 3.5% respectively in mAP performance. In addition, using the ResNet-50 backbone network, LWS-Det outperforms Bi-Real-Net and ReActNet by 11.2% and 3.8% respectively in mAP performance. In addition, under the ResNet-34 backbone network, compared with the 4-bit DoReFa-Net method, LWS-Det uses lower GFLOPs and memory space, but has a 0.2% higher mAP, indicating that the effect of the embodiment of the present application is significantly improved.

When the short and long sides of SSD are 300×300 pixels respectively (i.e., the input is 300×300 in Table 1), the backbone network Backbone adopts VGG-16, and the quantization methods are Real-valued, DeRoFa-Net, Bi-Real-Net, BiDet, ReActNet and LWS-Det respectively, the numerical type W/A, memory usage MemoryUsage, GFLOPs and mAP of the model after quantization by different quantization methods can be seen in Table 1, which will not be repeated here.

It can be seen that LWS-Det can achieve 14.76% computational acceleration and 4.81% storage compression on the SSD framework based on the VGG-16 backbone network. Compared with Real-valued, the mAP performance gap is very small (about 2.9%). Compared with Bi-Real-Net, BiDet, and ReActNet, LWS-Det can improve the mAP performance by 7.6%, 5.4%, and 3.0% respectively under the same GFLOPs and memory usage. The mAP performance of LWS-Det is 2.2% higher than that of the 4-bit DoReFa-Net, and significantly reduces GFLOPs and memory usage.

In summary, compared with the binary neural networks on various detection frameworks mentioned above, LWS-Det achieves state-of-the-art performance and reaches performance close to that of full-precision Real-vaued models, which is demonstrated in a large number of experiments, clearly verifying the advantages of LWS-Det and demonstrating its superiority and versatility in different application scenarios.

Table 1 is a comparison table of experimental results of binary neural network and other networks on PASCAL VOC dataset in the embodiment of this application

Please refer to FIG. 13 , which is a schematic diagram of a training device 1300 for a binary neural network model provided in an embodiment of the present application. The device 1300 includes a determination unit 1310 , a training unit 1320 and a decision unit 1330 ; wherein,

The determining unit 1310 is configured to execute step S1.

The training unit 1320 is used to execute step S2.

The decision unit 1330 is used to execute step S3.

Step S1: Determine the knowledge distillation framework; wherein the teacher network in the knowledge distillation framework is a trained neural network model, and the student network in the knowledge distillation framework is an initial binary neural network model M ₀ , and the teacher network and the student network respectively include N layers of neural networks, where N is a positive integer. Step S2: Use the j+1th batch of images and the target loss function to train the binary neural network model M _j to obtain the binary neural network model M _j+1 ; wherein the binary neural network model M _j is obtained based on the jth batch of images, and j is a positive integer; the target loss function includes an angle loss term, which is used to describe the difference between the first angle corresponding to the i-th layer of the neural network in the teacher network and the second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix of the j+1th batch of images in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix of the j+1th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N. Step S3: When the preset conditions are met, the binary neural network model M _j+1 is used as the target binary neural network model; otherwise, j=j+1 is set and step S2 is repeated.

In a feasible implementation, the above-mentioned target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; the first convolution output result is based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network in the teacher network for the j+1-th batch of images; the second convolution output result is based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, as well as the binary input matrix of the j+1-th batch of images in the i-th layer neural network in the student network.

In a feasible implementation, the above-mentioned objective loss function also includes a weight loss term; wherein the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network.

In a feasible implementation, the training unit is specifically used to: input the j+1th batch of images into the binary neural network model M _j to obtain the predicted values of the j+1th batch of images; update the parameters of each layer of the binary neural network model M _j based on the predicted values of the j+1th batch of images, the labels of the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 .

In a feasible implementation, in the aspect of inputting the j+1th batch of images into the binary neural network model M _j to obtain the predicted values of the j+1th batch of images, the training unit 1320 is specifically used for: P1: obtaining the binary weight matrix of the i-th layer of the neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer of the neural network in the binary neural network model M _j ; P2: obtaining the second convolution output result of the i-th layer of the neural network based on the binary input matrix and binary weight matrix of the j+1th batch of images in the i-th layer of the neural network; wherein, the element at any position in the probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P3: let i=i+1, repeat steps P1-P2, and obtain the predicted values of the j+1th batch of images based on the second convolution output result of the N-th layer of the neural network.

In a feasible implementation, the above-mentioned reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; in terms of obtaining the binary weight matrix of _{the i-th layer neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in} the binary neural network model M j, the training unit 1320 is specifically used to: determine the element at any position in the target binary weight matrix based on the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the second probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix; wherein, the element at any position in the first probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix.

In a feasible implementation, in the aspect of obtaining the second convolution output result of the i-th layer neural network based on the binary input matrix and the binary weight matrix of the j+1-th batch of images in the i-th layer neural network, the training unit 1320 is specifically used to: perform convolution operations based on the binary input matrix and the binary weight matrix of each image in the j+1-th batch of images in the i-th layer neural network, respectively, to obtain a reference feature matrix for each image; and scale the reference feature matrix of each image using the weight scaling factor of the i-th layer neural network to obtain the second convolution output result.

In a feasible implementation, the above parameters include at least one of a probability matrix or a weight scaling factor.

Please refer to Figure 14, which is a schematic diagram of another model training device 1400 provided in an embodiment of the present application. The device 1400 includes a training unit 1410 and a decision unit 1420. The model includes a teacher network and a student network, the teacher network is a trained neural network model, the student network is a binary neural network model, and the teacher network and the student network each include N layers of neural networks, where N is a positive integer.

Training unit 1410 is used to train a binary neural network model using a teacher network and a target loss function; wherein the target loss function includes an angle loss term, which is used to describe the difference between a first angle corresponding to the i-th layer of the neural network in the teacher network and a second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix of the i-th layer of the neural network in the teacher network and the input matrix in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix of the i-th layer of the neural network in the student network and the binary input matrix in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N.

The decision unit 1420 is used to repeatedly execute the above steps until the iteration termination condition is met to obtain the target binary neural network model.

In a feasible implementation, the target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; the first convolution output result is based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network in the teacher network; the second convolution output result is based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, as well as the binary input matrix in the i-th layer neural network in the student network.

In a feasible implementation, the objective loss function also includes a weight loss term; wherein the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network.

In a feasible implementation, in the above-mentioned aspect of training the binary neural network model using the teacher network and the target loss function, the training unit is specifically used to: input the training image into the binary neural network model to obtain the predicted value of the training image; and update the parameters in the binary neural network model based on the predicted value of the training image, the label of the training image and the target loss function.

In a feasible implementation, in the aspect of inputting the training image into the binary neural network model to obtain the predicted value of the training image, the training unit is specifically used for: P1: obtaining the binary weight matrix of the i-th layer neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model; wherein, the element at any position in the probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P2: obtaining the second convolution output result of the i-th layer neural network according to the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network; P3: setting i=i+1, and repeating steps P1-P2, and obtaining the predicted value of the training image based on the second convolution output result of the N-th layer neural network.

In a feasible implementation, the reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; in terms of obtaining the binary weight matrix of the i-th layer neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model, the training unit is specifically used to: determine the element at any position in the target binary weight matrix based on the first probability value corresponding to the element at any position in the first reference weight matrix in the first probability matrix and the second probability value corresponding to the element at any position in the second reference weight matrix in the second probability matrix; wherein the element at any position in the first probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to characterize the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix.

In a feasible implementation, in the aspect of obtaining the second convolution output result of the i-th layer neural network based on the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network, the training unit is specifically used to: perform a convolution operation on the binary weight matrix and the binary input matrix of the training image in the i-th layer neural network to obtain a reference feature matrix of the training image; and scale the reference feature matrix of the training image using the weight scaling factor of the i-th layer neural network to obtain the second convolution output result.

In one possible implementation, the parameter includes at least one of a probability matrix or a weight scaling factor.

Please refer to FIG15 , which is a schematic diagram of the structure of an image processing device 1500 provided in an embodiment of the present application. The device 1500 includes an acquisition unit 1510 and a processing unit 1520 .

The acquisition unit 1510 is used to acquire the image to be processed.

The processing unit 1520 is used to perform image processing on the image to be processed using the target binary neural network model to obtain a predicted value of the image to be processed; wherein the target binary neural network model is obtained through K trainings, and in the j+1th training of the K trainings: the binary neural network model M _j is trained using the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 ; the binary neural network model M _j is the student network in the knowledge distillation framework; the teacher network in the knowledge distillation framework is a trained neural network model, and the teacher network and the student network respectively contain N layers of neural networks, where N is a positive integer; the target loss function contains an angle loss term; K is a positive integer, and j is an integer greater than or equal to zero and less than or equal to K; the angle loss term is used to describe the difference between the first angle corresponding to the i-th layer of the neural network in the teacher network and the second angle corresponding to the i-th layer of the neural network in the student network; the first angle is based on the weight matrix corresponding to the i-th layer of the neural network in the teacher network and the input matrix of the j+1th batch of images in the i-th layer of the neural network in the teacher network; the second angle is based on the binary weight matrix corresponding to the i-th layer of the neural network in the student network and the binary input matrix of the j+1th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N.

In a feasible implementation, the above-mentioned image processing includes at least one of image classification, target detection or image segmentation.

Specifically, the image processing device 1500 can be used to process the corresponding steps of the image processing method 1100 described in FIG. 11 , which will not be described in detail here.

Please refer to Figure 16, which is a schematic diagram of the hardware structure of a model training device 1600 provided in an embodiment of the present application. The model training device 1600 shown in Figure 16 (the device 1600 can be a computer device) includes a memory 1601, a processor 1602, a communication interface 1603 and a bus 1604. Among them, the memory 1601, the processor 1602, and the communication interface 1603 are connected to each other through the bus 1604.

The memory 1601 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1601 may store a program. When the program stored in the memory 1601 is executed by the processor 1602, the processor 1602 and the communication interface 1603 are used to execute the various steps of the training method of the binary neural network model of the embodiment of the present application.

Processor 1602 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the training device of the binary neural network model in the embodiment of the present application, or to execute the model training method of the method embodiment of the present application.

Processor 1602 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the binary neural network model of the present application may be completed by an integrated logic circuit of hardware or software instructions in processor 1602. The above-mentioned processor 1602 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1601, and the processor 1602 reads the information in the memory 1601, and combines its hardware to complete the functions required to be performed by the units included in the training device of the binary neural network model of the embodiment of the present application, or executes the training method of the binary neural network model of the method embodiment of the present application.

The communication interface 1603 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1600 and other devices or a communication network. For example, training data can be obtained through the communication interface 1603.

The bus 1604 may include a path for transmitting information between various components of the device 1600 (eg, the memory 1601 , the processor 1602 , and the communication interface 1603 ).

Please refer to Figure 17, which is a schematic diagram of the hardware structure of the image processing device 1700 provided in an embodiment of the present application. Among them, the image processing device 1700 can be a car, a camera, a computer, a mobile phone, a wearable device or other possible terminal devices, which is not limited by the present application. The image processing device 1700 shown in Figure 17 (the device 1700 can specifically be a computer device) includes a memory 1701, a processor 1702, a communication interface 1703 and a bus 1704. Among them, the memory 1701, the processor 1702, and the communication interface 1703 realize communication connection with each other through the bus 1704.

The memory 1701 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1701 may store a program. When the program stored in the memory 1701 is executed by the processor 1702, the processor 1702 and the communication interface 1703 are used to execute each step of the image processing method of the embodiment of the present application.

Processor 1702 can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the image processing device of the embodiment of the present application, or to execute the image processing method of the method embodiment of the present application.

The processor 1702 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the image processing method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1702. The above-mentioned processor 1702 may also be a general-purpose processor, a digital signal processor (digitalsignal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmablegate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1701, and the processor 1702 reads the information in the memory 1701, and combines its hardware to complete the functions required to be performed by the units included in the image processing device of the embodiment of the present application, or executes the image processing method of the method embodiment of the present application.

The communication interface 1703 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1700 and other devices or a communication network. For example, training data can be obtained through the communication interface 1703.

The bus 1704 may include a path for transmitting information between various components of the device 1700 (eg, the memory 1701 , the processor 1702 , and the communication interface 1703 ).

It should be noted that although the apparatus 1600 and the apparatus 1700 shown in FIG. 16 and FIG. 17 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 1600 and the apparatus 1700 also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatus 1600 and the apparatus 1700 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the apparatus 1600 and the apparatus 1700 may also only include the devices necessary for implementing the embodiments of the present application, and do not necessarily include all the devices shown in FIG. 16 or FIG. 17.

It can be understood that the above-mentioned device 1600 is equivalent to the training device 120 in Figure 1, and the device 1700 is equivalent to the execution device 110 in Figure 1. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (21)

1. An image processing method, characterized in that the method comprises: Get the image to be processed; Performing image processing on the image to be processed using a target binary neural network model to obtain a feature vector of the image to be processed; Based on the feature vector, the target object in the image to be processed is identified and segmented to obtain a target area corresponding to the target object and a multidimensional vector corresponding to the target area, wherein each element in the multidimensional vector corresponds to an image category, and each element is used to represent a probability value that the image to be processed is the image category corresponding to each element; The target binary neural network model is obtained through K trainings, and in the j+1th training among the K trainings: the binary neural network model M _j is trained using the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 ; the binary neural network model M _j is the student network in the knowledge distillation framework; the teacher network in the knowledge distillation framework is a trained neural network model, and the teacher network and the student network respectively include N layers of neural networks, where N is a positive integer; the target loss function includes an angle loss term; K is a positive integer, and j is an integer greater than or equal to zero and less than or equal to K; The angle loss term is used to describe the difference between a first angle corresponding to the i-th layer of the neural network in the teacher network and a second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix corresponding to the i-th layer of the neural network in the teacher network and the input matrix of the j+1-th batch of images in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix corresponding to the i-th layer of the neural network in the student network and the binary input matrix of the j+1-th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N. 2. The method according to claim 1, characterized in that the target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; The first convolution output result is obtained based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network of the j+1-th batch of images in the teacher network; the second convolution output result is obtained based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, and the binary input matrix of the j+1-th batch of images in the i-th layer neural network in the student network. 3. The method according to claim 1 or 2, characterized in that the objective loss function also includes a weight loss term; Among them, the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network. 4. The method according to claim 1 or 2, characterized in that the step of training the binary neural network model M _j using the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 comprises: Inputting the j+1th batch of images into the binary neural network model M _j to obtain predicted values of the j+1th batch of images; The parameters of each layer of the neural network in the binary neural network model M _j are updated based on the predicted values of the j+1th batch of images, the labels of the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 . 5. The method according to claim 4, characterized in that the step of inputting the j+1th batch of images into the binary neural network model M _j to obtain the predicted values of the j+1th batch of images comprises: P1: Based on the reference weight matrix and probability matrix corresponding to the i-th layer neural network in the binary neural network model _Mj, the binary weight matrix of the i-th layer neural network is obtained; wherein the element at any position in the probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P2: Obtaining a second convolution output result of the i-th layer of the neural network according to the binary input matrix of the j+1-th batch of images in the i-th layer of the neural network and the binary weight matrix; P3: Let i=i+1, and repeat steps P1-P2 to obtain the predicted value of the j+1th batch of images based on the second convolution output result of the Nth layer neural network. 6. The method according to claim 5, characterized in that the reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; the binary weight matrix of the i-th layer of the neural network is obtained based on the reference weight matrix and probability matrix corresponding to the i-th layer of the neural network in the binary neural network model _Mj , comprising: Determine an element at any position in the target binary weight matrix based on a first probability value corresponding to an element at any position in the first reference weight matrix in the first probability matrix and a second probability value corresponding to an element at any position in the second reference weight matrix in the second probability matrix; Among them, the element at any position in the first probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix. 7. The method according to claim 5 or 6, characterized in that the step of obtaining the second convolution output result of the i-th layer of the neural network according to the binary input matrix of the j+1-th batch of images in the i-th layer of the neural network and the binary weight matrix comprises: Performing convolution operations on the binary input matrix of each image in the j+1th batch of images in the i-th layer of the neural network and the binary weight matrix to obtain a reference feature matrix for each image; The reference feature matrix of each image is scaled using the weight scaling factor of the i-th layer neural network to obtain the second convolution output result. 8. The method of claim 7, wherein the parameter comprises at least one of the probability matrix or the weight scaling factor. 9. An image processing method, characterized in that the method comprises: Get the image to be processed; Performing image processing on the image to be processed using a target binary neural network model to obtain a feature vector of the image to be processed; Based on the feature vector, the target object in the image to be processed is identified and segmented to obtain a target area corresponding to the target object and a multidimensional vector corresponding to the target area, wherein each element in the multidimensional vector corresponds to an image category, and each element is used to characterize the probability value of the image to be processed being the image category corresponding to each element; wherein the target binary neural network model is obtained by training the initial binary neural network model _M0 in the knowledge distillation framework through a target loss function, wherein the initial binary neural network model _M0 is a student network in the knowledge distillation framework, and the teacher network in the knowledge distillation framework is a trained neural network model; the target loss function includes an angle loss term, and the angle loss term is used to describe the difference between the angle between the feature matrix and the weight matrix in the teacher network and the angle between the feature matrix and the weight matrix in the student network. 10. An image processing device, characterized in that the device comprises: An acquisition unit, used for acquiring an image to be processed; A processing unit, used to perform image processing on the image to be processed using a target binary neural network model to obtain a feature vector of the image to be processed; The processing unit is further used to identify and segment the target object in the image to be processed based on the feature vector, to obtain a target area corresponding to the target object, and a multidimensional vector corresponding to the target area, wherein each element in the multidimensional vector corresponds to an image category, and each element is used to represent a probability value that the image to be processed is the image category corresponding to each element; The target binary neural network model is obtained through K trainings, and in the j+1th training among the K trainings: the binary neural network model M _j is trained using the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 ; the binary neural network model M _j is the student network in the knowledge distillation framework; the teacher network in the knowledge distillation framework is a trained neural network model, and the teacher network and the student network respectively include N layers of neural networks, where N is a positive integer; the target loss function includes an angle loss term; K is a positive integer, and j is an integer greater than or equal to zero and less than or equal to K; The angle loss term is used to describe the difference between a first angle corresponding to the i-th layer of the neural network in the teacher network and a second angle corresponding to the i-th layer of the neural network in the student network; the first angle is obtained based on the weight matrix corresponding to the i-th layer of the neural network in the teacher network and the input matrix of the j+1-th batch of images in the i-th layer of the neural network in the teacher network; the second angle is obtained based on the binary weight matrix corresponding to the i-th layer of the neural network in the student network and the binary input matrix of the j+1-th batch of images in the i-th layer of the neural network in the student network; i is a positive integer less than or equal to N. 11. The device according to claim 10, characterized in that the target loss function also includes a convolution result loss term; wherein the convolution result loss term is used to describe the difference between the first convolution output result of the i-th layer neural network in the teacher network and the second convolution output result of the i-th layer neural network in the student network; The first convolution output result is obtained based on the weight matrix of the i-th layer neural network in the teacher network and the input matrix of the i-th layer neural network of the j+1-th batch of images in the teacher network; the second convolution output result is obtained based on the binary weight matrix and the corresponding weight scaling factor corresponding to the i-th layer neural network in the student network, and the binary input matrix of the j+1-th batch of images in the i-th layer neural network in the student network. 12. The device according to claim 10 or 11, characterized in that the objective loss function also includes a weight loss term; Among them, the weight loss term is used to describe the difference between the weight matrix of the i-th layer neural network in the teacher network and the binary weight matrix of the i-th layer neural network in the student network. 13. The device according to claim 10 or 11, characterized in that the processing unit is specifically used for: Inputting the j+1th batch of images into the binary neural network model M _j to obtain predicted values of the j+1th batch of images; The parameters of each layer of the neural network in the binary neural network model M _j are updated based on the predicted values of the j+1th batch of images, the labels of the j+1th batch of images and the target loss function to obtain the binary neural network model M _j+1 . 14. The device according to claim 13, characterized in that, in the aspect of inputting the j+1th batch of images into the binary neural network model M _j to obtain the predicted values of the j+1th batch of images, the processing unit is specifically used to: P1: Obtain the binary weight matrix of the i-th layer of the neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer of the neural network in the binary neural network model _Mj ; P2: Obtain the second convolution output result of the i-th layer neural network according to the binary input matrix of the j+1-th batch of images in the i-th layer neural network and the binary weight matrix; wherein the element at any position in the probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the reference weight matrix; P3: Let i=i+1, repeat steps P1-P2, and obtain the predicted value of the j+1th batch of images based on the second convolution output result of the Nth layer neural network. 15. The device according to claim 14, characterized in that the reference weight matrix includes a first reference weight matrix and a second reference weight matrix, and the probability matrix includes a first probability matrix and a second probability matrix; in the aspect of obtaining the binary weight matrix of the i-th layer of the neural network based on the reference weight matrix and probability matrix corresponding to the i-th layer of the neural network in the binary neural network model M _j , the processing unit is specifically used to: Determine an element at any position in the target binary weight matrix based on a first probability value corresponding to an element at any position in the first reference weight matrix in the first probability matrix and a second probability value corresponding to an element at any position in the second reference weight matrix in the second probability matrix; Among them, the element at any position in the first probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the first reference weight matrix; the element at any position in the second probability matrix is used to represent the probability value of the element at any position in the binary weight matrix taking the element at any position in the second reference weight matrix. 16. The device according to claim 14 or 15, characterized in that, in the aspect of obtaining the second convolution output result of the i-th layer of the neural network according to the binary input matrix of the j+1-th batch of images in the i-th layer of the neural network and the binary weight matrix, the processing unit is specifically used to: Performing convolution operations on the binary input matrix of each image in the j+1th batch of images in the i-th layer of the neural network and the binary weight matrix to obtain a reference feature matrix for each image; The reference feature matrix of each image is scaled using the weight scaling factor of the i-th layer neural network to obtain the second convolution output result. 17. The apparatus of claim 16, wherein the parameter comprises at least one of the probability matrix or the weight scaling factor. 18. An image processing device, characterized in that the device comprises: An acquisition unit, used for acquiring an image to be processed; A processing unit, used to perform image processing on the image to be processed using a target binary neural network model to obtain a feature vector of the image to be processed; The processing unit is further used to identify and segment the target object in the image to be processed based on the feature vector, to obtain a target area corresponding to the target object, and a multidimensional vector corresponding to the target area, wherein each element in the multidimensional vector corresponds to an image category, and each element is used to represent a probability value that the image to be processed is the image category corresponding to each element; Among them, the target binary neural network model is obtained by training the initial binary neural network model _M0 in the knowledge distillation framework through the target loss function, the initial binary neural network model _M0 is the student network in the knowledge distillation framework, and the teacher network in the knowledge distillation framework is a trained neural network model; the target loss function includes an angle loss term, and the angle loss term is used to describe the difference between the angle between the feature matrix and the weight matrix in the teacher network and the angle between the feature matrix and the weight matrix in the student network. 19. A chip system, characterized in that the chip system comprises a processor and a memory; wherein: The memory is used to store the target binary neural network model and program instructions; The processor is used to read the program instructions to call the target binary neural network model to execute the method as described in any one of claims 1 to 9. 20. A terminal device, characterized in that the terminal device comprises the chip system as described in claim 19, and a discrete device coupled to the chip system; wherein the terminal device comprises a car, a camera, a computer, a mobile phone or a wearable device. 21. A computer-readable storage medium, characterized in that the computer-readable storage medium stores program codes for execution by a device, the program codes comprising code for executing the method according to any one of claims 1 to 9.

Images