CN116168400A - Knowledge distillation method, device, electronic device and storage medium - Google Patents

Abstract

The embodiment of the present application discloses a knowledge distillation method, device, electronic equipment and storage medium. Obtain the characteristics to be processed of the image to be processed extracted by the feature extraction layer of the network to be classified, and the classification results obtained by classifying the characteristics to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes a teacher network and a student network; according to The features to be processed of the network determine the feature weights of the network to be classified; according to the features to be processed, feature weights and classification results of different networks to be classified, determine the target loss function when the teacher network and the student network perform knowledge distillation; according to the target loss function, Adjust the network parameters of the student network. The embodiment of the present application improves the classification accuracy and speed, and improves the applicability of the classification network.

Description

Knowledge distillation method, device, electronic device and storage medium

technical field

The embodiments of the present application relate to image processing technologies, and in particular, to a knowledge distillation method, device, electronic equipment, and storage medium.

Background technique

During the production and construction of various power grid projects, the files will be stored in the file management system in the form of scanned copies. When archivists check various approval procedures, one of the most important tasks in the inspection content is to confirm whether the signatures, dates, approval opinions, etc. in relevant documents are handwritten or machine typed. Manual verification is not only inefficient, but also prone to errors due to visual fatigue.

In the prior art, a deep learning model can be used to determine whether the font in a specified area in the file of the power grid project is a handwritten font or a machine typed font.

However, the shallower network layers of the deep learning model cannot meet the accuracy requirements, and the deeper the network layers will affect the reasoning speed of the model, occupy more resources, and have poor applicability.

Contents of the invention

The present application provides a knowledge distillation method, device, electronic equipment and storage medium to improve classification accuracy and speed, and improve the applicability of classification networks.

In the first aspect, the embodiment of the present application provides a knowledge distillation method, the knowledge distillation method includes:

Obtaining the features to be processed of the picture to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes a teacher network and a student network;

Determine the feature weight of the network to be classified according to the characteristics to be processed of the network to be classified;

According to the features to be processed, feature weights and classification results of different networks to be classified, determine the target loss function when the teacher network and the student network perform knowledge distillation;

According to the objective loss function, the network parameters of the student network are adjusted.

In the second aspect, the embodiment of the present application also provides a knowledge distillation device, the knowledge distillation device includes:

The parameter acquisition module is used to obtain the characteristics to be processed of the image to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by classifying the characteristics to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes the teacher network and student network;

The feature weight determination module is used to determine the feature weight of the network to be classified according to the characteristics to be processed of the network to be classified;

The target loss function determination module is used to determine the target loss function when the teacher network and the student network perform knowledge distillation according to the characteristics to be processed, feature weights and classification results of different networks to be classified;

The network parameter adjustment module is used to adjust the network parameters of the student network according to the target loss function.

In the third aspect, the embodiment of the present application also provides electronic equipment, the electronic equipment includes:

one or more processors;

storage means for storing one or more programs;

When one or more programs are executed by one or more processors, one or more processors are made to implement any knowledge distillation method provided in the embodiments of the present application.

In a fourth aspect, the embodiment of the present application further provides a storage medium including computer-executable instructions, and the computer-executable instructions are used to execute any one of the knowledge distillation methods provided in the embodiments of the present application when executed by a computer processor.

This application obtains the features to be processed of the pictures to be processed extracted by the feature extraction layer of the network to be classified, and the classification results obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes a teacher network and a student network, It is convenient for subsequent supervised learning of the corresponding layer of the student network according to the feature extraction layer and classification layer of the teacher network; according to the characteristics of the network to be classified, the feature weight of the network to be classified is determined, and the learning of important features is strengthened through the feature weight. The classification accuracy of the subsequent student network; according to the characteristics to be processed, feature weights and classification results of different networks to be classified, determine the target loss function when the teacher network and the student network perform knowledge distillation; adjust the network parameters of the student network according to the target loss function ; The target loss function includes the loss function of the feature extraction layer and the classification layer. When the teacher network and the student network perform knowledge distillation, they can learn the parameters of the feature extraction layer and the classification layer of the teacher network at the same time, and improve the consistency of the student network and the teacher network. Improve the classification accuracy and speed of the student network, and at the same time, because the layer of the student network itself is shallow, it occupies less resources, thereby improving the applicability. Therefore, through the technical solution of this application, the problems of shallow network layers that cannot meet the accuracy requirements, deepened network layers will affect the reasoning speed of the model, occupy more resources, and poor applicability have been solved, and the problems of improving classification accuracy and Speed, the effect of improving the applicability of classification networks.

Description of drawings

FIG. 1 is a flow chart of a knowledge distillation method in Embodiment 1 of the present application;

Figure 2a is a flow chart of a knowledge distillation method in Embodiment 2 of the present application;

FIG. 2b is a schematic diagram of a feature weight determination process in Embodiment 2 of the present application;

Fig. 3a is a flow chart of a knowledge distillation method in Embodiment 3 of the present application;

Fig. 3b is a schematic diagram of a process for determining a target loss function in Embodiment 3 of the present application;

Fig. 4 is a schematic structural diagram of a knowledge distillation device in Embodiment 4 of the present application;

FIG. 5 is a schematic structural diagram of an electronic device in Embodiment 5 of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is an embodiment of a part of the application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

It should be noted that the terms "first" and "second" in the specification and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Embodiment one

Fig. 1 is a flow chart of a knowledge distillation method provided in Embodiment 1 of the present application. This embodiment can be applied to the case of performing knowledge distillation on the classification network when classifying handwritten characters in the files of power grid construction. It can be executed by a knowledge distillation device, which can be implemented by software and/or hardware, and specifically configured in electronic equipment, such as a server.

Referring to the knowledge distillation method shown in Figure 1, it specifically includes the following steps:

S110. Obtain the features to be processed of the picture to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes a teacher network and a student network.

The network to be classified may be a network capable of classifying handwritten characters and machine-typed characters, and is used to classify handwritten characters in archive documents of power grid construction projects. Exemplarily, the network to be classified may be a convolutional neural network, a residual network, or a deep belief network. Specifically, the network to be classified includes a teacher network and a student network. The teacher network can be a trained classification network, that is, a network with fixed network parameters, which is used to train the student network; the student network is a network that needs to be trained, that is, The network parameters need to be tuned, and the network parameters are shallow.

The knowledge distillation method can use the trained teacher network to obtain a student network that is more suitable for reasoning by transferring knowledge. The teacher network can be a network with a deep network layer, and the student network can be a network with a shallow network layer. Exemplarily, the teacher network may be resnet34 (Residual Network 34, 34-layer residual network), and the student network may be resnet18 (Residual Network 18, 18-layer residual network). Specifically, after building the teacher network, use the sample training set to train the teacher network, obtain the trained teacher network, and fix the teacher network; then use the trained teacher network to supervise the training of the student network, so that the student network can reduce the amount of parameters At the same time, it can also learn the excellent performance of the teacher network and improve the reasoning accuracy of the student network.

The feature extraction layer may be a convolutional layer of the network to be classified, and is used for feature extraction of the input image to be processed. Exemplarily, the feature extraction layer may be the last layer in the multi-layer convolutional layer, that is, there is no other convolutional layer between the feature extraction layer and the output layer. The image to be processed may be sample data, for example, a signature image in an archive document of a power grid construction project. The feature to be processed of the picture to be processed may be a picture output after the feature extraction layer performs feature extraction on the picture to be processed, which is used to represent the feature extracted by the feature extraction layer. The classification layer can be the output layer of the network to be classified, and is used to determine the classification result according to the features obtained by the feature extraction layer. Exemplarily, the classification layer may be a fully connected layer. The classification result may be a result of classifying the features to be processed. Exemplarily, when the network to be classified is used to classify whether the signature picture is handwritten, the classification result may include handwritten and machine typed.

S120. Determine feature weights of the network to be classified according to the characteristics to be processed of the network to be classified.

The feature weights may be weights determined according to the features to be processed, and are used to supervise the training of the student network, so that the student network can learn richer discriminative features of the teacher network. Exemplarily, the feature weight of the network to be classified can be determined through an attention mechanism, so that the student network can pay more attention to specific information in the teacher network, for example, edge features. Specifically, based on the attention mechanism, the feature weight of the teacher network is obtained according to the feature extraction layer of the teacher network; based on the attention mechanism, the feature weight of the student network is obtained according to the feature extraction layer of the student network.

S130. According to the features to be processed, feature weights, and classification results of different networks to be classified, determine a target loss function when the teacher network and the student network perform knowledge distillation.

The target loss function can be used for the student network to distill the teacher network, and supervise the learning of the student network through the loss function. Exemplarily, the loss function can be KL divergence (Kullback-Leibler Divergence, a loss function), mean square error (Mean Square Error, a loss function), L1 loss function (professional term, a loss function, average Absolute error) and at least one of deep mutual learning loss, etc. The target loss function may be the total loss function of the distillation network composed of the network to be classified.

Specifically, the loss function learned by the student network for the feature extraction layer of the teacher network can be determined according to the features to be processed and the feature weights of different networks to be classified; the classification of the teacher network by the student network can be determined according to the classification results of different networks to be classified The loss function for layer learning, and the target loss function is obtained according to the weighted sum of the two loss functions.

S140. Adjust network parameters of the student network according to the target loss function.

The network parameters may be parameters learned during training in the student network. Exemplarily, the network parameter may be a convolution kernel. After obtaining the value of the target loss function, according to the target loss function, the gradient descent algorithm is used to train the entire distillation network and adjust the network parameters of the student network.

The power grid construction project archives include a large number of archive project files, and many documents in these files need to be signed and approved by the corresponding person in charge to ensure the validity of the files. The case files will be stored in the archives department in the form of scanned copies, and then the archives department will check the various approval procedures. One of the most important tasks in the inspection content is to confirm whether the signatures, dates, and approval opinions in the relevant documents are handwritten or not. machine typing, and give alarm feedback to the violation procedures of machine typing. A project to be reviewed usually produces thousands of documents, and a document usually contains a large number of items to be reviewed. Currently, it is manually checked, which is not only inefficient, but also prone to errors due to visual fatigue. Using computer technology to locate the position of the item to be reviewed and automatically classify whether it is handwritten will greatly reduce the workload of the staff. At present, there is no automatic classification model for handwritten characters in the review of power grid construction projects.

Distinguishing machine typing and handwriting can be abstracted as a two-category task for pictures. In the prior art, a lot of work has been done on image classification tasks. The network model based on deep learning has become the mainstream model. The classic network has LeNet-5 (professional term, a deep learning network) convolutional neural network (5-layer network) , AlexNet (professional term, a deep learning network) network (8-layer network), VGG-16 (professional term, a deep learning network) network (16-layer network), GoogLeNet (professional term, a deep learning network) (22-layer network) and ResNet-101 (101-layer network), etc. As the number of network layers continues to deepen, the higher the classification accuracy of the network model, the more robust the algorithm; but at the same time, the number of parameters of the model will become larger and larger, which will affect the operation speed of the network model and occupy more computing resources. It is not conducive to the landing application of technology. However, if a shallow network model is used, the classification accuracy is poor, and it cannot meet the requirements of the technology's landing application.

In the technical solution of this embodiment, the features to be processed of the pictures to be processed extracted by the feature extraction layer of the network to be classified are obtained, and the classification results obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes teachers The network and the student network are convenient for subsequent supervised learning of the corresponding layer of the student network according to the feature extraction layer and classification layer of the teacher network; according to the characteristics of the network to be classified, the feature weight of the network to be classified is determined, and the important weight is strengthened through the feature weight. The learning of features improves the classification accuracy of subsequent student networks; according to the characteristics to be processed, feature weights and classification results of different networks to be classified, determine the target loss function when the teacher network and student network perform knowledge distillation; according to the target loss function, adjust The network parameters of the student network; the target loss function includes the loss function of the feature extraction layer and the classification layer. When the teacher network and the student network perform knowledge distillation, the parameters of the feature extraction layer and the classification layer of the teacher network can be learned at the same time, and the student network and teacher network can be improved. The consistency of the network improves the classification accuracy and speed of the student network. At the same time, because the layer of the student network itself is shallow, it occupies less resources, thereby improving the applicability. Therefore, through the technical solution of this application, the problems of shallow network layers that cannot meet the accuracy requirements, deepened network layers will affect the reasoning speed of the model, occupy more resources, and poor applicability have been solved, and the problems of improving classification accuracy and Speed, the effect of improving the applicability of classification networks.

Embodiment two

Fig. 2a is a flowchart of a knowledge distillation method provided in Embodiment 2 of the present application. The technical solution of this embodiment is further refined on the basis of the above technical solution.

Further, "determine the feature weight of the network to be classified according to the characteristics to be processed of the network to be classified" is refined as: "extract the edge features of the features to be processed of the network to be classified; according to the edge features, through the weight mapping function, determine The feature weight of the network to be classified" to determine the feature weight of the network to be classified.

See a knowledge distillation method shown in Figure 2a, including:

S210. Obtain the features to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes a teacher network and a student network.

S220. Extract edge features of the features to be processed of the network to be classified.

The edge feature can be a discontinuous feature among the features to be processed, and is used to determine the weight feature. Exemplary, can be by Roberts operator (Roberts operator, professional noun, a kind of edge detection operator), Sobel operator (Sobel operator, professional noun, a kind of edge detection operator) and Laplacian edge One of detection operators etc. to extract edge features. The edge feature is an important feature in the picture, and the classification accuracy of the picture can be improved by learning the edge feature. For handwriting and machine typing, its edge features are an important classification basis, and the classification accuracy can be improved by extracting edge features.

In an optional embodiment, extracting the edge features of the features to be processed in the network to be classified includes: determining the frequency domain features of the features to be processed by Fourier transform; determining the height of the frequency domain features by inverse Fourier transform Frequency features; determine the edge features of high-frequency features through the Sobel sobel operator.

The frequency domain feature may be the corresponding feature in the frequency domain of the feature to be processed, that is, the corresponding feature data after the feature to be processed is transformed from the time domain to the frequency domain. Specifically, the frequency domain features of the features to be processed can be obtained by performing Fourier transform on the features to be processed. High-frequency features can be features that change drastically in the picture and are used to extract edge features. Specifically, high-frequency features of the frequency-domain features may be obtained by inverse Fourier transforming the frequency-domain features. The Sobel operator is easy to implement in space, the edge detection effect is better, and it is less affected by noise, so the edge features of high-frequency features are extracted through the Sobel operator.

Determine the frequency domain features of the features to be processed through Fourier transform; determine the high frequency features of the frequency domain features through inverse Fourier transform, and use the Fourier transform and inverse transform of the picture to obtain the high frequency features of the features to be processed , which is conducive to the extraction of subsequent edge features and improves the extraction efficiency of subsequent edge features; through the Sobel sobel operator, the edge features of high-frequency features are determined, based on the characteristics of easy implementation, low noise and high accuracy of the sobel operator, improve Extraction efficiency and accuracy of edge features.

S230. Determine the feature weight of the network to be classified by using a weight mapping function according to the edge feature.

The weight mapping function may be a function that maps edge features to a weight matrix, and is used to determine feature weights according to edge features. Exemplarily, the weight mapping function may be a softmax function, and the softmax function may convert the edge features into a weight matrix between 0 and 1, so that the network to be classified pays more attention to the edge features in the picture. In the subsequent process of distillation learning, the feature weight can be used to make the student network focus on learning the edge feature extraction method of the teacher network, so that the student network can learn more robust edge features.

Fig. 2b is a schematic diagram of a feature weight determination process. As shown in Figure 2b, the frequency domain features are obtained by performing Fourier transform on the features to be processed; the high frequency features are obtained by performing inverse Fourier transform on the frequency domain features; the high frequency features are processed by the sobel operator to obtain Edge features, the edge features are mapped through the weight mapping function to obtain feature weights.

S240. According to the features to be processed, feature weights, and classification results of different networks to be classified, determine a target loss function when the teacher network and the student network perform knowledge distillation.

S250. Adjust network parameters of the student network according to the target loss function.

In the technical solution of this embodiment, by extracting the edge features of the features to be processed in the network to be classified; according to the edge features, through the weight mapping function, the feature weights of the network to be classified are determined, and the edge features are important feature basis for classifying pictures. Extract edge features and map them to feature weights according to the edge features. When knowledge distillation is performed on the student network through the teacher network, the feature weights of the teacher network can be used to supervise the feature weights of the student network, so that the student network can learn more finely from the teacher network. The edge features can improve the accuracy of student network feature extraction, provide more accurate classification basis for subsequent classification, and improve the accuracy of classification.

Embodiment Three

Fig. 3a is a flow chart of a knowledge distillation method provided in Embodiment 3 of the present application. The technical solution of this embodiment is further refined on the basis of the above technical solution.

Further, "according to the characteristics to be processed, feature weights and classification results of different networks to be classified, determine the target loss function when the teacher network and the student network perform knowledge distillation" as: "According to the characteristics to be processed of different networks to be classified Features and feature weights to determine the feature loss function; according to the classification results of different networks to be classified, determine the classification loss function; according to the feature loss function and classification loss function, determine the target loss function when the teacher network and the student network perform knowledge distillation ", to Refine how the objective loss function is determined.

See a knowledge distillation method shown in Figure 3a, including:

S310. Obtain the features to be processed of the picture to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes a teacher network and a student network.

S320. Determine feature weights of the network to be classified according to the characteristics to be processed of the network to be classified.

S330. Determine a feature loss function according to the features to be processed and feature weights of different networks to be classified.

The feature loss function can be a loss function determined according to the to-be-processed features and feature weights of the teacher network and the student network, which is used to supervise the learning of the feature extraction layer parameters of the teacher network by the student network, and improve the feature extraction ability of the feature extraction layer of the student network .

There may be at least one type of feature loss function, and for example, there may be two types of feature loss functions, namely, a loss function including feature weights and a loss function not including feature weights. Specifically, the feature loss function may be a mean square error function.

In an optional embodiment, determining the feature loss function according to the features to be processed and the feature weights of different networks to be classified includes: determining the first loss function according to the features to be processed and the corresponding feature weights of different networks to be classified; A second loss function is determined for different features to be processed of the network to be classified; and a feature loss function is determined according to the first loss function and the second loss function.

The first loss function may be a loss function including feature weights, and is used to supervise the student network to learn edge features of the features to be processed. Exemplarily, the first loss function may be a mean square error function, then the product of the feature weights corresponding to the features to be processed of the teacher network and the feature weights corresponding to the features to be processed, and the weight of the features to be processed of the student network and the features to be processed The product of the feature weights is respectively used as two parameters in the mean square error function to determine the first loss function. Through the first loss function, the student network can be supervised to learn richer edge features. The purpose is to enable the student network to learn the characteristics of the teacher network, and even expect to exceed the performance of the teacher.

The second loss function may be a loss function that does not include feature weights, and is used to supervise the student network to learn the features of the features to be processed. Exemplarily, the first loss function may be a mean square error function, and then the features to be processed of the teacher network and the features to be processed of the student network may be respectively used as two parameters in the mean square error function to determine the second loss function. In terms of feature learning, it is hoped that the last layer of the feature extraction layer of the student network and the teacher network is consistent to the greatest extent. Therefore, the second loss function is used to avoid the phenomenon that the student network deliberately learns the edge features of details and deviates from the parameter distribution characteristics of the teacher model. occur.

The weighted sum of the first loss function and the second loss function is used as the feature loss function. The weighting coefficients for weighting the first loss function and the second loss function can be adjusted according to experience and experiments, and are not specifically limited in this application. Optionally, the sum of the first loss function and the second loss function may be used as the feature loss function, that is, the weights of the first loss function and the second loss function are both 1.

The first loss function is determined according to the features to be processed and the corresponding feature weights of different networks to be classified. Can supervise the student network to learn rich edge features; determine the second loss function according to the characteristics to be processed of different networks to be classified, and can supervise the student network to be consistent with the feature map extraction layer of the teacher network to the greatest extent; according to the first loss function and the second loss function, determine the feature loss function, so that the student network can learn the feature extraction performance of the teacher network while learning more edge features.

In an optional embodiment, determining the first loss function according to the features to be processed and the corresponding feature weights of different networks to be classified includes: obtaining the teacher loss parameter according to the features to be processed and the corresponding feature weights of the teacher network; The features to be processed of the student network and the corresponding feature weights are used to obtain the student loss parameters; according to the teacher loss parameters and the student loss parameters, the first loss function is determined through the mean square loss function.

The teacher loss parameter may be a calculation parameter in the first loss function determined according to the teacher network. Specifically, the teacher loss parameter may be the product of the to-be-processed features of the teacher network and the corresponding feature weights. The student loss parameter may be a calculation parameter in the first loss function determined according to the student network. Specifically, the student loss parameter can be the product of the to-be-processed features of the teacher network and the corresponding feature weights. The first loss function is determined by using the teacher loss parameter and the student loss parameter as two parameters of the mean square loss function.

According to the to-be-processed features of the teacher network and the corresponding feature weights, the teacher loss parameters are obtained; according to the to-be-processed features of the student network and the corresponding feature weights, the student loss parameters are obtained; according to the teacher loss parameters and student loss parameters, through the mean square loss The function determines the first loss function, and the feature weight is used as the weighting coefficient of the feature to be processed, which can focus on learning edge features, improve the ability to extract edge features, and improve the accuracy of subsequent classification.

In an optional embodiment, according to the characteristics to be processed of the student network and the corresponding feature weights, the student loss parameters are obtained, including: using the difference between the identity matrix and the corresponding feature weights of the student network as the weighted weight of the student network; according to the student network The pending features and weighting weights determine the student loss parameters.

Taking the difference between the identity matrix and the corresponding feature weight of the student network as the weighted weight of the student network is equivalent to reducing the weight of the edge detail texture area of the feature to be processed in the student network and increasing the weight of the background area. Exemplarily, when the student network is used to classify handwriting and machine typing, the difference between the identity matrix and the corresponding feature weight of the student network is used as the weighted weight of the student network, which can ensure that a smaller weight is given to the edge of the font, and the background area Giving a larger weight, so that in the process of supervised training, in order to make the first loss function smaller, the student network is encouraged to pay more attention to the edge detail texture information of the features to be processed, learn more abundant features at the edge, and weaken the background. Regional learning can optimize student networks. The product of the features to be processed and the weighted weights of the student network is determined as the student loss parameter.

By using the difference between the unit matrix and the corresponding feature weight of the student network as the weighted weight of the student network; according to the characteristics to be processed and the weighted weight of the student network, the student loss parameter is determined, and the student network is prompted to pay more attention to the edge detail texture information of the characteristics to be processed , optimize the student network.

S340. Determine a classification loss function according to classification results of different networks to be classified.

The classification loss function may be a loss function determined according to the classification results of the teacher network and the student network, and is used to supervise the learning of the classification layer parameters of the teacher network by the student network and improve the classification accuracy of the classification layer of the student network. Exemplarily, the classification loss function may be a KL divergence loss function, and the classification result of the teacher network and the classification result of the student network may be respectively used as two parameters in the KL divergence loss function to determine the classification loss function.

S350. According to the feature loss function and the classification loss function, determine a target loss function when the teacher network and the student network perform knowledge distillation.

According to the weighted sum of the feature loss function and the classification loss function, the target loss function when the teacher network and the student network perform knowledge distillation is determined. The weighting coefficients for weighting the feature loss function and the classification loss function can be adjusted according to experience or experiments, which is not specifically limited in this application.

Fig. 3b is a schematic diagram of a process for determining a target loss function. As shown in Figure 3b, the image to be processed is input into the teacher network and the student network at the same time, and the features to be processed obtained by the last convolutional layer of the student network and the teacher network are respectively obtained, as shown in the left dashed box in Figure 3b , according to the feature to be processed, determine the second loss function; weight the corresponding feature to be processed according to the feature weight of the teacher network and the weighted weight of the student network, as shown in the multiplier in Figure 3b, and the upper part of Figure 3b The weighting coefficient of the multiplier for weighting is the feature weight corresponding to the teacher network. The weighting coefficient of the multiplier at the bottom of Figure 3b is the difference between the identity matrix and the corresponding feature weight of the student network, that is, the weighting weight. After the weighting operation, we get The weighted features to be processed are shown in the dotted line box on the right in Figure 3b. According to the features to be processed, the first loss function is determined; the classification results obtained by the classification layers of the student network and the teacher network are obtained respectively, and the classification loss function is determined. The target loss function is obtained according to the first loss function, the second loss function and the classification loss function, that is, the target loss function shown in FIG. 3b.

S360. Adjust network parameters of the student network according to the target loss function.

In the technical solution of this embodiment, by determining the feature loss function according to the features to be processed and the feature weights of different networks to be classified, the student network can learn the feature extraction ability of the teacher network; according to the classification results of different networks to be classified, the classification loss is determined The function can make the student network learn the ability of the teacher network to classify according to the extracted features to be processed; according to the feature loss function and the classification loss function, determine the target loss function when the teacher network and the student network perform knowledge distillation, so that the student network can learn More robust features to be processed that are conducive to image classification, improve the learning ability of the student network and the accuracy of classification, and use a smaller student network to achieve excellent classification performance.

Embodiment four

FIG. 4 is a schematic structural diagram of a knowledge distillation device provided in Embodiment 4 of the present application. This embodiment is applicable to the case of performing knowledge distillation on a classification network when classifying handwritten characters in files for power grid construction. The specific structure of the knowledge distillation device is as follows:

The parameter acquisition module 410 is used to obtain the features to be processed of the picture to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes the teacher network and student networks;

The feature weight determination module 420 is used to determine the feature weight of the network to be classified according to the characteristics to be processed of the network to be classified;

The target loss function determination module 430 is used to determine the target loss function when the teacher network and the student network perform knowledge distillation according to the characteristics to be processed, feature weights and classification results of different networks to be classified;

The network parameter adjustment module 440 is configured to adjust the network parameters of the student network according to the target loss function.

In the technical solution of this embodiment, the features to be processed of the pictures to be processed extracted by the feature extraction layer of the network to be classified are obtained through the parameter acquisition module, and the classification results obtained by classifying the features to be processed by the classification layer of the network to be classified; The network includes a teacher network and a student network, which facilitate subsequent supervised learning of the corresponding layer of the student network according to the feature extraction layer and classification layer of the teacher network; the feature weight determination module determines the characteristics of the network to be classified according to the characteristics to be processed of the network to be classified Weight, through the feature weight to strengthen the learning of important features, improve the classification accuracy of the subsequent student network; through the target loss function determination module, according to the characteristics to be processed, feature weight and classification results of different networks to be classified, determine the teacher network and student network. The target loss function during knowledge distillation; adjust the network parameters of the student network according to the target loss function through the network parameter adjustment module; the target loss function includes the loss function of the feature extraction layer and the classification layer, and the teacher network and the student network can perform knowledge distillation. At the same time, learn the parameters of the feature extraction layer and classification layer of the teacher network, improve the consistency between the student network and the teacher network, and improve the classification accuracy and speed of the student network. applicability. Therefore, through the technical solution of this application, the problems of shallow network layers that cannot meet the accuracy requirements, deepened network layers will affect the reasoning speed of the model, occupy more resources, and poor applicability have been solved, and the problems of improving classification accuracy and Speed, the effect of improving the applicability of classification networks.

Optionally, the feature weight determination module 420 includes:

The edge feature extraction unit is used to extract the edge feature of the feature to be processed of the network to be classified;

The edge feature mapping unit is used to determine the feature weight of the network to be classified through the weight mapping function according to the edge feature.

Optionally, the edge feature extraction unit includes:

A frequency-domain feature determining subunit is used to determine the frequency-domain feature of the feature to be processed through Fourier transform;

The high-frequency feature determination subunit is used to determine the high-frequency feature of the frequency domain feature through inverse Fourier transform;

The edge feature determination subunit is used to determine edge features of high-frequency features through a Sobel operator.

Optionally, the target loss function determination module 430 includes:

A feature loss function determination unit is used to determine the feature loss function according to the features to be processed and feature weights of different networks to be classified;

The classification loss function determination unit is used to determine the classification loss function according to the classification results of different networks to be classified;

The target loss function combination unit is used to determine the target loss function when the teacher network and the student network perform knowledge distillation according to the feature loss function and the classification loss function.

Optionally, the feature loss function determines the unit, including:

The first loss function determination subunit is used to determine the first loss function according to the features to be processed and the corresponding feature weights of different networks to be classified;

The second loss function determination subunit is used to determine the second loss function according to the characteristics to be processed of different networks to be classified;

The feature loss function combining subunit is used to determine the feature loss function according to the first loss function and the second loss function.

Optionally, the first loss function determination subunit is specifically used to: obtain the teacher loss parameter according to the to-be-processed features of the teacher network and the corresponding feature weights; obtain the student loss according to the to-be-processed features of the student network and the corresponding feature weights parameter; according to the teacher loss parameter and the student loss parameter, the first loss function is determined through the mean square loss function.

Optionally, the first loss function determines the subunit, which is also specifically used to: use the difference between the identity matrix and the corresponding feature weight of the student network as the weighted weight of the student network; determine the student loss according to the characteristics to be processed and the weighted weight of the student network parameter.

The knowledge distillation device provided in the embodiment of the present application can implement the knowledge distillation method provided in any embodiment of the present application, and has corresponding functional modules and beneficial effects for performing the knowledge distillation method.

Embodiment five

FIG. 5 is a schematic structural diagram of an electronic device provided in Embodiment 5 of the present application. As shown in FIG. 5 , the electronic device includes a processor 510, a memory 520, an input device 530, and an output device 540; The quantity can be one or more, and a processor 510 is taken as an example in FIG. Take the bus connection as an example.

The memory 520, as a computer-readable storage medium, can be used to store software programs, computer-executable programs and modules, such as program instructions/modules corresponding to the knowledge distillation method in the embodiment of the present application (for example, parameter acquisition module 410, feature weight Determination module 420, target loss function determination module 430 and network parameter adjustment module 440). The processor 510 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 520 , that is, implements the above-mentioned knowledge distillation method.

The memory 520 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the terminal, and the like. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some examples, the memory 520 may further include memory located remotely relative to the processor 510, and these remote memories may be connected to the electronic device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 530 can be used to receive input character information, and generate key signal input related to user settings and function control of the electronic device. The output device 540 may include a display device such as a display screen.

Embodiment six

Embodiment 6 of the present application also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a knowledge distillation method. The method includes: obtaining the feature extraction layer of the network to be classified The features to be processed of the extracted pictures to be processed, and the classification results obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes a teacher network and a student network; according to the characteristics to be processed of the network to be classified, determine the The feature weight of the classification network; according to the characteristics to be processed, feature weights and classification results of different networks to be classified, determine the target loss function when the teacher network and the student network perform knowledge distillation; adjust the network parameters of the student network according to the target loss function.

Of course, in the storage medium containing computer-executable instructions provided in the embodiments of the present application, the computer-executable instructions are not limited to the method operations described above, and can also perform the knowledge distillation method provided in any embodiment of the present application. related operations.

Through the above description about the implementation, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary general-purpose hardware, and of course it can also be realized by hardware, but in many cases the former is a better implementation . Based on this understanding, the essence of the technical solution of this application or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a floppy disk of a computer , read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or optical disc, etc., including several instructions to make an electronic device (which can be a personal computer, A server, or a network device, etc.) executes the methods described in various embodiments of the present application.

It is worth noting that in the embodiments of the search device above, the units and modules included are only divided according to functional logic, but are not limited to the above-mentioned divisions, as long as the corresponding functions can be realized; in addition, each function The specific names of the units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application.

Note that the above are only preferred embodiments and technical principles used in this application. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present application, and the present application The scope is determined by the scope of the appended claims.

Claims (10)

1. A knowledge distillation method, characterized in that, comprising: Obtain the features to be processed of the picture to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the network to be classified includes a teacher network and student networks; determining the feature weight of the network to be classified according to the characteristics to be processed of the network to be classified; According to the features to be processed, feature weights and classification results of different networks to be classified, determine the target loss function when the teacher network and the student network perform knowledge distillation; According to the target loss function, network parameters of the student network are adjusted. 2. The method according to claim 1, wherein said determining the feature weight of said network to be classified according to the characteristics to be processed of said network to be classified comprises: Extracting the edge features of the features to be processed of the network to be classified; According to the edge features, the feature weights of the network to be classified are determined through a weight mapping function. 3. method according to claim 2, is characterized in that, described extracting the edge feature of the feature to be processed of described network to be classified comprises: Determine the frequency domain features of the features to be processed by Fourier transform; Determining the high-frequency features of the frequency-domain features by inverse Fourier transform; The edge features of the high-frequency features are determined through a Sobel operator. 4. The method according to claim 1, wherein, according to the characteristics to be processed, feature weights and classification results of different networks to be classified, determining the target loss function when the teacher network and the student network carry out knowledge distillation includes: Determine a feature loss function according to the features to be processed and feature weights of the different networks to be classified; According to the classification results of the different networks to be classified, determine a classification loss function; According to the feature loss function and the classification loss function, the target loss function when the teacher network and the student network perform knowledge distillation is determined. 5. The method according to claim 4, wherein the feature loss function is determined according to the features to be processed and the feature weights of the different networks to be classified, comprising: Determine a first loss function according to the features to be processed and the corresponding feature weights of the different networks to be classified; Determining a second loss function according to the characteristics to be processed of the different networks to be classified; The feature loss function is determined according to the first loss function and the second loss function. 6. The method according to claim 5, wherein said determining a first loss function according to the features to be processed and the corresponding feature weights of said different networks to be classified comprises: Obtain teacher loss parameters according to the features to be processed and the corresponding feature weights of the teacher network; According to the characteristics to be processed of the student network and the corresponding feature weights, student loss parameters are obtained; The first loss function is determined by means of a mean square loss function according to the teacher loss parameter and the student loss parameter. 7. The method according to claim 6, characterized in that, said according to the characteristics to be processed and the corresponding feature weights of said student network, obtain student loss parameters, comprising: Using the difference between the identity matrix and the corresponding feature weight of the student network as the weighted weight of the student network; The student loss parameter is determined according to the characteristics to be processed of the student network and the weighted weights. 8. A knowledge distillation device, comprising: The parameter acquisition module is used to obtain the features to be processed of the picture to be processed extracted by the feature extraction layer of the network to be classified, and the classification result obtained by classifying the features to be processed by the classification layer of the network to be classified; wherein, the The networks to be classified include teacher networks and student networks; A feature weight determination module, configured to determine the feature weight of the network to be classified according to the features to be processed of the network to be classified; The target loss function determination module is used to determine the target loss function when the teacher network and the student network perform knowledge distillation according to the characteristics to be processed, feature weights and classification results of different networks to be classified; A network parameter adjustment module, configured to adjust the network parameters of the student network according to the target loss function. 9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor implements any of claims 1-7 when executing the program. The knowledge distillation method described. 10. A computer-readable storage medium, on which a computer program is stored, wherein when the program is executed by a processor, the knowledge distillation method according to any one of claims 1-7 is implemented.

Images