CN111507403B - Image classification method, device, computer equipment and storage medium - Google Patents

Abstract

The present application relates to the field of artificial intelligence technology, and provides an image classification method, device, computer equipment and storage medium, which obtains an image to be classified, and inputs at least two image features of the image to be classified into at least two image classifiers, the at least two image classifiers correspond to at least two classification levels, and the image features input to the image classifiers corresponding to the adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features; and obtains the hierarchical classification result of the image to be classified according to the classification result of the image to be classified output by the image classifier at the corresponding classification level. The scheme reduces the similarity between the image features input to the image classifiers corresponding to the adjacent classification levels by applying a similarity constraint relationship between the image features, so that the image classifiers corresponding to the different classification levels pay attention to the different image features on the image to be classified, thereby improving the accuracy of hierarchical classification of the image.

Description

Image classification method, device, computer equipment and storage medium

Technical Field

The present application relates to the field of artificial intelligence technology, and in particular to an image classification method, apparatus, computer equipment and storage medium.

Background Art

With the development of artificial intelligence technology, there are technologies for classifying images based on deep learning technologies such as deep neural networks. For example, an image classifier can be built based on a deep neural network to classify input images. Among them, for traditional image classification tasks, the status of each image category is generally equal, that is, no category distinction is made for all images. In this case, the image classifier is relatively easy to achieve for relatively simple classification tasks, such as distinguishing between car images and other categories such as cat and dog images. By using hierarchical image classification technology, it is possible to complete the hierarchical classification task by first determining whether the image belongs to the animal category or the non-animal category, and then further learning the cat and dog categories from the animal category, for example.

The hierarchical image classification method provided by the traditional technology often directly inputs the image into two different image classifiers, for example, one for classification training for a large category and the other for classification training for subcategories under the large category. However, the accuracy of hierarchical classification of images using this method is low.

Summary of the invention

Based on this, it is necessary to provide an image classification method, device, computer equipment and storage medium to address the above technical problems.

A method for image classification, comprising:

Get the image to be classified;

Inputting at least two image features of the image to be classified into at least two image classifiers respectively; the at least two image classifiers correspond to at least two classification levels respectively; the image features input into the image classifiers corresponding to adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features;

According to the classification result of the image to be classified at the corresponding classification level output by the image classifier, a hierarchical classification result of the image to be classified is obtained.

An image classification device, comprising:

An image acquisition module, used to acquire images to be classified;

A feature input module, used for inputting at least two image features of the image to be classified into at least two image classifiers; the at least two image classifiers correspond to at least two classification levels respectively; the image features input to the image classifiers corresponding to adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features;

The result acquisition module is used to acquire the hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.

A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

Acquire an image to be classified; input at least two image features of the image to be classified into at least two image classifiers; the at least two image classifiers correspond to at least two classification levels respectively; there is a similarity constraint relationship between the image features input to the image classifiers corresponding to adjacent classification levels, which is used to reduce the similarity between the image features; and obtain a hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

Acquire an image to be classified; input at least two image features of the image to be classified into at least two image classifiers; the at least two image classifiers correspond to at least two classification levels respectively; there is a similarity constraint relationship between the image features input to the image classifiers corresponding to adjacent classification levels, which is used to reduce the similarity between the image features; and obtain a hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.

The above-mentioned image classification method, device, computer equipment and storage medium obtain an image to be classified, and input at least two image features of the image to be classified into at least two image classifiers, the at least two image classifiers correspond to at least two classification levels respectively, and the image features input into the image classifiers corresponding to the adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features; then, the hierarchical classification result of the image to be classified can be obtained according to the classification result of the image to be classified output by the image classifier at the corresponding classification level. The scheme imposes a similarity constraint relationship between image features, thereby reducing the similarity between image features input into the image classifiers corresponding to the adjacent classification levels as much as possible, so that the image classifiers corresponding to different classification levels can pay attention to different image features on the same image to be classified, and classify the image at their respective classification levels according to the corresponding image features, thereby improving the accuracy of hierarchical classification of images, and can simultaneously complete the classification task of the image to be classified at multiple classification levels.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram of an application environment of an image classification method according to an embodiment;

FIG2 is a schematic diagram of an image classification task in one embodiment;

FIG3 is a schematic diagram of a flow chart of an image classification method in one embodiment;

FIG4 is a schematic flow chart of steps for constructing an image classifier in one embodiment;

FIG5 is a schematic diagram of the principle of image classification in one embodiment;

FIG6 is a schematic flow chart of steps for obtaining sample image features in one embodiment;

FIG7 is a schematic diagram of an interface for displaying image information in one embodiment;

FIG8 is a schematic diagram of the principle of image classification in an application example;

FIG9 is a structural block diagram of an image classification device in one embodiment;

FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

The image classification method provided by the present application can be applied in an application environment as shown in FIG1 , which is an application environment diagram of the image classification method in one embodiment. The terminal 110 can communicate with the server 120 through a network. The terminal 110 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 120 can be implemented as an independent server or a server cluster consisting of multiple servers.

The image classification method provided in this application relates to the field of artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that can use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, and artificial intelligence can perceive the environment, acquire knowledge and use knowledge to obtain the best results.

Among them, artificial intelligence technology includes computer vision technology. Computer vision technology (CV) refers to the use of terminal devices such as cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing to make the computer processing into an image that is more suitable for human eye observation or transmission to instrument detection. Computer vision technology can include image recognition and image classification technologies, such as identifying whether the image is an image of a car, a cat or a dog.

Combining machine learning technology with computer vision technology can enable the terminal device to intelligently classify the images to be classified based on the learned image classification knowledge. Furthermore, the terminal device can perform hierarchical classification of the images to be classified.

In which, the hierarchical classification task is explained in conjunction with FIG2, FIG2 is a schematic diagram of an image classification task in an embodiment, and FIG2 shows the difference between ordinary classification and hierarchical classification. In general classification tasks, the status of the categories of each image is equal, and the image classifier will not distinguish between the categories, that is, directly identify the image as a cat image, a dog image, a bicycle image or a car image, etc., but in fact, the relationship between different categories is different. For example, in the four categories of cats, dogs, cars and bicycles, cats and dogs both belong to the animal category, and the relationship is relatively close, while the relationship between the animal category and the vehicle category to which cars and bicycles belong is actually relatively far. Therefore, the hierarchical classification task can determine whether the image belongs to the animal category or a non-animal category such as the vehicle category, and then further identify cats, dogs, bicycles or cars in the animal category and the vehicle category, wherein the animal category and the vehicle category belong to the same classification level, which can be called the major classification level; cats and dogs under the animal category and bicycles and cars under the vehicle category belong to another classification level, which can be called the sub-class classification level. The image classification method provided in the present application can obtain the classification results of the image to be classified at each classification level, and obtain the hierarchical classification results of the image to be classified. For example, the hierarchical classification results of the image to be classified belonging to the animal category and the cat image can be obtained.

The image classification method provided in this application can be applied to content review and content understanding tasks involving various categories of images. For example, for video-related services, a video frame extraction strategy can be adopted to achieve content review and understanding of video services.

Specifically, the image classification method provided in the present application may be executed by the terminal 110 or the server 120 alone, or may be executed by the terminal 110 and the server 120 in cooperation with each other.

First, taking the execution of terminal 110 alone as an example, the terminal 110 can obtain an image to be classified, and input at least two image features of the image to be classified into at least two image classifiers; wherein the at least two image classifiers can be pre-configured on the terminal 110, corresponding to at least two classification levels respectively, and there is a similarity constraint relationship between the image features input to the image classifiers corresponding to adjacent classification levels, which is used to reduce the similarity between the image features; finally, the terminal 110 can obtain the hierarchical classification result of the image to be classified according to the classification result of the image to be classified output by the image classifier at the corresponding classification level.

The image classification method provided in the present application can also be executed by the terminal 110 and the server 120 in cooperation. Specifically, the terminal 110 can obtain the image to be classified and send the image to be classified to the server 120. The server 120 inputs the at least two image features of the image to be classified into at least two image classifiers; wherein the at least two image classifiers can be pre-configured on the server 120, corresponding to at least two classification levels respectively, and the image features input to the image classifiers corresponding to adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features; then the server 120 can send the classification results of the image to be classified output by each image classifier at the corresponding classification level to the terminal 110, and the terminal 110 can obtain the hierarchical classification result of the image to be classified based on the classification result.

In one embodiment, as shown in FIG. 3 , FIG. 3 is a flow chart of an image classification method in one embodiment, which provides an image classification method, and takes the method applied to the terminal 110 in FIG. 1 as an example for explanation, including the following steps:

Step S301, obtaining an image to be classified;

In this step, the terminal 110 may obtain an image to be classified. The image to be classified may be an image captured by the terminal 110 through an image acquisition device such as a camera, or may be an image pre-stored in an electronic gallery of the terminal 110. The image to be classified may include objects such as animals and plants. Specifically, the terminal 110 may capture an image of a cat in real time through a camera configured therein, and the image of the cat may be used as the image to be classified.

Step S302, inputting at least two image features of the image to be classified into at least two image classifiers accordingly;

In this step, the terminal 110 may input at least two image features of the image to be classified into at least two image classifiers. The at least two image classifiers may be pre-configured on the terminal 110, and the at least two image classifiers correspond to at least two classification levels, that is, different image classifiers correspond to different classification levels. In conjunction with FIG. 2, in a hierarchical classification task, the terminal 110 may be configured with, for example, two image classifiers, one image classifier for classification at the level of "animals, vehicles", and the other image classifier for classification at the level of "cats and dogs under animals, bicycles and cars under vehicles".

Furthermore, there is a similarity constraint relationship between the image features input to the image classifiers corresponding to the adjacent classification levels, and the similarity constraint relationship is used to reduce the similarity between the image features input to the image classifiers corresponding to the adjacent classification levels. Specifically, taking a three-level classification task as an example, let the first level of classification be: plants or animals; the second level of classification, taking animals as an example: mammals or reptiles; the third level of classification, taking reptiles as an example: lizards or snakes. In this case, two similarity constraints need to be added, the first similarity constraint relationship is applied between the image features input to the image classifiers corresponding to the first and second level classification levels, and the second similarity constraint relationship is applied between the image features input to the image classifiers corresponding to the second and third level classification levels. Among them, image features can usually be represented by vectors, so that similarity constraints can be applied to vectors by reducing the similarity between vectors to reduce the similarity between image features input to the image classifiers corresponding to the adjacent classification levels.

Exemplarily, the similarity constraint relationship may include a mutual information constraint relationship or an orthogonal constraint relationship. For the mutual information constraint relationship, the mutual information between the image features input to the image classifiers corresponding to the adjacent classification levels may be obtained to reduce the similarity between the two image features in a manner that minimizes the mutual information between the two image features. Similarly, for the orthogonal constraint relationship, the similarity between the two image features may be reduced by making the image features input to the image classifiers corresponding to the adjacent classification levels orthogonal to each other.

In the above manner, the terminal 110 can decouple the two image features input to the image classifier corresponding to adjacent classification levels, thereby eliminating the similar parts between different types of image features, so that the two image features correspond to different characteristics on the image respectively, and then enable the image classifier to pay attention to different types of characteristics on the image based on the aforementioned decoupled image features of the input, and use different image features that are more suitable for the corresponding classification level to better and accurately classify the image to be classified at the corresponding classification level, and complete the classification tasks of multiple classification levels at the same time.

Step S303, obtaining a hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.

In this step, the terminal 110 can obtain the classification results output by each image classifier, wherein the classification results may include the classification results of the image to be classified at each classification level. For example, in the case of three image classifiers, the terminal 110 can obtain the classification results corresponding to the three classification levels output by the three image classifiers respectively. The terminal 110 can use the three classification results as the hierarchical classification results of the image to be classified, or select the classification result of one of the classification levels as the required hierarchical classification result. Thus, the terminal 110 can simultaneously complete the classification task of the image to be classified at multiple classification levels.

In the above-mentioned image classification method, the terminal 110 obtains the image to be classified and inputs at least two image features of the image to be classified into at least two image classifiers, the at least two image classifiers correspond to at least two classification levels respectively, and the image features input into the image classifiers corresponding to the adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features; then the terminal 110 can obtain the hierarchical classification result of the image to be classified according to the classification result of the image to be classified output by the image classifier at the corresponding classification level. This scheme imposes a similarity constraint relationship between image features, thereby reducing the similarity between the image features input into the image classifiers corresponding to the adjacent classification levels as much as possible, so that the image classifiers corresponding to different classification levels can pay attention to different image features on the same image to be classified, and classify the image at their respective classification levels according to the corresponding image features, thereby improving the accuracy of hierarchical classification of images, and can simultaneously complete the classification task of the image to be classified at multiple classification levels.

In one embodiment, the step S302 of inputting the at least two image features of the image to be classified into at least two image classifiers respectively may include:

At least two image features are obtained through a pre-built feature extractor and are input into at least two image classifiers accordingly.

In this embodiment, the terminal 110 can use a pre-built feature extractor to obtain at least two image features from the image to be classified, and input them into the aforementioned at least two image classifiers. The feature extractor is constructed based on the aforementioned similarity constraint relationship with the at least two image classifiers. The feature extractor can be implemented based on a neural network model. The terminal 110 can input the image to be classified into a feature extractor based on a neural network model, divide the image features output by the last convolution layer of the feature extractor into the aforementioned at least two image features, and input them into the at least two image classifiers accordingly.

In the scheme of this embodiment, the terminal 110 can extract features of the image to be classified by pre-cooperating with at least two image classifiers and the feature extractor trained under the aforementioned similarity constraint relationship, so as to obtain a variety of image features that can be applied to each image classifier for classification. There is no need to recalculate the similarity between image features each time the image is processed, thereby improving the image classification efficiency.

In one embodiment, further, the feature extractor may further include a feature extraction network and an encoder; the step of obtaining at least two image features through the pre-built feature extractor may specifically include:

The image to be classified is input into a feature extraction network to obtain initial image features output by the feature extraction network; the initial image features are input into an encoder to obtain encoded initial image features output by the encoder; and at least two image features are obtained based on the encoded initial image features.

In this embodiment, the feature extractor may further include a feature extraction network and an encoder, wherein the feature extraction network and the encoder may be implemented based on a neural network model, such as a ResNet residual network model.

The feature extraction network can be used to preliminarily obtain the image features of the image to be classified, and the image features obtained by the feature extraction network are often of high dimension and contain redundant information. Therefore, the terminal 110 can first input the image to be classified into the feature extraction network to obtain the initial image features output by the feature extraction network, and then the terminal 110 further inputs the initial image features into the encoder, wherein the terminal 110 uses the encoder to map the image features to the coding features, which can be used to reduce the feature dimension and play a role in removing the redundant information in the initial image features. Thus, the terminal 110 obtains at least two image features based on the encoded initial image features output by the encoder. Among them, the feature extraction network, encoder and at least two classifiers are trained based on the aforementioned similarity constraint relationship. Using the solution of this embodiment, the terminal 110 can directly use the trained feature extraction network to obtain the initial image features when classifying the image. After using the encoder to perform feature dimensionality reduction, the initial image features encoded by the encoder can be divided into at least two image features, and there is a constraint relationship between the image features input into the image classifiers corresponding to adjacent classification levels, thereby improving the efficiency and accuracy of image classification.

In one embodiment, as shown in FIG. 4 , FIG. 4 is a flow chart of steps for constructing an image classifier in one embodiment. Before obtaining at least two image features through a pre-constructed feature extractor, the feature extractor and the image classifier may be constructed through the following steps:

Step S401, obtaining a sample image and obtaining classification labels of the sample image at at least two classification levels as true classification labels of at least two image classifiers;

In this step, the terminal 110 can obtain sample images, and the number of the sample images is generally multiple. The terminal 110 also needs to obtain the classification label corresponding to the sample image, and the classification label includes the classification label of the sample image at each classification level. In conjunction with Figure 2, the terminal 110 can obtain a cat image as a sample image, and the terminal 110 also needs to obtain the classification labels at two classification levels corresponding to the cat image, namely, "animal" and "cat". Furthermore, the terminal 110 can use the classification labels of the sample image at at least two classification levels as the true classification labels of at least two image classifiers for training the image classifiers and feature extractors.

Step S402: input the sample image into a feature extractor, and obtain at least two sample image features of the same dimension according to the image features of the sample image output by the feature extractor.

Referring to FIG. 5 , FIG. 5 is a schematic diagram of the principle of image classification in one embodiment. The terminal 110 inputs the sample image into the feature extractor, and the feature extractor outputs the image features of the sample image. The terminal 110 further divides the sample image features into at least two sample image features of the same dimension. Specifically, the terminal 110 can divide the sample image features into image features A and image features B of the same dimension. That is, assuming that the dimension of the sample image features is 2d, the terminal 110 splits the sample image features into image features A and image features B of 1d dimensions respectively. Exemplarily, assuming that the image feature dimension of the sample image output by the acquired feature extractor is a 2048-dimensional vector, the splitting process can be to correspond the first 1/2-dimensional vector in the 2048-dimensional vector, that is, the 0th to 1023rd dimension vectors to the image feature A, and the last 1/2-dimensional vector, that is, the 1024th to 2047th dimension vectors to the image feature B.

In one embodiment, inputting the sample image into the feature extractor in step S402 may specifically include: preprocessing the sample image to obtain a sample image with an image size of a preset image size; and inputting the sample image with the preset image size into the feature extractor.

The present embodiment mainly involves that the terminal 110 can pre-process the sample image before inputting it into the feature extractor, and can adjust the image size of the sample image to a preset image size by scaling the image, thereby inputting the sample image of the preset image size into the feature extractor. Specifically, for example, when training models such as image classifiers and feature extractors, images of a fixed image size are usually required. Therefore, the present embodiment can scale sample images of any image size to, for example, an image of a 256×256 image size, and then randomly crop an image of 224×224 from it as a sample image of a preset graphic size for training.

Step S403, inputting at least two sample image features into at least two image classifiers respectively, and obtaining predicted classification labels of the sample images output by the at least two image classifiers at corresponding classification levels;

In this step, referring to FIG5 , the terminal 110 may input the image feature A into the image classifier A, and input the image feature B into the image classifier B. Among them, the image classifier A and the image classifier B are used to classify the images to be classified at different classification levels. In the process of model construction, the image classifier A may predict the classification result according to the input image feature A, and obtain the predicted classification label A. Similarly, the image classifier B may obtain the predicted classification label B according to the input image feature B. The predicted classification label may correspond to the probability value of belonging to a certain category at the corresponding classification level. Specifically, it can be explained in conjunction with FIG2 , and the predicted classification label may be the probability value of the sample image belonging to an animal or a vehicle at the classification level of “animals, vehicles”.

Step S404, constructing a similarity constraint relationship between sample image features input to image classifiers corresponding to adjacent classification levels;

In this step, the terminal 110 constructs a similarity constraint relationship between sample image features input to image classifiers corresponding to adjacent classification levels.

Step S405 , training a feature extractor and at least two image classifiers based on the real classification labels, the predicted classification labels, and the similarity constraint relationship, to construct the feature extractor and at least two image classifiers.

According to the technical solution of this embodiment, the terminal 110 can jointly train the feature extractor and at least two image classifiers based on the real classification labels, predicted classification labels and similarity constraint relationships of the sample images, and construct a feature extractor and at least two image classifiers, so that the trained feature extractor can obtain at least two image features with a similarity constraint relationship from the image to be classified, so as to input them into the aforementioned at least two image classifiers for classification, thereby realizing fast and accurate classification of the image to be classified.

In one embodiment, as shown in FIG. 6 , FIG. 6 is a flow chart of the steps of obtaining sample image features in one embodiment, the feature extractor may include a feature extraction network and an encoder; the step of inputting the sample image into the feature extractor in the above step S402, and obtaining at least two sample image features of the same dimension according to the image features of the sample image output by the feature extractor, may include:

Step S601, inputting the sample image into the feature extraction network to obtain the initial sample image features output by the feature extraction network;

Step S602, inputting the initial sample image features into the encoder to obtain the encoded initial sample image features output by the encoder;

Step S603: split the initial sample image features into at least two sample image features with the same dimension.

The present embodiment mainly involves that the terminal 110 can obtain the aforementioned at least two sample image features based on the feature extraction network and encoder included in the feature extractor. Referring to FIG5 , FIG5 is a schematic diagram of the principle of image classification in one embodiment, the terminal 110 can first input the sample image into the feature extraction network, the feature extraction network can be used to preliminarily obtain image features from the sample image, the terminal 110 obtains the initial sample image features output by the feature extraction network, as described in the above embodiment, the image features obtained by the feature extraction network often contain redundant information and the feature dimension is relatively high, so the terminal 110 further inputs the initial sample image features into the encoder, the encoder can be used to map the image features to the encoding features, so as to reduce the feature dimension of the initial sample image features obtained by the feature extraction network, and remove the redundant information in the initial sample image features, and finally the terminal 110 splits the encoded initial sample image features output by the encoder into the aforementioned at least two sample image features of the same dimension. By adopting the solution of this embodiment, the feature extraction network, the encoder and at least two image classifiers can be trained together based on the similarity constraint relationship, so that the trained feature extraction network, the encoder and at least two image classifiers can be used as a whole as an image classification tool to quickly and accurately classify the images to be classified.

In one embodiment, the step of training the feature extractor and at least two image classifiers based on the real classification labels, the predicted classification labels and the similarity constraint relationship in the above step S405 may include:

According to the true classification labels and the predicted classification labels, first loss functions corresponding to the two classification levels are constructed to obtain at least two first loss functions; according to the similarity constraint relationship, a second loss function is constructed; based on the at least two first loss functions and the second loss function, a feature extractor and at least two image classifiers are trained to maximize the at least two first loss functions and the second loss function.

This embodiment provides a specific method for training a feature extractor and at least two image classifiers. Specifically, the terminal 110 can construct a first loss function based on the real classification label and the predicted classification label, wherein the first loss function may include multiple ones, corresponding to different classification levels respectively. For example, when there are three classification levels, the first loss function includes three. In addition, the terminal 110 also constructs a second loss function based on the similarity constraint relationship, that is, the second loss function is constructed by the similarity constraint relationship between the image features input to the image classifiers corresponding to the adjacent classification levels. Correspondingly, if there are two classification levels, the number of constructed second loss functions is one, and if there are three classification levels, the number of constructed second loss functions is two. Thus, the terminal 110 uses the at least two first loss functions and the second loss function to train the feature extractor and the at least two image classifiers so as to maximize the at least two first loss functions and the second loss function.

Specifically, an image classification task with two classification levels is used for illustration, corresponding to large category classification and subcategory classification. Assume that the large category label is y _super and the subcategory label is y _sub . The first loss functions corresponding to the two classification levels are:

Among them, L _super represents the first loss function of the super category, L _sub represents the first loss function of the sub category, C _super represents the total amount of the super category, and C _sub represents the total amount of the sub category. Represents the true classification label of the large category, represents the true classification label of the subcategory, Represents the predicted classification label of the large category, that is, the probability value of the image to be classified belonging to category i under the large category, The predicted classification label of the subcategory is the probability value that the image to be classified belongs to category i under this subcategory.

In addition, suppose the image features input to the image classifiers of the large category and the subcategory are E _α (x) and E _β (x) respectively. The similarity constraint relationship imposed on the two image features can be a mutual information constraint relationship or an orthogonal constraint relationship. Taking the imposition of mutual information constraint as an example, the corresponding second loss function is:

Among them, L _mul represents the second loss function, and the r in the mutual information constraint represents the gradient reversal layer, which multiplies the gradient by -1 when the network back-propagates the gradient, that is, "reversing the gradient". Among them, ⊙ represents the element-by-element multiplication operation. Note that its value range can reach negative infinity. In order to avoid the image classifier and feature extractor from learning this situation, for the decoupled features E _α (x) and E _β (x), L2 normalization (i.e., L2Normalization) can be used to limit the value range of the features. In this way, only when the two image features are exactly the same, that is, E _α (x) == E _β (x), the second loss function L _mul obtains the minimum value -1, and when the two image features are completely orthogonal and different from each other, the second loss function L _mul obtains the maximum value 0. Since the gradient reversal layer is used, minimizing the mutual information loss after gradient reversal is equivalent to maximizing the second loss function, that is, requiring the two image features to reduce the degree of similarity as much as possible.

Finally, the two first loss functions and the second loss function together coordinately train the entire network based on L=L _super +L _sub +L _mul maximization, including training the feature extractor and at least two image classifiers. When the feature extractor includes a feature extraction network and an encoder, the feature extraction network, the encoder and at least two image classifiers are trained together.

In one embodiment, the step S302 of inputting the at least two image features of the image to be classified into at least two image classifiers respectively may include:

The image to be classified is sent to the server, so that the server inputs at least two image features of the image to be classified into at least two image classifiers to obtain the classification results of the image to be classified at the corresponding classification level output by the image classifier; and the classification results obtained by the server are received.

Referring to FIG1 , the main process of image classification processing in this embodiment can be handled by the server 120. Specifically, the terminal 110 can obtain the image to be classified, and then send the image to be classified to the server 120, and the server 120 can be pre-configured with the above-mentioned at least two image classifiers. After receiving the image to be classified, the server 120 inputs the at least two image features of the image to be classified into at least two image classifiers, obtains the classification result of the image to be classified output by the image classifier at the corresponding classification level, and then sends the classification result to the terminal 110, and the terminal 110 receives the classification result sent by the server 120.

By adopting the technical solution of this embodiment, the terminal 110 can transfer the task of image classification processing to the server 120 for processing, so as to reduce the data processing pressure of the terminal 110.

In one embodiment, as shown in FIG. 7 , which is a schematic diagram of an interface for displaying image information in one embodiment, after obtaining the hierarchical classification result of the image to be classified at the corresponding classification level according to the classification result of the image to be classified output by the image classifier in step S303, the following steps may also be included:

Obtain image classification information carrying hierarchical classification results; and display the image classification information on the image to be classified.

The main feature of this embodiment is that the terminal 110 can directly display the classification result of the image to be classified on the image to be classified. Referring to FIG7 , the image to be classified 700 can be displayed on the terminal 110. After the terminal 110 obtains the hierarchical classification result of the image to be classified 700, it can display the image classification information carrying the hierarchical classification result in the information display area 710. Among them, the hierarchical classification result of the image to be classified 700 may include the major classification result A1 and the sub-class classification result B2 of the image to be classified. Specifically, assuming that the image to be classified 700 is a cat image, the image classification information displayed by the terminal 110 may include the major classification result: animal; sub-class classification result: cat. By adopting the technical solution of this embodiment, the hierarchical classification result can be superimposed and displayed on the image to be classified, thereby improving the display efficiency of the hierarchical classification result.

In order to more clearly illustrate the technical solution provided by the present application, the principle of image classification is described in detail in conjunction with FIG8 , which is a schematic diagram of the principle of image classification in an application example.

In general, the input image (x) can be an image of any size, and the training of the model (including the training of the feature extraction network, encoder, and major and sub-category classifiers) generally requires the use of images of fixed image size. Therefore, the image of any image size can be first adjusted to an image size of 256×256, and then a 224×224 image size can be randomly cropped from it as the image to be processed. Next, the image features f(x) can be extracted using the feature extraction network. Then, an encoder is used to map the image features to the encoding features E(x), which can be regarded as the image features before decoupling. The encoding features are then divided into two parts of features, corresponding to two parts of decoupled features. Among them, the first part of the features E _α (x) is used to train the major classifier, and the second part of the features E _β (x) is used to train the sub-category classifier. At the same time, a mutual information constraint is imposed between the two decoupled features to reduce the similarity between the two decoupled features. It should be noted that the two parts of features extracted from the image to be processed and input into the major category classifier and the subcategory classifier may not have any correlation characteristics, that is, it is only necessary to provide the image to be processed for the major category and subcategory classifiers, and there is no need to do special processing on the image features of the image in advance. By inputting the image to be processed into the model, the training of models including feature extraction network, encoder, major category and subcategory classifier can be completed automatically. Based on the trained model, the hierarchical classification of the image can be realized by inputting the image to be classified.

Specifically, assuming that the input image is x, its major category label is y _super , and its subcategory label is y _sub , a feature extraction network is used to extract image features. The feature extraction network is not specifically limited, for example, various neural network models can be used. Generally speaking, the output of the last convolutional layer in the neural network can be used as the image extraction feature f(x). Next, an encoder is used to map the image extraction feature f(x) to the encoding feature E(x), and the dimension of the encoding feature is 2d. The structure of the encoder can use a single-layer fully connected layer. Then, the 2d-dimensional encoding feature E(x) is split into two parts of the same dimension: E(x)→[E _α (x); E _β (x)]. Among them, the features E _α (x) and E _β (x) are both 1d-dimensional features, which are used for major category classification and subcategory classification, respectively. The corresponding loss functions are:

Among them, L _super represents the loss function of the super category, L _sub represents the loss function of the sub category, C _super represents the total number of super categories, and C _sub represents the total number of sub categories. Represents the true classification label of the large category, represents the true classification label of the subcategory, Represents the predicted classification label of the large category, that is, the probability value of the image belonging to category i under the large category, Represents the predicted classification label of the subcategory, that is, the probability value of the image belonging to category i under this subcategory.

In addition, in order to ensure that the features E _α (x) and E _β (x) can learn as different image features as possible, a mutual information constraint is imposed between the two image features:

Among them, L _mul represents the mutual information loss function, and the r in the mutual information constraint represents the gradient reversal layer, which multiplies the gradient by -1 when the network backpropagates the gradient, that is, "reversing the gradient". Among them, ⊙ represents the element-by-element multiplication operation. Note that its value range can reach negative infinity. In order to avoid the network learning this situation during training, for the decoupled features E _α (x) and E _β (x), L2 normalization (i.e. L2Normalization) can be used to limit the value range of the features.

In this case, only when the two image features are exactly the same, that is, E _α (x) == E _β (x), the mutual information loss function takes the minimum value -1, and when the two image features are completely orthogonal and different from each other, the second loss function L _mul takes the maximum value 0. Since the gradient reversal layer is used, minimizing the mutual information loss is equivalent to maximizing the mutual information loss function after gradient reversal, that is, requiring the two image features to reduce the similarity as much as possible. Finally, the three loss functions can be used to jointly train the entire network (including the feature extraction network, encoder, and major and sub-category classifiers): L = L _super + L _sub + L _mul .

The technical solution provided by the above application example can decouple the image features of the image into two parts of image features suitable for large category classification and subcategory classification, and at the same time use mutual information constraints to reduce the similarity between the two parts of image features, so that they can focus on different characteristics in the image as much as possible, and better complete the hierarchical classification task.

It should be understood that, although the various steps in the flow charts of Figures 3 to 6 are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in Figures 3 to 6 may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

In one embodiment, as shown in FIG. 9 , FIG. 9 is a structural block diagram of an image classification device in one embodiment, which provides an image classification device. The device may adopt a software module or a hardware module, or a combination of the two to form a part of a computer device. The device 900 specifically includes:

An image acquisition module 901 is used to acquire an image to be classified;

The feature input module 902 is used to input at least two image features of the image to be classified into at least two image classifiers; the at least two image classifiers correspond to at least two classification levels respectively; the image features input to the image classifiers corresponding to adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features;

The result acquisition module 903 is used to acquire the hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.

In one embodiment, the feature input module 902 is further used to obtain at least two image features through a pre-built feature extractor, and input them into at least two image classifiers accordingly; the feature extractor and at least two image classifiers are constructed based on a similarity constraint relationship.

In one embodiment, the feature extractor includes a feature extraction network and an encoder; the feature input module 902 is further used to input the image to be classified into the feature extraction network to obtain the initial image features output by the feature extraction network; input the initial image features into the encoder to obtain the encoded initial image features output by the encoder; based on the encoded initial image features, at least two image features are obtained.

In one embodiment, the apparatus 900 may further include:

A classifier construction module is used to obtain a sample image and obtain classification labels of the sample image at at least two classification levels as real classification labels for at least two image classifiers; input the sample image into a feature extractor, and obtain at least two sample image features of the same dimension based on the image features of the sample image output by the feature extractor; input the at least two sample image features into at least two image classifiers respectively, and obtain predicted classification labels of the sample image output by at least two image classifiers at corresponding classification levels; construct a similarity constraint relationship between sample image features input into image classifiers corresponding to adjacent classification levels; train the feature extractor and at least two image classifiers based on the real classification labels, predicted classification labels and similarity constraint relationships, and construct the feature extractor and at least two image classifiers.

In one embodiment, the feature extractor includes a feature extraction network and an encoder; a classifier construction module is further used to: input the sample image into the feature extraction network to obtain the initial sample image features output by the feature extraction network; input the initial sample image features into the encoder to obtain the encoded initial sample image features output by the encoder; split the initial sample image features into at least two sample image features of the same dimension.

In one embodiment, the classifier construction module is further used to: construct first loss functions corresponding to two classification levels according to the true classification labels and the predicted classification labels to obtain at least two first loss functions; construct a second loss function according to the similarity constraint relationship; based on the at least two first loss functions and the second loss function, train the feature extractor and at least two image classifiers to maximize the at least two first loss functions and the second loss function.

In one embodiment, the similarity constraint relationship includes a mutual information constraint relationship or an orthogonality constraint relationship.

In one embodiment, the classifier construction module is further used to: pre-process the sample image to obtain a sample image with an image size of a preset image size; and input the sample image with the preset image size into the feature extractor.

In one embodiment, the apparatus 900 may further include:

The information display module is used to obtain image classification information carrying hierarchical classification results; and display the image classification information on the image to be classified.

In one embodiment, the feature input module 902 is further used to send the image to be classified to the server, so that the server inputs at least two image features of the image to be classified into at least two image classifiers to obtain the classification results of the image to be classified output by the image classifier at the corresponding classification level; and receives the classification results obtained by the server.

For the specific definition of the image classification device, please refer to the definition of the image classification method above, which will not be repeated here. Each module in the above-mentioned image classification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG10, which is an internal structure diagram of a computer device in one embodiment. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image classification method is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims (12)

1. A method of classifying images, the method comprising:

acquiring a sample image and a classification label of large-class classification and a classification label of sub-class classification of the sample image, wherein the classification label and the classification label of the sub-class classification are respectively used as a real classification label of an image classifier of the large-class classification and a real classification label of the sub-class classification;

Inputting the sample image to a feature extraction network of a feature extractor to obtain initial sample image features output by the feature extraction network; inputting the initial sample image characteristics to an encoder of the characteristic extractor to obtain encoded initial sample image characteristics output by the encoder; splitting the initial sample image features into first partial features and second partial features with the same dimensions;

Inputting the first partial features and the second partial features into the image classifier of the large category classification and the image classifier of the sub-category classification respectively, and acquiring a prediction classification label of a sample image output by the image classifier of the large category classification in the large category classification and a prediction classification label of a sample image output by the image classifier of the sub-category classification in the sub-category classification;

Constructing a similarity constraint relationship between the first partial feature and the second partial feature;

constructing first loss functions respectively corresponding to the large category classification and the sub-category classification according to the real classification label and the prediction classification label to obtain two first loss functions; constructing a second loss function according to the similarity constraint relation; the similarity constraint relationship is a mutual information constraint relationship, and the second loss function is ，Representing the second loss function in question,Representing the gradient inversion layer,Representing an element-by-element multiplication operation,A first partial characteristic is indicated and is indicated,Representing a second partial feature;

training the feature extractor and the two image classifiers based on the two first loss functions and the second loss function such that the two first loss functions and the second loss function are maximized, constructing the feature extractor, the large class classified image classifier, and the sub-class classified image classifier;

acquiring an image to be classified;

Inputting the image to be classified into a feature extraction network of the pre-constructed feature extractor to obtain initial image features output by the feature extraction network, inputting the initial image features into an encoder of the feature extractor, and performing coding mapping on the initial image features through the encoder to obtain coded features with redundant information removed; the feature dimension of the coding feature is lower than the feature dimension of the initial image feature; dividing the coding features into two image features with the same dimension;

correspondingly inputting the two image features into two image classifiers; the two image classifiers are respectively corresponding to the two classification levels of the large category classification and the sub-category classification;

And acquiring a hierarchical classification result of the image to be classified according to the classification result of the image to be classified, which is output by the image classifier, on the corresponding classification level.

2. The method of claim 1, wherein the feature extractor is implemented based on a neural network model.

3. The method of claim 1, wherein the inputting the sample image to a feature extraction network of a feature extractor comprises:

preprocessing the sample image to obtain a sample image with an image size being a preset image size;

and inputting the sample image with the preset image size into a feature extraction network of the feature extractor.

4. The method according to claim 1, wherein the step of obtaining the hierarchical classification result of the image to be classified according to the classification result of the image to be classified on the corresponding classification level output by the image classifier comprises:

acquiring image classification information carrying the hierarchical classification result;

and displaying the image classification information on the image to be classified.

5. The method of claim 1, wherein the inputting the two image features into the two image classifiers comprises:

The image to be classified is sent to a server, so that the server correspondingly inputs the two image features of the image to be classified to the two image classifiers to obtain a classification result of the image to be classified, which is output by the image classifier, on a corresponding classification level;

and receiving the classification result obtained by the server.

6. An image classification apparatus, the apparatus comprising:

The classifier construction module is used for acquiring a sample image and a classification label of large class classification and a classification label of sub class classification of the sample image, and respectively used as a real classification label of the image classifier of the large class classification and a real classification label of the sub class classification; inputting the sample image to a feature extraction network of a feature extractor to obtain initial sample image features output by the feature extraction network; inputting the initial sample image characteristics to an encoder of the characteristic extractor to obtain encoded initial sample image characteristics output by the encoder; splitting the initial sample image features into first partial features and second partial features with the same dimensions; inputting the first partial features and the second partial features into the image classifier of the large category classification and the image classifier of the sub-category classification respectively, and acquiring a prediction classification label of a sample image output by the image classifier of the large category classification in the large category classification and a prediction classification label of a sample image output by the image classifier of the sub-category classification in the sub-category classification; constructing a similarity constraint relationship between the first partial feature and the second partial feature; constructing first loss functions respectively corresponding to the large category classification and the sub-category classification according to the real classification label and the prediction classification label to obtain two first loss functions; constructing a second loss function according to the similarity constraint relation; the similarity constraint relationship is a mutual information constraint relationship, and the second loss function is ，Representing the second loss function in question,Representing the gradient inversion layer,Representing an element-by-element multiplication operation,A first partial characteristic is indicated and is indicated,Representing a second partial feature;

training the feature extractor and the two image classifiers based on the two first loss functions and the second loss function such that the two first loss functions and the second loss function are maximized, constructing the feature extractor, the large class classified image classifier, and the sub-class classified image classifier;

the image acquisition module is used for acquiring images to be classified;

The feature input module is used for inputting the image to be classified into a feature extraction network of the feature extractor, obtaining initial image features output by the feature extraction network, inputting the initial image features into an encoder of the feature extractor, and performing coding mapping on the initial image features through the encoder to obtain coded features with redundant information removed; the feature dimension of the coding feature is lower than the feature dimension of the initial image feature; dividing the coding features into two image features with the same dimension; correspondingly inputting the two image features into two image classifiers; the two image classifiers are respectively corresponding to the two classification levels of the large category classification and the sub-category classification;

the result acquisition module is used for acquiring the hierarchical classification result of the image to be classified according to the classification result of the image to be classified on the corresponding classification hierarchy, which is output by the image classifier.

7. The apparatus of claim 6, wherein the feature extractor is implemented based on a neural network model.

8. The apparatus of claim 6, wherein the classifier construction module is further to:

preprocessing the sample image to obtain a sample image with an image size being a preset image size;

and inputting the sample image with the preset image size into a feature extraction network of the feature extractor.

9. The apparatus of claim 6, wherein the apparatus further comprises:

the information display module is used for acquiring image classification information carrying the hierarchical classification result;

and displaying the image classification information on the image to be classified.

10. The apparatus of claim 6, wherein the feature input module is further configured to:

The image to be classified is sent to a server, so that the server correspondingly inputs the two image features of the image to be classified to the two image classifiers to obtain a classification result of the image to be classified, which is output by the image classifier, on a corresponding classification level;

and receiving the classification result obtained by the server.

11. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 5 when the computer program is executed.

12. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of any one of claims 1 to 5.